The default values for these properties should work for most environments and should rarely need to be changed.
Basic Configuration Properties
| Property | Description |
|---|---|
| topics | A comma-separated list of Kafka topics to ingest from |
| tasks.max | The maximum number of Scalyr Sink Connector tasks to send data to DataSet. This should not exceed the number of partitions for the topics. See Scalability. |
| scalyr_server | Defaults to app.scalyr.com and can be omitted. For the EU, this should be set to upload.eu.scalyr.com |
| api_key | Logs Write API key from app.scalyr.com/keys |
| event_enrichment | Optional additional attributes to add the DataSet log event specified as a comma-separated list of key-value pairs. For example, env=prod. |
| custom_app_event_mapping | JSON config describing how to map custom application nested Kafka messages to DataSet events. See Custom Application Event Mapping. |
Advanced Configuration Properties
| Property | Default | Description |
|---|---|---|
| compression_type | deflate | Compression type to use for sending log events. Valid values (deflate, none) |
| compression_level | 6 | Compression level for the compression_type. Valid values depend on the compression_type. The default will be used if not specified. |
| add_events_timeout_ms | 20000 | Timeout in milliseconds for DataSet add events call. |
| add_events_retry_delay_ms | 500 | Delay in milliseconds for initial add events retry. This delay is increased exponentially for each retry. |
| batch_send_size_bytes | 4000000 | The batch size that must be reached before events are sent. This is to buffer events into larger batches for increased throughput. |
| batch_send_wait_ms | 5000 | Maximum time to wait in milliseconds between batch sends. This ensures events are sent to DataSet in a timely manner on systems under light load where batch_send_size_bytes may not be reached for longer periods of time. |
Continue to Custom Sink Mappings in Kafka
Comments
0 comments
Please sign in to leave a comment.