Overcoming Agent Throughput Limitations

By default, the Scalyr Agent can comfortably transmit logs at a rate of 2MB/s using the default settings, often much more with the right conditions (including low RTT between your server and DataSet's, and longer log lines). Nonetheless, it is possible to hit limits with the agent's throughput. If you are frequently seeing skipForTooFarBehind messages in the agent log, please consider the following:

If you started using DataSet before June 2020, and you manually configured any settings like max_allowed_request_size or  max_log_offset_size, note that with version 2.1.6 of the Scalyr Agent, the default settings were updated for improved throughput limits. More detail is available here. You may wish to consider updating the agent, removing your customizations and testing to see whether the current agent defaults provide sufficient performance

A core concept of our ingestion API is sessions. By default, the Agent maintains a single session with the DataSet servers, in order to ensure that log lines are delivered in order and prevent duplicates. Version 2.1.15 of the Agent introduces the ability to create multiple sessions with the DataSet servers to increase upload throughput. Note that any single file will be sent over one session, but in the scenario where several log files are being streamed to DataSet, multiple sessions can significantly increase the throughput, subject to CPU and RAM availability on your server. This is configured via default_workers_per_api_key . We also offer the ability to use multiple processes per session, which allocates additional CPU resources to the agent. This is configured via use_multiprocess_copying_workers. For more detail, see the release notes for version 2.1.15.