What does the "monitor_log_write" warning mean?

Introduction

You may have encountered log events within the Agent's metrics log files that is similar to:

.... Warning, skipped writing X log lines due to limit set by monitor_log_write_rate option ...

(The above example originated from the linux_system_metrics.log file)

This typically occurs for customers who have multiple filesystems mounted, as each filesystem has its own system metrics. Ultimately, the combined output can exceed the default monitor write rate. A possible symptom of this scenario is missing graph intervals when viewing the "System" dashboard.

Configuration Update

You can update the Scalyr Agent configuration to resolve this:

{ 
  ....
  
  implicit_metric_monitor: false,
  
  ...
  monitors: [ 
        { 
           module: "scalyr_agent.builtin_monitors.linux_system_metrics",
           monitor_log_write_rate: -1,
           monitor_log_max_write_burst: -1 
        },
        ...
  ]
}

First, disable implicit_metric_monitor, as we will be defining a metric monitor with specific parameters in the next step.

Next, set both parameter values to -1 to suppress the rate limit. Alternatively, creating the Linux system metrics monitor with a higher write rate will work too. (ex. increase the monitor_log_write_rate to 256000 and monitor_log_max_write_burst to 512000 while ensuring that the  monitor_log_max_write_burst is greater than the monitor_log_write_rate)

Other Agent Monitors

Please note that the config option is also applicable to other monitors such as the Agent's Windows Event Log monitor.