What does the "monitor_log_write" warning mean?

Here is an example message in the linux_system_metrics.log file:

.... Warning, skipped writing X log lines due to limit set by monitor_log_write_rate option ...

This issue typically occurs for customers who have multiple filesystems mounted, as each filesystem has its own the system metrics. Ultimately, the combined output can exceed the default monitor write rate. A possible symptom of this scenario is missing graph intervals when viewing the "System" dashboard.

You can update your Scalyr Agent configuration to resolve this:

{ 
  ....
  
  implicit_metric_monitor: false,
  
  ...
  monitors: [ 
        { 
           module: "scalyr_agent.builtin_monitors.linux_system_metrics",
           monitor_log_write_rate: 64000,
           monitor_log_max_write_burst: 100000 
        },
        ...
  ]
}

First, disable the implicit_metric_monitor, as we will be defining a metric monitor with specific parameters in the next step.

Next, create the Linux system metrics monitor with a higher write rate. If log lines are still being skipped, increase the write rate to 128000 and the write burst to 256000. Just make sure the  monitor_log_max_write_burst is greater than the monitor_log_write_rate.