Restoring Logs with the Scalyr Agent

Introduction

This article explains how to configure a Scalyr Agent instance to upload logs from a log file when the log file contains a significant (> 200 MB) of data from the past.

Configuration

1. If possible, establish a separate host to perform the upload of these log files. This minimizes the likelihood of any interference with an Agent that is uploading current logs.
2. Extract any compressed log data. Scan the start / end dates within each file to confirm there's no overlap. Prune any log data that is older than your account's retention period. For example, if you have a 30 day retention period, any log events that are older than 30 days won't be stored when uploaded to your account (if a parser is configured to recognize their timestamps).

Note: If there is no timestamp or the log's timestamp isn't recognized, the associated log events will be associated with the ingestion time, which may (or may not) be desirable.

3. Use the cat command to concatenate the uncompressed (rotated) log files into a single file, preferably using the same path as the original
4. Install a Scalyr Agent (but do not run yet) on the host from step (1). Its configuration will be similar to the existing configuration used for the log file, with the following differences:

a. Set the serverHost to an easily identifiable value, ex. "hostname123-reup"
b. Set disable_max_send_rate_enforcement_overrides to true
c. Set the max_log_offset_size to the maximum expected size (in bytes) of the log file from step (3) above, ex. if the log file is 5GB, then use a value of 5 * 1024^3 = 5368709120 This ensures that the Scalyr Agent uploads the log from the beginning.
d. Set max_existing_log_offset_size to 200000000. This is the default amount of caching (in bytes) performed by the Scalyr Agent.
e. (Optional) Adjust the send rate max_send_rate_enforcement to accommodate concurrent network traffic, ex. 1.5MB/s
f. Confirm that the log file is assigned the correct parser and the file path is valid. This will most likely be copied from the original Scalyr Agent configuration.
g. Confirm the API key and other essential settings. An example is provided below:

{
  ...
  "api_key": "XXX",
  ...
  "server_attributes": {
    serverHost: "hostname123-reup",
    ...
  }
  // No need for metric monitoring for purposes of this host
  implicit_metric_monitor: false,
  implicit_agent_process_metrics_monitor: false,
  ...
  disable_max_send_rate_enforcement_overrides: true,
  max_log_offset_size: 5368709120, // (5 * 1024^3)
  max_existing_log_offset_size: 200000000,
  max_send_rate_enforcement: "1.5MB/s",
  ...
}

g. More information on the max_log_offset_size and max_send_rate_enforcement parameters can be found here: Agent settings to allow older logs to be ingested

5. Run the Scalyr Agent. When the Scalyr Agent first starts, check the /var/log/scalyr-agent-2/agent.log file and confirm that the values from step (4) are being used. For example,

...
max_send_rate_enforcement: 1.5MB/s
disable_max_send_rate_enforcement_overrides: True
...
max_log_offset_size: 5368709120
max_existing_log_offset_size: 200000000
...

6. Check the associated DataSet account by searching for the appropriate serverHost + logfile values to verify that logs are being uploaded to the correct destination. Logs will be available once uploaded by the Agent.