This article explains how to configure a Scalyr Agent instance to upload logs from a log file when the log file contains a significant (> 200 MB) of data from the past.
1. If possible, establish a separate host to perform the upload of these log files. This minimizes the likelihood of any interference with an Agent that is uploading current logs.
2. Extract any compressed log data. Scan the start / end dates within each file to confirm there's no overlap. Prune any log data that is older than your account's retention period. For example, if you have a 30 day retention period, any log events that are older than 30 days won't be stored when uploaded to your account (if a parser is configured to recognize their timestamps).
Note: If there is no timestamp or the log's timestamp isn't recognized, the associated log events will be associated with the ingestion time, which may (or may not) be desirable.
3. Use the cat command to concatenate the uncompressed (rotated) log files into a single file, preferably using the same path as the original
4. Install a Scalyr Agent (but do not run yet) on the host from step (1). Its configuration will be similar to the existing configuration used for the log file, with the following differences:
a. Set the
serverHost to an easily identifiable value, ex. "hostname123-reup"
c. Set the
max_log_offset_size to the maximum expected size (in bytes) of the log file from step (3) above, ex. if the log file is 5GB, then use a value of 5 * 1024^3 = 5368709120 This ensures that the Scalyr Agent uploads the log from the beginning.
max_existing_log_offset_size to 200000000. This is the default amount of caching (in bytes) performed by the Scalyr Agent.
e. (Optional) Adjust the send
rate max_send_rate_enforcement to accommodate concurrent network traffic, ex.
f. Confirm that the log file is assigned the correct parser and the file path is valid. This will most likely be copied from the original Scalyr Agent configuration.
g. Confirm the API key and other essential settings. An example is provided below:
// No need for metric monitoring for purposes of this host
max_log_offset_size: 5368709120, (5 * 1024^3)
g. More information on the
max_send_rate_enforcement parameters can be found here: Agent settings to allow older logs to be ingested
5. Run the Scalyr Agent. When the Scalyr Agent first starts, check the
/var/log/scalyr-agent-2/agent.log file and confirm that the values from step (4) are being used. For example,
6. Check the associated DataSet account by searching for the appropriate
logfile values to verify that logs are being uploaded to the correct destination. Logs will be available once uploaded by the Agent.