Including (and Excluding) Log Files

Introduction

The Scalyr Agent makes it easy to upload by using conventional wildcards (globs) in the log section of the configuration file (either /etc/scalyr-agent-2/agent.json or /etc/scalyr-agent-2/agent.d/*.json). Granular control is achieved by using the exclude statement, which accepts regular expressions. This article explains how to configure both.

Including Files

To include files with a wildcard (glob), simply add an entry to the logs section of the Scalyr Agent's configuration file. For example,

...
  logs: [
    {
      path: "/var/log/app/*.log",
      attributes: {parser: "appLog"}
    }
  ]
...

• The Scalyr Agent will upload log files that match the glob pattern
• As new log files are created, they are uploaded
• The glob should be as specific as possible
• Previous log files will only be updated (and not completely re-uploaded) if additional logs events are appended. The Scalyr Agent uses checkpoints to ensure that it does not re-upload log events (or entire log files) that have already been processed.

Excluding Files

You can add an exclude statement to the above entry to exclude files that match a regular expression:

...
    { 
      path: "/var/log/app/*.log",
      exclude: ["*.log.[0-9]+$"],
      attributes: {parser: "appLog"}
    },
...

Log files ending with a numerical suffix suffix (ex. "/var/log/app/logging_test-005.log.1" or "logging_test-006.log.2", not including the double quotes) will not be uploaded.

Confirm Your Changes

Once you have updated the logs section, save the configuration file and run

sudo scalyr-agent-2 status -v

For this particular test case, I created several log files with different numerical suffixes in the filename to test that everything was working: 

Glob: /var/log/app/*.log:: last scanned for glob matches at Thu May 13 20:57:10 2021 UTC
/var/log/app/jt_test-000.log: copied 0 bytes (0 lines), 0 bytes pending, last checked Thu May 13 20:57:40 2021 UTC
/var/log/app/jt_test-001.log: copied 0 bytes (0 lines), 0 bytes pending, last checked Thu May 13 20:57:40 2021 UTC
/var/log/app/jt_test-003.log: copied 0 bytes (0 lines), 0 bytes pending, last checked Thu May 13 20:57:40 2021 UTC
/var/log/app/jt_test-004.log: copied 0 bytes (0 lines), 0 bytes pending, last checked Thu May 13 20:57:40 2021 UTC
Glob: /var/log/scalyr-agent-2/agent-worker-session-*.log:: last scanned for glob matches at Thu May 13 20:57:10 2021 UTC