S3 Archive Ingests Empty Files

Q: When doing an export to S3 we are getting a huge amount of archives in the s3 bucket that are empty? Just 20 bytes… Why is the export creating empty archives?  

A: The empty files are deposited in the destination S3 bucket if no log events occurred during the window represented by the file. The intent to generate those empty files by default is to ensure that the run did (or didn't) complete. You could turn it off by adding a parameter omitIfEmpty to true (default false) to the logArchiveRules.

For example:

{
      match: "$logfile = '/var/log/accessLog'",
      includeParsedFields: false,
      omitIfEmpty: true,
      destination: {
        type: "s3",
        ....
}

In addition to the files named .40.gz, you could see empty files with a suffix of  10040 and 20040. Those serial numbers indicate to us that stale log ingestion was in effect on your account, meaning that you stale-ingested some events that don't match the archive rules. Setting omitIfEmpty would also prevent empty archives to be uploaded.