How do I search the logs I've archived to S3?

Q: Our S3 log export function exports log files in 5-minute blocks. How do I search this data?

A: If your search pertains to a specific log file, you can use an S3 Select statement similar to:

aws s3api select-object-content --input-serialization '{"CompressionType": "GZIP", "JSON": {"Type": "LINES"}}' --output-serialization '{"JSON": {}}' --bucket s3-export-scalyr --key scalyr/2020-01-23T18-00-00Z.52.gz --expression-type SQL --expression "SELECT * FROM S3Object s WHERE s.serverHost IN ('web-7','web-5') AND CAST(s.\"timestamp\" AS timestamp) > TO_TIMESTAMP('2020-01-23T18:03:50Z') AND s.logfile = '/var/log/nginx/access.log'" output.txt

Given a bucket (s3-export-scalyr) and a key (scalyr/2020-01-23T18-00-00Z.52.gz), extract all log lines with:

• A logfile equal to /var/log/nginx/access.log (case insensitive)
• A serverHost value of ‘web-7’ or ‘web-5’, and (case insensitive)
• A timestamp value > 2020-01-23 18:03:50 (timestamp attribute is a reserved keyword, so it's escaped) - omit this if you just want all matching lines in this file

Finally, output all results to the output.txt file

A2: If you are looking across multiple keys/files, you could run the query multiple times and with different key names. Or, AWS Athena could be used to perform sophisticated queries against multiple S3 files.