How do I search the logs I've archived to S3?

Q: Our S3 log export function exports log files in 5-minute blocks. How do I search this data?

A: If your search only pertains to a specific log file, you can use an S3 Select statement similar to:

aws s3api select-object-content --input-serialization '{"CompressionType": "GZIP", "JSON": {"Type": "LINES"}}' --output-serialization '{"JSON": {}}' --bucket s3-export-scalyr --key scalyr/2020-01-23T18-00-00Z.52.gz --expression-type SQL --expression "SELECT * FROM S3Object s WHERE s.serverHost IN ('web-7','web-5') AND CAST(s.\"timestamp\" AS timestamp) > TO_TIMESTAMP('2020-01-23T18:03:50Z') AND s.logfile = '/var/log/nginx/access.log'” output.txt

Given a bucket (s3-export-scalyr) and a key (scalyr/2020-01-23T18-00-00Z.52.gz), extract all log lines with:

• A logfile equal to /var/log/nginx/access.log (case insensitive)
• A serverHost value of ‘web-7’ or ‘web-5’, and (case insensitive)
• A timestamp value > 2020-01-23 18:03:50 (timestamp attribute is reserved keyword, so escaped) - omit this if you just want all matching lines in this file

to the output.txt file. 

A2: If you are looking across multiple keys/files, you could run the query multiple times and with different key names. Additionally, AWS Athena is another tool for doing more sophisticated queries against S3 files, including querying multiple s3 files at once.