Q: Our S3 log export function exports log files in 5-minute blocks. How do I search this data?
A: If your search only pertains to a specific log file, you can use an S3 Select statement similar to:
aws s3api select-object-content --input-serialization '{"CompressionType": "GZIP", "JSON": {"Type": "LINES"}}' --output-serialization '{"JSON": {}}' --bucket s3-export-scalyr --key scalyr/2020-01-23T18-00-00Z.52.gz --expression-type SQL --expression "SELECT * FROM S3Object s WHERE s.serverHost IN ('web-7','web-5') AND CAST(s.\"timestamp\" AS timestamp) > TO_TIMESTAMP('2020-01-23T18:03:50Z') AND s.logfile = '/var/log/nginx/access.log'” output.txt
Given a bucket (s3-export-scalyr) and a key (scalyr/2020-01-23T18-00-00Z.
- A
logfile
equal to /var/log/nginx/access.log (case insensitive) - A
serverHost
value of ‘web-7’ or ‘web-5’, and (case insensitive) - A
timestamp
value > 2020-01-23 18:03:50 (timestamp attribute is reserved keyword, so escaped) - omit this if you just want all matching lines in this file
to the output.txt file.
A2: If you are looking across multiple keys/files, you could run the query multiple times and with different key names. Additionally, AWS Athena is another tool for doing more sophisticated queries against S3 files, including querying multiple s3 files at once.
Comments
0 comments
Please sign in to leave a comment.