DataSet supports traditional Docker environments incredibly well using the Scalyr Docker Agent. The default way the Docker Agent collects logs is by utilizing a dedicated log aggregation container that collects logs streaming to Stdout. DataSet utilizes either the
Sysloglogging drivers with this method.
Docker, ECS using EC2, and K8s all use this method or a derivative thereof; however, when it comes to Fargate, we need to use a different method of ingestion. This will utilize our Serverless log subscriber and streamer functions in Lambda.
The following will demonstrate the instrumentation of Fargate logs within DataSet.
What we will need:
- AWS Account
What you will do:
- Configure your ECS Task Definitions
- Deploy DataSet serverless log aggregator in AWS
- Send logs to DataSet
Configure your ECS Task Definitions
The next portion of the tutorial is on how to configure ECS to send its logs to CloudWatch Logs. we need to configure the log driver as
awslogs. See full AWS docs regarding this technique here.
- Navigate to the ECS Console
- Navigate to Task Definitions
- Select a Task Definition Name
- Select a Task Definition Revision
- Edit JSON
- Configure the logConfiguration stanza
- Your Amazon ECS container instances also require
logs:PutLogEventspermission on the IAM role with which you launch your container instances.
Deploy DataSet serverless log aggregator in AWS and send logs to DataSet:
- Deploy the serverless logging functions using the Cloud Formation template provide here.
- Two Lambda functions should have populated
- Streamer Function
- Subscriber Function
- Streamer Function
- Navigate to the Streamer Function
- Select Add Trigger and use a CloudWatch Logs trigger
- From here we can subscribe to any LogGroup added to CloudWatch Logs.
- Choose the ECS LogGroup. In the case of my task definition, the LogGroup will be named
- Once you do that, you should have a new serverHost and logfile named