Ingesting ECS Fargate Logs with DataSet

Introduction

DataSet supports traditional Docker environments incredibly well using the Scalyr Docker Agent. The default way the Docker Agent collects logs is by utilizing a dedicated log aggregation container that collects logs streaming to Stdout. DataSet utilizes either the json-file or Sysloglogging drivers with this method. 

Docker, ECS using EC2, and K8s all use this method or a derivative thereof; however, when it comes to Fargate, we need to use a different method of ingestion. This will utilize our Serverless log subscriber and streamer functions in Lambda.

The following will demonstrate the instrumentation of Fargate logs within DataSet. 

What we will need:

1. AWS Account

What you will do:

1. Configure your ECS Task Definitions
2. Deploy DataSet serverless log aggregator in AWS
3. Send logs to DataSet

Configure your ECS Task Definitions

The next portion of the tutorial is on how to configure ECS to send its logs to CloudWatch Logs. we need to configure the log driver asawslogs. See full AWS docs regarding this technique here. 

1. Navigate to the ECS Console 
2. Navigate to Task Definitions
3. Select a Task Definition Name 
4. Select a Task Definition Revision
5. Edit JSON
6. Configure the logConfiguration stanza

   •  "logConfiguration": {
     
           "logDriver": "awslogs",
     
           "secretOptions": null,
     
           "options": {
     
                "awslogs-group": "/ecs/first-run-task-definition",
     
                "awslogs-region": "us-east-1",
     
                "awslogs-stream-prefix": "ecs"
     
      }

7. Your Amazon ECS container instances also require logs:CreateLogStream and logs:PutLogEvents permission on the IAM role with which you launch your container instances.

Deploy DataSet serverless log aggregator in AWS and send logs to DataSet:

1. Deploy the serverless logging functions using the Cloud Formation template provide here.   

2. Give it a name, set your API Key (EU), and log group options (full instructions here)

3. Deploy template

4. Two Lambda functions should have populated 
   
   • Streamer Function
     <name>-CloudWatchStreamerFunction-<id>
   • Subscriber Function 
     <name>-CloudWatchSubscriberFunction-<id>
5. Navigate to the Streamer Function
6. Select Add Trigger and use a CloudWatch Logs trigger
   
7. From here we can subscribe to any LogGroup added to CloudWatch Logs. 
8.  Choose the ECS LogGroup. In the case of my task definition,  the LogGroup will be named  /ecs/first-run-task-definition
9. Once you do that, you should have a new serverHost and logfile named /ecs/first-run-task-definition