Importing CloudWatch Logs and Metrics

Introduction

This article expands upon our CloudWatch integration documentation, and provides additional information about the process. It was written from the perspective of a user who has not begun installing the CloudFormation template (see URL provided on the CloudWatch documentation page), however, these instructions can be implemented at any time.

How it Works

Two Lambda-based functions (CloudWatchStreamerFunction and CloudSubscriberFunction) are implemented by the CloudFormation template. The CloudWatchSubscriber function automatically subscribes log groups that meet the criteria defined within the LogGroupOptions parameter. When logs appear within the associated log group, the CloudWatchStreamerFunction is triggered and uploads it to DataSet. No configuration of these functions is required. Settings are provided via the parameters above.

Installation

Lambda Scripts

For the purposes of this article, we created two Lambda-based Python scripts which output to the /aws/lambda/appKarateFunction and /aws/lambda/appTest1 log groups. These scripts (and the steps associated with their configuration) are entirely optional and are provided for testing purposes.

/aws/lambda/appKarateFunction

/aws/lambda/appTest1

These scripts are effectively the same and create differentiated output that we'll use to manually confirm the workings of the CloudWatch import functions

Setup

1. From AWS, visit the Lambda page and click "Create Function"
2. Enter appTest1 and appKarateFunction as the function names and choose Python 3.8 as the runtime
3. Choose "Create a new role with basic Lambda permissions"
4. Click "Create Function" and paste the Python scripts into the respective Lambda functions
5. Click the "Test" button. On the "Configure test event" dialog, select "Create new test event" and Choose the "hello-world" event template
6. Use appTest1Event and appKarateTestEvent (respectively) as the event names and click "Save."
7. Click the "Deploy" button to deploy the script

Parameters

Follow steps 1-2 of the CloudWatch integration documentation and run the CloudFormation template. The parameters / values below are applied to step 3:

• Set AutoSubscribeLogGroups to true (it is false by default)
• For WriteLogsKey, input a DataSet API key with write access (obtain by clicking the User Menu -> "API Keys")
• The LogGroupOptions parameter accepts regular expressions for the log group parameter. Consequently, one field can apply to multiple log groups (Note: these parameters are limited in length by AWS, so allows a more compact configuration). For our example, use:

{
  "/aws/lambda/app[KT].*": { "serverHost": "cloudwatch-test", "parser": "blah"}
}

This statement matches log groups:

• • Which are associated with /aws/lambda and begin with app
  • And are followed by either a capital K or T
  • And are succeeded by 0 or more characters
  • Furthermore, each log group that matches this criteria will be assigned the serverHost value "cloudwatch-test" and will be parsed by the "blah" parser
  • Log groups that were previously subscribed but no longer match the LogGroupOptions field will automatically be unsubscribed

Important

Please pay close attention to the JSON formatting when updating the LogGroupOptions parameter. Minor format issues can cause the CloudFormation deployment to fail, and the resulting errors aren't the most straightforward.

Deploy the CloudFormation Template

Once you have updated the settings, complete the installation of the CloudFormation template (this may take up to 5-10 minutes).

Testing

Now that the CloudFormation template has been applied, we can confirm that the logging features work as expected.

1. Access the appTest1 and appKarateFunction Lambda functions
2. Click the "Test" button. The script's output will be displayed in the "Execution Result" sub-window of the UI.
3. From the "Services" menu, choose "CloudWatch." There should be 2 corresponding log groups in CloudWatch for these Lambda functions, with the Python output from step 2 should be displayed. Similarly, these events should also be available via your DataSet account. For example,

/aws/lambda/appTest1

/aws/lambda/appKarateFunction

DataSet

If you click on an individual log line, the "Inspect Log Line" dialog will be displayed. In the "Server Fields" section, the serverHost and parser fields match the LogGroupOptions parameter that we previously configured

CloudWatch Metrics

Follow the steps for creating an IAM role here: https://app.scalyr.com/solutions/import-cloudwatch#createIAMRole

Then, copy the ARN of the IAM role to your DataSet monitors configuration. In this example, I'm sending CPU / network metrics from an EC2 instance:

DataSet downloads CloudWatch metrics directly, and doesn't rely upon the CloudFormation template to do so. You can check on the status of this process by running the following query:

tag='cloudwatchMonitor' and valuesFetched > 0