Skip to main content

Getting Started with AWS Lambda Logging

Joel Mora
  • Updated


This document is intended to help teach users how to implement DataSet's Log subscriber and streamer functions that take log events from Cloudwatch Logs and stream events to DataSet. You can view a full video tutorial here. 

Table of Contents

  1. Create Slack Bot
    1. Configure a sample set of services to monitor. 
    2. These three sample services are DynamoDB, API Gateway, and Lambda
  1. Configure Observational Visibility Data
    1. Configure Logging
    1. Configure Metrics
  1. Configure DataSet
    1. Configure Parsers
    1. Configure Dashboard
    1. Configure Alerts

Create Slack Bot 

This step will show you how to setup demo microservices to monitor. You can view the full tutorial here 

    Configure Serverless Logging

    1. Get API Key
    1. Deploy CloudFormation Template
    1. Subscribe to CloudWatch Log Groups
    1. Configure Log Groups

    Log Group

    { ".*": { "parser": "my_func_parser", "prefix_timestamp": true, "serverHost": "webinar"} }

    Configure Cloudwatch Metrics

    1. Create role or user 
    1. Configure monitor reference
    {
      type: "cloudwatch",
      region: "us-east-1",
      accessKey: "xxx",
      secretKey: "xxx",
      executionIntervalMinutes: 2,
      period: 1,
      metrics: [
        // lambda
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "Invocations"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "Duration"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "Errors"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "DeadLetterErrors"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "DestinationDeliveryFailures"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "Throttles"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "ProvisionedConcurrencyInvocations"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "ProvisionedConcurrencySpilloverInvocations"}
        {namespace: "AWS/Lambda", dimensions: {FunctionName: "webinar3"}, metric: "IteratorAge"}
        
				//api gateway
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "4XXError"}
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "5XXError"}
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "CacheHitCount"}
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "CacheMissCount"}
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "Count"}
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "IntegrationLatency"}
        {namespace: "AWS/ApiGateway", dimensions: {ApiName: "webinar3-API"}, metric: "Latency"}
        
				//ddb
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ConditionalCheckFailedRequests"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ConsumedReadCapacityUnits"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ConsumedWriteCapacityUnits"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ReadThrottleEvents"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ReturnedBytes"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ReturnedItemCount"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ReturnedRecordsCount"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "SuccessfulRequestLatency"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "SystemErrors"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "TimeToLiveDeletedItemCount"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "ThrottledRequests"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "TransactionConflict"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "UserErrors"}
        {namespace: "AWS/DynamoDB", dimensions: {TableName: "ProductCatalog"}, metric: "WriteThrottleEvents"}
      ]
    }

     

    Configure DataSet

    1. Create parser

    {
  // specify a time zone if the timestamps in your log are not in GMT
  // timezone: "GMT-0800"
  
  formats: [
    {
      format: "$timestamp=number$ "
    },
    {
      format: ".*$=json{parse=json}$.*",
      repeat: true
    }
    {
      format: ".*REPORT RequestId: $RequestId$	Duration: $duration$ ms	Billed Duration: $billed_duration$ ms	Memory Size: $mem_size$ MB	Max Memory Used: $max_mem_used$ MB",
      repeat: true
    }
    {
    format: ".*RequestId\\s?:\\s?$RequestId$",
    }
    {
    format: ".*RequestId\\s?:\\s?$RequestId$ ",
    }
    
    
    

  ]
}

    2. create dashboard

    {
  // duration: "4h",
  
  graphs: [
    {
      columnSpan: 0,
      graphStyle: "line",
      plots: [
        {
          color: "#00BDD6",
          filter: "(\"REPORT\") ($serverHost = \"webinar12\") ($logfile = \"/aws/lambda/webinar12\")",
          label: "Rate"
        }
      ],
      rowSpan: 0,
      title: "Lambda Requests"
    },
    {
      columnSpan: 0,
      graphStyle: "line",
      plots: [
        {
          color: "#00BDD6",
          facet: "rate",
          filter: "('ERROR') ($serverHost = \"webinar12\") ($logfile = \"/aws/lambda/webinar12\")",
          label: "Rate"
        }
      ],
      rowSpan: 0,
      title: "Lambda Errors"
    },
    {
      label: "Lambda Duration",
      plots: [
        {
          facet: "duration",
          filter: "(\"REPORT\") ($serverHost = \"webinar12\") ($logfile = \"/aws/lambda/webinar12\")",
          label: "duration"
        }
      ]
    },
    {
      label: "API Gateway Requests",
      plots: [
        {
          filter: "(\"Extended Request Id\") ($serverHost = \"webinar12\")",
          label: "Rate"
        }
      ]
    },
    {
      label: "Api Gateway Errors",
      plots: [
        {
          facet: "value",
          filter: "($namespace == *  $namespace == \"AWS/ApiGateway\" $metric in (\"4XXError\",\"5XXError\")) ($serverHost = \"cloudwatch-us-east-1\") ($logfile = \"cloudwatch\")",
          label: "value"
        }
      ]
    },
    {
      label: "API Gateway Duration",
      plots: [
        {
          facet: "value",
          filter: "($namespace == *  $namespace == \"AWS/ApiGateway\"  $metric == \"Latency\") ($serverHost = \"cloudwatch-us-east-1\") ($logfile = \"cloudwatch\")",
          label: "value"
        }
      ]
    },
    {
      columnSpan: 0,
      graphStyle: "line",
      plots: [
        {
          color: "#00BDD6",
          filter: "((dynamo && $requestParametersTableName == \"ProductCatalog\") && $requestID == * \"DescribeTable\") ($serverHost = \"webinar12\") ($logfile = \"CloudTrail/Webinar12/DDB\")",
          label: "Rate"
        }
      ],
      rowSpan: 0,
      title: "DDB Requests"
    },
    {
      label: "DDB Errors",
      plots: [
        {
          facet: "value",
          filter: "( $namespace == \"AWS/DynamoDB\"  $metric contains \"error\") ($serverHost = \"cloudwatch-us-east-1\") ($logfile = \"cloudwatch\")",
          label: "value"
        }
      ]
    },
    {
      label: "DDB Duration",
      plots: [
        {
          facet: "value",
          filter: "($namespace == \"AWS/DynamoDB\"  $metric == \"SuccessfulRequestLatency\") ($serverHost = \"cloudwatch-us-east-1\") ($logfile = \"cloudwatch\")",
          label: "value"
        }
      ]
    },
    
  ],
  graphSize: "medium"
}

    3. Create alert

    		{
      alertAddress: "slack:webhookUrl=https://hooks.slack.com/services/xxx",
      description: "Lambda Invocation Error",
      trigger: "countPerSecond:1 minutes(('ERROR') ($serverHost = \"cloudwatch-630972250024\") ($logfile = \"/aws/lambda/webinar3\")) > 0",
    },
    {
      alertAddress: "xxx",
      description: "High latency",
      renotifyPeriodMinutes: 60,
      resolutionDelayMinutes: 5,
      trigger: "mean:1 minutes(duration where (duration > 600) ($serverHost = \"cloudwatch-630972250024\") ($logfile = \"/aws/lambda/webinar3\")) > 0"
    },
		{
      alertAddress: "slack:webhookUrl=https://hooks.slack.com/services/xxx",
      description: "High Amount of Requests",
      trigger: "countPerSecond:1 minutes(('REPORT') ($logfile = \"/aws/lambda/webinar3\")) > 10",
    },



    Supplemental Reading (configure role) 

     
    mceclip0.png

    External ID: "CWNYIRPOWR64HG73ZUTS5EXM3VHR3C5OUUE77X6IIQHWUYZUMPGQ===="

    AccountID: "913057016266"

     

    Related articles

    Comments

    0 comments

    Please sign in to leave a comment.