Azure EventHub Monitor

Introduction

Within Azure, one of the easiest ways to grab data from your Azure account is to utilize the EventHub Monitor. This solution is still in beta so be mindful and test before implementing it in large-scale production systems. 

Limits

DataSet has a single event size limit of 100kb. Some Azure services have events that exceed this.  

Prereqs

Existing Azure Account with ability to configure diagnostic logs and create EventHubs 

Existing DataSet Account with Full access

Solution

This guide illustrates how to get Azure Logs for a Service with Diagnostics into DataSet. This solution uses AAD but is not limited to that service.

Step 1: Create EventHub

The first thing we will do will be to create a new Event Hub namespace and in that namespace create an Event Hub. 

1. Search for EventHubs in the Marketplace

2. Create EventHub Namespace

3. Click into the newly created namespace

4. Under Entities select Event Hubs

5. Select + Event Hub and create it 

Step 2: Configure EventHub

Now that we have our EventHub let's give DataSet permission to view these data.

1. Navigate to your EventHub namespace and under Settings select Shared Access Policies 2. Add new policy   3. Give it a name and configure the Listen permission

4. Copy Connection String Primary Key and save it. You will use this to configure DataSet.

Step 3: Configure DataSet

Now we will configure a DataSet Monitor. In the previous step you copied the Connection String Primary Key. You will configure the monitor with the data in this string. 

1. log in to an existing DataSet Account. If you don't have one. Sign up for a free trial. 

2. Select Monitors from the user menu:

3. Edit JSON

4. Add this snippet to configure EventHub

You will need two pieces of information. 
   a. EventHub Name 

   b. Your Connection String Primary Key will look like this

Endpoint=sb://{namespaceHostname}/;SharedAccessKeyName={sharedAccessKeyName};SharedAccessKey={sharedAccessKey}

{
  type: "eventhub",
  eventHubName: "{eventHubName}",
  namespaceHostname: "{namespaceHostname}.servicebus.windows.net",
  sharedAccessKey: "{sharedAccessKey}",
  sharedAccessKeyName: "{sharedAccessKeyName}",
  timeoutSeconds: 60.0
}

5. Modify the "eventhub" parser

You may wish to modify the "eventhub" parser to extract attributes from specific log events.

You can access the parser by either:

1. Clicking the "Search" option and searching for parser='eventhub'. Next, click an Event Hub log event. The "Inspect Log Line" dialog is displayed. Click the "Edit Parser" button:
   
   Or,
2. Once Event Hub logs are being ingested by DataSet, click the User Menu -> "Manage Logs" -> "Parsers":
   
   From here, select the "eventhub" parser by clicking the "Edit" button beside it:
   

To start, we recommend extracting the timestamp from log events, as this ensures that the timestamps assigned by your platform match the timestamps on DataSet. Please note that your logs may differ, and the following parser is intended to be a generic example:

{
  // specify a time zone if the timestamps in your log are not in GMT
  // timezone: "GMT-0800"
  formats: [
    {
      format: "$=json{parse=dottedJson}$" // Extract each log line into dotted JSON format
    },
    {
      format: ".*\"time\":", // if time attribute present
      rewrites: [
        {
          input: "time", // take value of time attribute and 
          output: "timestamp", // move it to the reserved timestamp parameter
          match: "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d+Z)", // match the ISO-8601 format
          replace: "$1" // replace value of timestamp with regex group
        }
      ] 
    },
    {
      format: ".*\"Level\":", // if Level attribute present
      rewrites: [
        {
        input: "Level", // take value of Level attribute and
        output: "severity", // transfer it to the reserved severity parameter
        match: "(\\d+)", // only match numerical values
        replace: "$1" // replace value of severity with regex group
        }
      ] 
    }
  ]
}

Prior to saving the parser, be sure to test it out by clicking the "Test Parser" button. Parsers only apply to logs that are ingested after the parser is saved. Parsers do not apply to log events from the past.

Step 4: Configure Azure Service

Now we will configure our app to send its logs to your new Event Hub which is being monitored by DataSet. In this case, we will use Active Directory Logs, but any service with Diagnostic Logs will work. 

1. Open your service

2. Scroll down to Monitoring and select Diagnostic settings

3. Add Diagnostic Settings

4. Choose Datasources: Select the logs you want to send from the list of available datasets.

5. Choose Destination and set it to EventHub

6. Select the Destination we configured in Section 1 Step 1