Enable / Disable / Modify Alerts with DataSet API

Introduction

A customer recently requested a way to enable or disable particular alerts via an API call. By using the getFile and putFile commands, a customer can load their existing alert configuration, modify it, and re-upload. Our alerts framework will maintain the state of existing alerts while instantaneously refreshing any alerts that were changed.

If you aren't familiar with the DataSet Alerts JSON configuration file, we have a video explanation here: https://youtu.be/s5XhogirGig?t=136

Process

The getFile and putFile API calls are explained in detail here:

• https://app.scalyr.com/help/api#getFile
• https://app.scalyr.com/help/api#putFile

Note: Be sure to use the Configuration Access keys (and not the Log Access keys) for the token parameter. A token with configuration "write" access is needed. You can retrieve these keys by clicking the User Menu -> "API Keys" and scrolling to the bottom of this page, as shown here:

To retrieve the alert config, issue a getFile call as follows. The alert configuration contents are returned in JSON format via the content parameter, with escaped newlines:

{
"path": "/scalyr/alerts",
"modDate": 1600988587827,
"stalenessSlop": 5000,
"version": 46639831151,
"content": "{\n alerts: [\n {\n alertAddress: \"webhook-trigger:POST https://outlook.office.com/webhook/000000d0-00c0-0e00-00dd-af0000f00000@0f0000f0-d000-00c0-0a00-f000000a0fbd/IncomingWebhook/0bc000a0000000e00a000db0000e0fe0/af0a0dbc-c000-0000-a000-0000000f0e0c[[{\\\"@type\\\":\\\"MessageCard\\\",\\\"@context\\\":\\\"http://schema.org/extensions\\\",\\\"themeColor\\\":\\\"0076D7\\\",\\\"summary\\\":\\\"Scalyr Alert\\\",\\\"sections\\\":[{\\\"activityTitle\\\":\\\"#title#\\\",\\\"activitySubtitle\\\":\\\"#trigger#\\\",\\\"facts\\\":[{\\\"name\\\":\\\"View Alert\\\",\\\"value\\\":\\\"#link#\\\"}],\\\"markdown\\\":true}]}]]\",\n description: \"Test Alert\",\n gracePeriodMinutes: 0,\n renotifyPeriodMinutes: 1,\n resolutionDelayMinutes: 5,\n trigger: \"sumPerSecond:5 minutes($tag='logVolume' $metric='logBytes') < 1024\"\n }, {\n alertAddress: \"testing@scalyr.com\",\n description: \"Constantly triggered alert\",\n gracePeriodMinutes: 0,\n renotifyPeriodMinutes: 2,\n resolutionDelayMinutes: 5,\n trigger: \"count:5 minutes($tag='ingestionSuccess' $serverHost = \\\"scalyr-metalog\\\") > 1\"\n }\n ]\n}",
"status": "success",
"createDate": 1569264251136
}

Next, modify the alert section(s) that you wish to change. You can either do this programmatically, or by switching between the different configurations as needed.

We disable the two alerts by commenting out 2 sections and use the putFile call to submit it

{
"token": "... Redacted ...",
"path": "/scalyr/alerts",
"content":"{\n alerts: [\n /*{\n alertAddress: \"webhook-trigger:POST https://outlook.office.com/webhook/000000d0-00c0-0e00-00dd-af0000f00000@0f0000f0-d000-00c0-0a00-f000000a0fbd/IncomingWebhook/0bc000a0000000e00a000db0000e0fe0/af0a0dbc-c000-0000-a000-0000000f0e0c[[{\\\"@type\\\":\\\"MessageCard\\\",\\\"@context\\\":\\\"http://schema.org/extensions\\\",\\\"themeColor\\\":\\\"0076D7\\\",\\\"summary\\\":\\\"Scalyr Alert\\\",\\\"sections\\\":[{\\\"activityTitle\\\":\\\"#title#\\\",\\\"activitySubtitle\\\":\\\"#trigger#\\\",\\\"facts\\\":[{\\\"name\\\":\\\"View Alert\\\",\\\"value\\\":\\\"#link#\\\"}],\\\"markdown\\\":true}]}]]\",\n description: \"Test Alert\",\n gracePeriodMinutes: 0,\n renotifyPeriodMinutes: 1,\n resolutionDelayMinutes: 5,\n trigger: \"sumPerSecond:5 minutes($tag='logVolume' $metric='logBytes') < 1024\"\n }, {\n alertAddress: \"testing@scalyr.com\",\n description: \"Constantly triggered alert\",\n gracePeriodMinutes: 0,\n renotifyPeriodMinutes: 2,\n resolutionDelayMinutes: 5,\n trigger: \"count:5 minutes($tag='ingestionSuccess' $serverHost = \\\"scalyr-metalog\\\") > 1\"\n }*/\n ]\n}"
}

The alert configuration is immediately updated in DataSet:

Any alerts that were not modified will retain their states, while modified alerts will be automatically re-initialized. Please note that you may receive notifications that were in the pipeline prior to the alerts being updated.