Custom alerts in Slack – DataSet Customer Portal

Introduction

The default Slack alert integration is easy to set up but does have some limitations. Namely, it cannot carry a payload. As an alternative, the webhook plugin can be used to send alert messages to Slack with a custom JSON body

Configuration

There are three primary differences between the default Slack integration and using webhooks to send Slack notifications:

The default Slack integration handles the trigger and resolution messages. However, when using the webhook configuration, these states will need to be manually defined.
The webhook allows users to customize the content / appearance of Slack notifications by adding / removing elements, changing colors, and streamlining the associated messaging.
Webhooks require a bit more initial effort to set up, as the JSON formatting is more complex than the default Slack integration.

Substitution Tokens

These dynamic values can be used as payload in the webhook-trigger. More info can be found here: https://app.scalyr.com/help/alerts#webhook

#trigger# - The alert's trigger
#description# - The name associated with alert.
#title# - First line of alert description
#link# - Link to alert on DataSet platform
#id# - DataSet-specific alert identifier (ex.: scalyr-alert-RiTIF6OxN5GYcGD9)
#lastLogLines# (currently there is only support for webhook-trigger)

Any values (for example, #host#, or an extracted attribute defined in the fields parameter) that are configured within an alert template will also be accessible. See https://app.scalyr.com/help/alerts#alertTemplatesfor more information.

Slack Message Body

Slack message body configuration can be found here https://api.slack.com/incoming-webhooks
We recommend removing formatting characters (newlines, spaces and tabs) from your JSON for use within the DataSet platform
Make sure to escape your characters (for example, quotation marks: \"). A search and replace works pretty well.
When updating your alert configuration, visit the "Alerts" page, then click "Settings"->"Alerts JSON" to paste your escaped JSON string into the configuration file directly
It is also possible to mention specific users by enclosing the Slack user ID within brackets and preceding it with an ampersand (ex. <@Slack_user_id>) within the text field, however, you will need to look up the user ID first

Example 1

Here is an example of a Slack POST within the alert address field.

Minified

// Alert if there is more that 120gb ingested in a 24 hour period - change 120 as needed 
{ 
trigger: "sumPerSecond:60s(tag='logVolume' metric='logBytes') > 1", 
alertAddress: "webhook-trigger:POST https://hooks.slack.com/services/APIKEY[[{\"text\":\"Test API Call\",\"attachments\":[{\"text\":\"Test API call\",\"fallback\":\"test\",\"callback_id\":\"wopr_game\",\"color\":\"#3AA3E3\",\"attachment_type\":\"default\",\"actions\":[{\"name\":\"test\",\"text\":\"Button A\",\"type\":\"button\",\"url\":\"#link#\"},{\"name\":\"test\",\"text\":\"Button C\",\"type\":\"button\",\"value\":\"maze\"},{\"name\":\"test\",\"text\":\"View Details\",\"style\":\"danger\",\"type\":\"button\",\"value\":\"test\",\"confirm\":{\"title\":\"Are you sure?\",\"text\":\"#description# on #host#\",\"ok_text\":\"Ok\",}}]}]}]]", description: "Slack with a payload" 
}

Expanded

// Alert if there is more that 120gb ingested in a 24 hour period - change 120 as needed 
{ 
trigger: "sumPerSecond:60s(tag='logVolume' metric='logBytes') > 1", 
alertAddress: "webhook-trigger:POST https://hooks.slack.com/services/APIKEY[[
{
\"text\":\"Test API Call\",
    \"attachments\":[
            {
                \"text\":\"Test API call\",
                \"fallback\":\"test\",
                \"callback_id\":\"wopr_game\",
                \"color\":\"#3AA3E3\",
                \"attachment_type\":\"default\",
                \"actions\":[
                       {
                          \"name\":\"test\",
                          \"text\":\"Button A\",
                          \"type\":\"button\",
                          \"url\":\"#link#\"
                       },
                       {
                          \"name\":\"test\",
                          \"text\":\"Button C\",
                          \"type\":\"button\",
                          \"value\":\"maze\"
                       },
                       {
                          \"name\":\"test\",
                          \"text\":\"View Details\",
                          \"style\":\"danger\",
                          \"type\":\"button\",
                          \"value\":\"test\",
                          \"confirm\":
                                {
                                   \"title\":\"Are you sure?\",
                                   \"text\":\"#description# on #host#\",
                                   \"ok_text\":\"Ok\",
                                }
                       }
                  ]
            }
        ]
    }
  ]
]", 
description: "Slack with a payload" 
}

Message

Example 2

A customer requested a more streamlined alert format than the default Slack integration.

We removed the "New DataSet Alert" title and trigger fields
Kept the "Latest Logs" field

Set a button to the alert URL as follows:

... 
alertAddress: "webhook-trigger:POST https://hooks.slack.com/services/APIKEY[[{\"text\":\"Test API Call\",\"attachments\":[{\"text\":\"Test API call\",\"fallback\":\"test\",\"callback_id\":\"wopr_game\",\"color\":\"#3AA3E3\",\"attachment_type\":\"default\",\"text\":\"Latest Logs:```#lastLogLines#```\",\"actions\":[{\"name\":\"test\",\"text\":\"Button A\",\"type\":\"button\",\"url\":\"#link#\"},{\"name\":\"test\",\"text\":\"Button C\",\"type\":\"button\",\"value\":\"maze\"},{\"name\":\"test\",\"text\":\"View Details\",\"style\":\"danger\",\"type\":\"button\",\"value\":\"test\",\"confirm\":{\"title\":\"Are you sure?\",\"text\":\"#description# on #host#\",\"ok_text\":\"Ok\",}}]}]}]]",
description: "Test Alert - logLines",
...

Note: The #lastLogLines# token won't be displayed if you are using the boolean value of true for the trigger (as no logs were actually processed)