Configuring Custom Alerts

Introduction

This article explains how to configure alerts from log data. It assumes that you have already installed the Scalyr Agent on a host and are able to upload log data to DataSet.

Configure the Scalyr Agent

Using the Scalyr Agent's configuration file (default of /etc/scalyr-agent-2/agent.json, or may be set by a user-defined auxiliary file in /etc/scalyr-agent-2/agent.d/*.json), set the parser (and any other attributes as needed) for the log file. This could be something as simple as:

...  
  logs: [ 
    { path: "/var/log/my_log.log", attributes: {parser: "myLog"} },
    ...
  ]
...

Where myLog is the parser that is associated with the my_log.log file.

Set up the Parser

In this example, we want to trigger an alert from the amount of cache used by a particular user. Given the following logs:

2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=US, id=1234567890, foo=YmxhaGJsYWhibGFoCg, username=jt} : Current cache used is : 100
2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=NL, id=9876543210, bar=a2FyYXRlY2hvcAo, username=joop} : Current cache used is : 251
2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=US, id=5555555555, baz=Zm9vYmFyZm9vCg, username=raoul} : Current cache used is : 10

The parser (below) will extract the timestamp, severity, key-value pairs and the cache fields

Parser

{
  patterns: {
    tsPattern: "\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}",
    genPattern: "[a-zA-Z0-9\\.\\-_]+"
  }
  formats: [
    {
      format: "$timestamp=tsPattern$ $severity=identifier$ [:]{3} $app=genPattern$ .*"
    },
    {
      format: ".*$_=genPattern$=$_=genPattern$.*",
      repeat: true
    },
    {
      format: ".* : Current cache used is : $cache=number$$$"
      attributes: { type : "cacheEval" }
    }
  ]
}

Output

2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=US, id=1234567890, foo=YmxhaGJsYWhibGFoCg, username=jt} : Current cache used is : 100
cache: 100
countryCode: US
foo: YmxhaGJsYWhibGFoCg
id: 1234567890
message: 2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=US, id=1234567890, foo=YmxhaGJsYWhibGFoCg, username=jt} : Current cache used is : 100
type: session
username: jt

2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=NL, id=9876543210, bar=a2FyYXRlY2hvcAo, username=joop} : Current cache used is : 251
bar: a2FyYXRlY2hvcAo
cache: 251
countryCode: NL
id: 9876543210
message: 2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=NL, id=9876543210, bar=a2FyYXRlY2hvcAo, username=joop} : Current cache used is : 251
type: session
username: joop

2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=US, id=5555555555, baz=Zm9vYmFyZm9vCg, username=raoul} : Current cache used is : 10
baz: Zm9vYmFyZm9vCg
cache: 10
countryCode: US
id: 5555555555
message: 2021-05-11 19:47:08.407 INFO RandomInfo ::: {countryCode=US, id=5555555555, baz=Zm9vYmFyZm9vCg, username=raoul} : Current cache used is : 10
type: cacheEval
username: raoul

Notes

• An attribute (type: cacheEval) is assigned to these log lines by the parser -- this is important as it will be used in the alert. This was included to demonstrate how to quickly isolate log events that you wish to use as an alert trigger; however, a well-defined search query would also work.
• Whenever possible, we use a defined regular expression pattern (ex. tsPattern, genPattern, number) when applying the parser to these fields. This ensures that the parser only extracts the values we anticipate. Some of these patterns (ex. number) are predefined for your convenience.
• The cache value is extracted from each log event, this is an essential part of the alert

Configure the Alert

Now that we have extracted the attribute (cache) we wish to alert from, the alert configuration is relatively straightforward. Click the "Alerts" menu option -> "Settings" -> "Alerts JSON" and add the alert entry:

...
  {    
    alertAddress: "your_email@email.com",
    description: "cache > 5",
    gracePeriodMinutes: 0, # Trigger alert immediately
    renotifyPeriodMinutes: 0, # No reminders
    resolutionDelayMinutes: 9999, # Disable resolution notification
    trigger: "mean:10m(cache where type='cacheEval') > 5"
  }
...

Notes

1. mean is likely the best function to use on these values, as it's not impacted by the time period
2. The trigger statement extracts the cache field from log events where type='cacheEval'. In order for the alert to work, it needs a valid condition (ex. > 5)

Conclusion

This article provided a complete overview of how to configure alerts from logs that are uploaded by the Scalyr Agent.