Using the byHosts Alert Template

Introduction

This article describes how to effectively configure per-server alerts with the byHosts clause

Alert Templates - byHosts

Alert templates are an advanced configuration that simplifies the process of creating alerts for multiple servers with common traits. This is accomplished by:

1. Identifying servers on your platform that match a desired criteria via the filter parameter, and
2. Retrieving attributes from the associated log lines with the fields parameter 

{
  alerts: [
    {
      byHosts: {
        filter: "", // Blank means "all hosts"
        fields: ["serverHost"], // Retrieve the "serverHost" (hostname) field for use in alert templates.
                                // (You can specify "serverHost", "serverIP", and/or any server-level fields
                                // defined in the Scalyr Agent configuration.)
        maxAgeHours: 4 // Ignore hosts which have not sent any data in the last 4 hours
      },
      alerts: [
        {
          description: "#serverHost#: high CPU usage",
          trigger: "mean:5m(source='tsdb' serverHost='#serverHost#' metric='proc.stat.cpu_rate' type='user') > 400.0"
        }, {
          description: "#serverHost#: root disk almost full"
          trigger: "mean:10m(source='tsdb' metric='df.1kblocks.free' serverHost == '#serverHost#' mount=='/' ) < 500000"
        }
      ]
    }
  ]
}

In both of these cases, server (aka server-level) attributes are supplied to the filter and fields parameters.

What are server attributes?

It's important to understand the distinction between user and server attributes.

User Attributes

User attributes (displayed in the top half of the panel) include attributes like timestamp, image, and version and are extracted by the parser associated with the file where log events originate. There's no guarantee that user attributes will be available for each log event, as the log event must first match a formatting rule before extraction can occur. Even when this occurs, there's no guarantee that the attributes will be the same (or similar). Therefore, user events aren't well suited for the type of processing that the byHosts statement requires.

Server Attributes

Server attributes are assigned by the Scalyr Agent automatically (in the case of session, serverHost, serverIP, etc.), or manually (by way of a user defined attributes declaration in the Scalyr Agent configuration). Since server attributes are consistently available for every log event that originates from a particular server, they are utilized to process log events and essential alert variables (the filter and fields parameters). 

IMPORTANT! If user-level attributes are specified in the filter or fields parameters of a byHosts clause, no error will be returned. However, the alert will not function as expected.

Identification

Server attributes can be readily identified by clicking any log event in the search UI. For example:

Best Practices

When using the byHosts statement, we recommend using the filter statement to ensure that the resultant alerts only apply to the servers you are monitoring. The filter statement will process server attributes identically to a search query, and (if necessary) additional server attributes can be defined within the Scalyr Agent's configuration file. Consequently, the relevance of alerts created by byHosts can improved with a nominal amount of fine tuning.

Values that are extracted by the fields parameter can be used within the context of the alerts block by surrounding the attribute name with hash marks (#).

For example, #serverHost# or #serverIp#