This article describes how to effectively configure per-server alerts with the
Alert Templates -
Alert templates are an advanced configuration that simplifies the process of creating alerts for multiple servers with common traits. This is accomplished by:
- Identifying servers on your platform that match a desired criteria via the
- Retrieving attributes from the associated log lines with the
filter: "", // Blank means "all hosts"
fields: ["serverHost"], // Retrieve the "serverHost" (hostname) field for use in alert templates.
// (You can specify "serverHost", "serverIP", and/or any server-level fields
// defined in the Scalyr Agent configuration.)
maxAgeHours: 4 // Ignore hosts which have not sent any data in the last 4 hours
description: "#serverHost#: high CPU usage",
trigger: "mean:5m(source='tsdb' serverHost='#serverHost#' metric='proc.stat.cpu_rate' type='user') > 400.0"
description: "#serverHost#: root disk almost full"
trigger: "mean:10m(source='tsdb' metric='df.1kblocks.free' serverHost == '#serverHost#' mount=='/' ) < 500000"
In both of these cases, server (aka server-level) attributes are supplied to the
What are server attributes?
It's important to understand the distinction between user and server attributes.
User attributes (displayed in the top half of the panel) include attributes like
version and are extracted by the parser associated with the file where log events originate. There's no guarantee that user attributes will be available for each log event, as the log event must first match a formatting rule before extraction can occur. Even when this occurs, there's no guarantee that the attributes will be the same (or similar). Therefore, user events aren't well suited for the type of processing that the
byHosts statement requires.
Server attributes are assigned by the Scalyr Agent automatically (in the case of
serverIP, etc.), or manually (by way of a user defined
attributes declaration in the Scalyr Agent configuration). Since server attributes are consistently available for every log event that originates from a particular server, they are utilized to process log events and essential alert variables (the
IMPORTANT! If user-level attributes are specified in the
fields parameters of a
byHosts clause, no error will be returned. However, the alert will not function as expected.
Server attributes can be readily identified by clicking any log event in the search UI. For example:
When using the
byHosts statement, we recommend using the
filter statement to ensure that the resultant alerts only apply to the servers you are monitoring. The
filter statement will process server attributes identically to a search query, and (if necessary) additional server attributes can be defined within the Scalyr Agent's configuration file. Consequently, the relevance of alerts created by
byHosts can improved with a nominal amount of fine tuning.
Values that are extracted by the
fields parameter can be used within the context of the
alerts block by surrounding the attribute name with hash marks (#).