Preparation
Parsing
Before configuring alerts, confirm that the data you need from your log events is readily available. This is as simple as:
- Modifying parser(s) to extract values from log events of interest
- Assigning attributes / id values as necessary to quickly isolate log data via the alert
- Time zones are of particular significance. An incorrect time zone could result in false alarms, or alerts failing to trigger.
- First, Identify the time zone associated with the timestamps contained in your log data.
- Then, set the time zone associated with the log data manually within the parser, or extracting it from the log data (see: https://app.dataset.com/help/parsing-logs#specialAttrs).
- Verifying that your parser is correctly interpreting the timestamp associated with the logs
A combination of the above factors are typically the culprit when alerts don’t behave as expected.
Notifications
Q: Alerts are no longer sent to my email address, yet they previously were.
A: It’s possible that the email account rejected or bounced an incoming alert email. Once this happens, our email platform will refrain from sending further notifications. Please contact support@dataset.com to investigate, and whitelist alerts@dataset.com on your mail server from any possible spam recognition or rate limiting in the future.
Slack
Q: Are the fields provided with Slack alerts modifiable?
A: To an extent, yes. There are several built-in fields that can be used to modify the structure of Slack alerts. See the “Substitution Tokens” section of https://app.dataset.com/help/alerts. Also, you’ll want to set up your Slack alerts via webhook. The resultant alertAddress field is:
webhook-trigger:POST https://hooks.slack.com/services/XXX/YYY/ZZZ[[{\"text\": \"<@userId> #title# <#link#>\",\"username\": \"scalyr\"}]]
Triggers
Syntax
When specifying field names (for example, timestamp, parser, yourField) in alerts, the dollar sign ($, also known as a sigil) is no longer needed / recommended.
Q: I want to trigger an alert if a parsed attribute called queue_size reaches 5000 more than 10 times over a 24 hour period
A: The alert trigger would be something like:
count:24h(serverHost="HOST" metric="queue_size" value > 5000) > 10
Where metric and value are extracted from the log events by a parser
Alert Templates
Q: Is it possible to dynamically configure alerts for a static list of servers?
A: Yes! This can be done for a list of servers (or all servers). See: https://app.dataset.com/help/alerts#alertTemplates. Note: This requires a list of servers to be defined for the alert(s).
Grouping - Grouped Alerts
Q: Can grouped alerts be used to detect when a host / log file stops generating logs?
A: Our grouped alerts work well when a data source is actively generating logs. However, they won't be triggered if a value or count of events from a host / log file is no longer present.
PowerQuery Support
Q: Can PowerQueries be used to construct alerts?
A: Yes -- PowerQueries can be implemented directly into alerts or via a PowerQuery monitor. PowerQuery alerts will trigger based on the value contained within a 1x1 table (tableValue) or by the count of rows returned (tableRows). PowerQuery monitors periodically the results from a PowerQuery as log events, these logs can then be used to trigger alerts.
Whitelisting
IP Addresses
Q: Could you please provide a list of IP addresses where alert webhook notifications originate from?
A: Please see our HTTP Monitors FAQ. Note that the addresses are specific to where your account is based (US or EU).
Comments
0 comments
Article is closed for comments.