Why I am not Getting a New Alert Notification for Every Matching Event

DataSet alert feature is not designed to send notifications for individual errors ("1 alert notification per 1 matching message"). It's intended to draw attention to a general circumstance ("hey there have been errors, you should log into DataSet to check that out").

Users can disable reminder alerts altogether by setting renotifyPeriodMinutes to 0 and yet that could result in missing a "new alert" notification on a matching event within alert triggered time interval. Here is a hypothetical scenario that could explain why you might miss alert notifications.

The timestamps of the triggered event

• 11:00:37
• 11:02:39
• 11:03:16
• 11:07:22 

Alert Definition

count:1m($logfile == '/var/log/accessLog' and $env == 'prod')

DataSet alert evaluation thread wakes up every minute, but the system determines the trigger condition (i.e. "new alert", "ongoing alert", or "no alert/alert reset") based on the defined alert interval rather than a fixed time period. 

Let's assume that DataSet alert evaluation thread wakes up at the 56th second of the minute.

The sequence of the alerts could be as followed:

1. 11:01:56 -> The system checks all the message in the previous minute (11:00:55 - 11:01:55 ) + padding of 30 secs before and after (11:00:25 - 11:02:25) -> MATCHING EVENT: 11:00:37 new alert
2. 11:02:56 -> Same logic as step 1 (11:01:25 - 11:03:25) -> MATCHING EVENT: 11:02:39 ongoing alert
3. 11:03:56 -> 11:02:25 - 11:04:25 -> MATCHING EVENT: 11:02:39, 11:03:16 ongoing alert
4. 11:04:56 -> 11:03:25 - 11:05:25 -> NO MATCHING EVENT: alert reset
5. 11:05:56 -> 11:04:25 - 11:06:25 -> NO MATCHING EVENT
6. 11:06:56 -> 11:05:25 - 11:07:25 -> NO MATCHING EVENT
7. 11:07:56 -> 11:06:25 - 11:08:25 -> MATCHING EVENT:  11:07:22 new alert
8. 11:08:56 -> 11:07:25 - 11:09:25 -> NO MATCHING EVENT: alert reset

Therefore, an user could've received 4 notifications but only get 2 due to renotifyPeriodMinutes is set to 0. 

Consequently, the number of alerts received from DataSet might be different from other monitoring platforms due to differences in implementation. We do have an open feature request to guarantee that the user gets the exact same number of alert notifications for every single matching event.