One of my user recently reported about 4h delay between the check execution (custom active python check) and the corresponding notification. Looking at the history i noticed daily notification at regular time (12:14pm) because of the check_interval = 24h. The point is that last daily check was executed at 7:30am in the morning. At this time the device was in Error (hard-state) and no notification was sent. At 11:00am the user fixed the error and at 12:14pm he received an Error notification although the problem was fixed for more than 1h.
Could you tell me what’s wrong here, i do not expect receving an error notification if the problem has been fixed?!
I was wondering if the notification reminder (HARD-state when check_interval is defined) can be decoupled from (not synchronized with) the check retry. I mean the notification treated in periodic cycle (24h in my case, at 12:14pm) independently from the check retry cycle (at 7:30am in my case)?
are there any notification delays that you have configured?
what does your notification rule look like?
include templates, apply rules, etc…
Something you might also look out for (it affected us recently, but wasn’t monitoring specific); are there any delays happening from your mailserver? Assuming that you are using mail for notifications. You can check this by lining up the notification timestamp in the icinga2 logs with the time that you actually receive the notification; even if it’s not mail specific, it still could be delayed on the receiving end (ie slack, pagerduty, etc…)