Deadman switch via notifications?

Icinga 2
1

Hi all,

We recently had an issue where our notifications locked up and weren’t sending - we identified a likely root cause and it has been addressed, however I was wondering if there is a way to send a dummy notification using the API?

We would like to setup a deadman switch using Opsgenie Heartbeats - essentially I am imagining a dummy service, which is constantly alerting, and sends this to the Opsgenie Heartbeat API. If no alert is received within X minutes, an alert in Opsgenie should be raised (indicating that Icinga has locked up).

Could this be achieved with a custom notification via the API?

EDIT: I found this after posting - Monitoring Nagios using Heartbeats | Opsgenie | Atlassian Support - this mostly achieves what we need, however we would still want this check to fire a notification on every check execution - is it possible to do so?

2

We are using a similar approach and have created a service that is scheduled periodically like every other services. However, it does not run a check plugin but the notification script. We’ve called this heartbeat. Our notification script returns critical if it failed to send a message.

1 Like
3

You could use the dummy’s freshness timing to switch from OK to CRITICAL on every API call and thus force a sending a notification.

1 Like