Delta thresholds

Service Monitoring
1

Hi.

Is there any way of setting a “delta threshold” for an alert, such that I don’t care what the actual value of the measurement at the time is, but I do care if it goes up by X points, (or maybe Y%), or in some cases, similarly down (depending on what my measurement means).

For example, I have a service check which counts the number of SIP devices currently registered and online to my Asterisk server.

If that number goes up, it just means I have some more devices (which probably means more customers) - that’s fine.

If that number goes down (and especially if it drops by some noticeable %age), then I want an alert.

I can’t set an absolute threshold for the alert, because one day I might have 100 devices online, the next day 110, and a week later, 150.

What I want to know is if the 100 goes down to 90, or the 150 goes down to 130, etc., so I have a threshold which is a delta of the current number, rather than an absolute value.

Anyone got an idea how to do this?

Thanks, Antony.

2

Hi,

if you use Grafana you could do this there. My colleague is playing with it at the moment. But he did it with simple SQL statements. As backend we are using Influx.

3

If you send your Icinga Perfdata into a Database (Influx) and visualize it in Grafana, then you can create alerts in Grafana, depending on a percent_diff. To visualize these alerts also in your Icinga environment, you need an icinga check, which checks if there exists an alert.
It’s a bit tricky, but it works…