Is there any way of setting a “delta threshold” for an alert, such that I don’t care what the actual value of the measurement at the time is, but I do care if it goes up by X points, (or maybe Y%), or in some cases, similarly down (depending on what my measurement means).
For example, I have a service check which counts the number of SIP devices currently registered and online to my Asterisk server.
If that number goes up, it just means I have some more devices (which probably means more customers) - that’s fine.
If that number goes down (and especially if it drops by some noticeable %age), then I want an alert.
I can’t set an absolute threshold for the alert, because one day I might have 100 devices online, the next day 110, and a week later, 150.
What I want to know is if the 100 goes down to 90, or the 150 goes down to 130, etc., so I have a threshold which is a delta of the current number, rather than an absolute value.
Anyone got an idea how to do this?