Separate notification interval values for critical and warning states

Icinga 2
1

Hey all,

I’m looking for a way to implement different intervals for warning and critical service states so that it would for example be possible to have a mail notification every 2 hours for a warning state and every 30 minutes for a critical.

So far I’ve tried separate notifications for each state, which works except when the state gets from warning to critical and then back to Ok. In that case, two recovery mails are sent, which is plausible since I’ve defined two notifications, but it’s not entirely what I’m looking for.

Is there any way to set separate interval values in a notification for warning and critical states?

Thanks in advance,
Christoph.

2

No, not with Icinga 2 only, but you can send notifications to another solution (or build your own using a database and some scripting) which could then be more flexible. I have seen such solutions like opsgenie, datadog or self-written ones at customers but never configured them myself.

1 Like
3

try to split it in 3 separate notifications

  1. warning state notification interval 2 hours
    => transition type problem / no recovery
  2. critical/unknown state notification interval 2 hours
    => transition type problem / no recovery
  3. OK interval 0
    => transition type recovery
1 Like
4

I wonder whether there might be a way to manipulate the runtime variable
next_notification inside the notification object itself:
https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#notification

Antony.

5

This looks exactly right. Unfortunately, I’ve been spectacular unsuccessful implementing it. In particular,

 states = [ OK, Critical ]
  types = [ Recovery, Problem ]

works, meaning problem and recovery mails are sent as expected, but taking away the “Critical” and “Problem”, resulting in a

 states = [ OK ]
  types = [ Recovery ]

doesn’t, meaning that I’m getting nothing at all anymore (I tried some other variations, but had no success either). According to the Monitoring Basics documentation, “Icinga 2 v2.10 allows you to configure Acknowledgement and/or Recovery without a Problem notification”. I’m running 2.13.2, so the latter example is supposed to work(?). Any ideas what I’m doing wrong here?

6

@chrender yes, but I don’t know, why my commit isn’t in the documentation as my pull request was merged.

In short, it only works on the user object.

1 Like
7

Documentation “latest” is for latest release (2.13), but your commit is only in master for the next release (2.14). But there is again a “snapshot” documentation which is the current master, so here you see your commit: Monitoring Basics - Icinga 2 (Snapshot)

8

You could perhaps “misuse” an event handler script for this, depending on how complex you want re-notifications to be if there is no state change. Event handler scripts offer different handling for different states if you want them to.

1 Like
9

Thanks for the clarification. This is unfortunate, as the information is valid for the current version as well and the current documentation is misleading as the promised filters combinations only work on user objects not on notification objects.

10

From all your feedback it appears an external solution is the best/only available option. Since this would probably cause more work that it’s worth I’ll just attempt to find a good average interval between warnings and criticals, if I’m lucky I might get away with this. Thanks all!