Singe Downtime Notification for all Children

Icinga 2
1

Hello,

I am using icinga 2.15.2 with icingadb 1.5.1 .

When creating a downtime for a host + all services, the start and end of the downtime will trigger a notification for all Downtimes, which can get super spammy.

I would like to only send one Notification for the “main” downtime, and then just have the children mentioned in the message.

I was thinking of filtering is out in the notification or the notificaiton command object, but for that i need a way to discern the different downtimes. Is there a Variable that i can use to filter out Downtimes that are Children of other downtimes?

All the best,

Nicolas

2

you could adapt the notification rule.

  • split it into two
    • downtime start + downtime end only for parent is not set
    • everything except downtime start end
3

Hi,

I don’t know how things are organised at your place, but here this issue would be tricky to resolve. Let me show with an example:

Host 1 is a server with the following Services:

  1. System disk
  2. CPU load
  3. Memory
  4. HTTPS service
  5. SSL certificate validity
  6. Icinga check
  7. cluster-zone check

Plus of course a (0) “hostalive” (ping) check on the Host itself.

Because of the split of responsibilities, the following downtime start/end notifications should be sent:

  • To the Servers team: 0, 1
  • To the Applications team: 0, 2, 3, 4, 5
  • To the PKI team: 5
  • To the Icinga administration team: 6, 7
  • To the Operations team: 0, 4

Suppose a downtime is set on the Host level. How the Service notification script can figure out whether the downtime notification for service 1/2/3/4/5/6/7 needs to be sent to each of the teams will be very difficult to establish.

So my advice would be to be very careful in capturing the business model for these notifications, and having the tools in place for the notification script to handle them.

Perhaps https://icinga.com/docs/icinga-notifications/latest/ could be part of the solution?

My two cents,

Jean

4

@moreamazingnick How can i access the downtime start and end inside of the Notification or NotificationCommand object?

@jeanm

In my case, all alerts are handled by the same team, so your concern would not apply to my usecase.

For the icinga notifications plugin: i haven’t found any clue in that documentation that it would support my usecase

5

right above

Notification
the example already shows:

types = [ Problem, Recovery ]

this means this only applies to these two, not Downtime
PS: if this is unset it applies to all types

Optional. A list of type filters when this notification should be triggered. By default everything is matched.

and these are the types:

DowntimeStart
DowntimeEnd
DowntimeRemoved
Custom
Acknowledgement
Problem
Recovery
FlappingStart
FlappingEnd

but this is the notification object.

and here is the chapter how to apply a notification rule:

where you can also set filters for the notification

assign where host.vars.notification.mail

6

Ah, i think there is a missunderstanding, i do want no notifications if the Downtime has a parent, not the service.

E.g. if a downtime is a child downtime of another.

7

do you use parent host definitions and the “Schedule triggered downtime for all child hosts” feature?
or is it just the host => all services trigger you use?

8

Sorry for the super late reply, i kind of forgot about this.

When i create a downtime for a host and check the “All Services” checkbox. Icinga generates downtimes for the Host and all services.

All of these trigger an individual notification, which, on a host with many checks, kind of floods any notification channel.

Disabling notifications for services is not an option, since we also sometimes only add a downtime for individual services, where we would like to get notified on.

Looking into the icinga API, in these Scenarios, the Downtime for the services have an attribute (parent) that references the downtime of the host.

E.g.

“parent”: “test.host!1f2926a3-cb35-4b91-a208-c2119f09b0cf”,

I would like to filter notifications for downtimes to only notify on downtimes that do not have a parent.

(And as a bonus, maybe, add an info about all children of a downtime.)

9

this is just some brainstormin:

  • host has the runtime attribute downtime_depth
  • a dependency can disable notifications
  • maybe you can use the host.downtime_depth in a dependency apply rule

but I don’t know if runtime attributes work in config apply rules

or some other not so nice idea:

in your notification script

  • check if it is a downtime
  • use icinga2 api and check if the service is in a triggered downtime
  • decide if you want to send a notification for that.