HA setup duplicate notifications and history

Icinga 2
, , ,
1

Hello,

We are facing a problem with duplicate notifications / notification history within our HA icinga2 setup.

We have a master zone with 2 servers in it each with the following features enabled :
api checker debuglog ido-mysql mainlog notification

About a dozen of satellites zones with each 2 servers in it as well, with the following features enabled :
api checker debuglog mainlog notification

We also have a separate server running icingaweb and icinga-director.

Finally we have a couple of servers running mysql in a master/master configuration.

We manage all of our config with the director except for the zone/features/constants that are managed by puppet.
Each of our host objects are attributed to a satellite zone, the masters do not do checks (except for a couple a specific checks).

We apply notifications with the director based on multiple rules. They do get applied correctly but, and this where our problem starts, the notifications objects are created on satellites machines (only for the hosts/services which the satellite is responsible for which is perfect) and masters as well (which get the notifications for every hosts/services objects).
This ends up in us receiving every notifications twice; once from one of the satellite and a second from one of the masters.

We ended up trying the solution mentioned here (which btw, will not work with the director because it will add quotes to ZoneName in the apply rule, but that is another issue) it does create the notifications objects only on the satellites machines but we do not have any notification history in icingaweb (same as we simply disabling notification feature on master zone).

My question would be, is there a way to have our setup working without duplicate notifications and with notification history in icingaweb ?

I should add, having only master sending alerts is not possible for our case, for geographic and reliability reason.

I hope this is enough informations, I did not put zones.conf or objects details because I think this is more of an architecture design issue but if this setup is theoretically possible and this info is needed I can add it.

Some details :

  • Version used : 2.12.3
  • CentOS 8
  • Icinga Web 2 version and modules 2.8.2, director 1.8.0
2

Hello @camskkz,
Welcome to the Icinga community. You are receiving duplicate notifications because you have the the notification feature enabled on your satellite servers. Notification should only get sent from your master server. Please disable notification on your satellite servers and your duplicate notification should stop.

I hope the helps.
Alex

3

Thanks for the reply, unfortunately for our setup it is not possible to rely only on masters to send notifications so I guess the setup we wanted is not possible.

That might be something to consider in the future, if it is possible, to have notification events send through the API just like check results are (this way masters would be aware of notifications history, write those in DB and icingaweb would be able to show them).

4

Hi - it is possible. We wrote a wrapper script for our notification scripts on Icinga2 satellites, that reads the name of the normally notifying master from a parameter file, then checks that the master is reachable and if not, calls the real notification script (on the satellite), passing on the identical arguments; if he is reachable, it just logs an info line in a separate log file.

If the mentioned parameter file is missing, the wrapper assumes it is running on the master (i.e. it’s reachable). So the wrapper can be used universally.

I think this idea could help you. This way you don’t have to do any special config’s on your Icinga2 instances, but still have redundancy should the master(s) ever fail.

It of course assumes that masters handle all notifications normally, but this you could fine-tune… (Our main aim was redundancy of notifications for the customer.)

5

That’s indeed a possible solution, thanks.
We decided to change the notification scripts on masters to simply log and not send anything. That way we have alerts from satellites and still see the notification history in icingaweb but it’s a bit of a “hack” and I was wondering is there was a proper way to do it in Icinga that I might have missed.