We are facing a problem with duplicate notifications / notification history within our HA icinga2 setup.
We have a master zone with 2 servers in it each with the following features enabled :
api checker debuglog ido-mysql mainlog notification
About a dozen of satellites zones with each 2 servers in it as well, with the following features enabled :
api checker debuglog mainlog notification
We also have a separate server running icingaweb and icinga-director.
Finally we have a couple of servers running mysql in a master/master configuration.
We manage all of our config with the director except for the zone/features/constants that are managed by puppet.
Each of our host objects are attributed to a satellite zone, the masters do not do checks (except for a couple a specific checks).
We apply notifications with the director based on multiple rules. They do get applied correctly but, and this where our problem starts, the notifications objects are created on satellites machines (only for the hosts/services which the satellite is responsible for which is perfect) and masters as well (which get the notifications for every hosts/services objects).
This ends up in us receiving every notifications twice; once from one of the satellite and a second from one of the masters.
We ended up trying the solution mentioned here (which btw, will not work with the director because it will add quotes to
ZoneName in the apply rule, but that is another issue) it does create the notifications objects only on the satellites machines but we do not have any notification history in icingaweb (same as we simply disabling notification feature on master zone).
My question would be, is there a way to have our setup working without duplicate notifications and with notification history in icingaweb ?
I should add, having only master sending alerts is not possible for our case, for geographic and reliability reason.
I hope this is enough informations, I did not put zones.conf or objects details because I think this is more of an architecture design issue but if this setup is theoretically possible and this info is needed I can add it.
Some details :
- Version used : 2.12.3
- CentOS 8
- Icinga Web 2 version and modules 2.8.2, director 1.8.0