I’m currently working on building out a distributed monitoring cluster running with:
Icinga2: r2.13.3-1
Icinga web 2: 2.10.1
Director: 1.9.1
PHP: 7.4.3
Ubuntu 20.04
I’m running a number of other modules I’m not sure are relevant but here is the System tab in IcingaWeb2:
Presently director, web2, and icinga2 are all running from the same host. I have observed multiple times now where a hostgroup applies to a host, where it seems like it shouldn’t. One such instance occurred yesterday:
I have blanked identifying information but this is a Unifi 48-port switch. The blanked out hostgroup is a group with no apply-to rules just for grouping all of the customer’s devices. The “Unifi” group gets applied vi a “Unifi Switch” hostgroup:
This then allows the service-set “Unifi” to properly apply to the host:
This service-set contains 2 services, a CPU and RAM check. Yesterday, nearly every Unifi device in our cluster had some goofy assignments as you can see in the second screenshot where the Ubuntu Linux and Linux hostgroups are applied. The appy-to statement on Linux appears as so:
and the Ubuntu Linux group:
This caused many of these devices to have no services at all, or the completely wrong set of services within the web interface. Weirdly, you would think this would cause a MASSIVE slew of notifications, but it did not. Additionally, when I look at the activity log and render the config files, the devices remain unchanged from the prior week up through fixing the issue:
Config as of Friday, August 12, 2022
Config as of Monday August 15, 2022 (when issue was reported)
Config as of Tuesday, August 16, 2022 (issue has been resolved for 24 hours)
As far as I can tell, absolutely nothing changed both before and after the issue. To resolve the issue, I can change the apply-to statements on the hostgroups or service sets, deploy, then change it back, and deploy again. This has typically fixed the issue. From what I can tell, this appears to be cosmetic considering the rendered config never changes. That said, I’d really like to know what is causing this and how I can prevent it.