Icinga 2 master schedules the same check on both agents (at the wrong time)

Hello everyone, we have just discovered that in our distributed setup with one icinga master and two agents, the master schedules the same check on both agents (at the wrong time): we found that the value “Next check” in the icinga web interface is not consistent, i.e. at a certain time (when the Next check is not “0s”) the Last check is reset to “0s ago” and the Check Source changes to the other agent.

We have configured our service & host templates to run the checks in our custom “Agent” zone, as to avoid checks directly from the master.

Do you have any suggestions?

Thank you.

Hello everyone, sorry for necro-bumping this thread, but the problem still persists even after updating icinga a year later. Maybe I wasn’t very clear in explaining the problem: it basically doesn’t load balance checks between agents but instead duplicates them. Is this the intended behaviour?

Thank you.

Hi @francesco.cepparo,

I’m not sure why no one from the community could help you for so long.

If I understand it correctly, you have both Agents in one zone. If that’s correct you need to make sure, that both Agents also know each other and are connected. That way they’ll make sure to only run the check on one of them.

If that doesn’t help, please show us your zones.conf of all 3 instances and your check config.

Greetings
Noah

1 Like

Hi Noah,

I’m not sure why no one from the community could help you for so long.

I think that my original question was not clear enough. My own fault.

If I understand it correctly, you have both Agents in one zone. If that’s correct you need to make sure, that both Agents also know each other and are connected. That way they’ll make sure to only run the check on one of them.

That’s correct.

After applying your suggestion the checks are now being distributed correctly. Thank you very much for your help.

May I suggest fixing the documentation in order to account for what you just told me? The doc currently explicitly says that “The two agent nodes do not need to know about each other. The only important thing is that they know about the parent zone and their endpoint members (and optionally the global zone).” Reference:
https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#master-with-agents

Hi @francesco.cepparo,

the documentation isn’t wrong here. It’s just a naming problem. Agents, in most setups, only check local things on the machine itself and thus don’t use the HA features and have separate zones.

The agents in the documentation all have their own zones and don’t need to know each other for HA.

Greetings
Noah

I see. I was misguided by that sentence but it makes sense now. Thank you again!