Some objects were missing on new satellite

I’ve a well running multi tier setup with one master and two satellites. Adding a third satellite in the same way (means adding an new zone, same configuration etc.) result in missing some service objects for host objects in that new zone.

It took me some hours to find the root cause: a empty file named .authoritative was in /var/lib/icinga2/api/zones/windows-commands/

Were does this file comes from? Might that happen again?

All nodes having icinga r2.10.5-1 installed.


it seems that there is a bug with syncing this file to other endpoints. I might have fixed this with 2.11 and the stages.

The main intention of this file is the following:

  • As you know, there’s only one config master with /etc/icinga2/zones.d
  • This endpoint must not receive configuration from the secondary master, this would result in a loop
  • Therefore the configuration is copied from /etc/icinga2/zones.d to /var/lib/icinga2/api/zones*
    • The .authoritative file marker is added in there
    • On reconnect, everything except for the .authoritative file marker is synced to the connecting endpoints (master in the same zone, satellites in the child zone).
    • Whenever this config master receives a configuration from the other master, it checks for having the config authority for this zone. If that’s true, you’ll see a message like this:
Ignoring config update from endpoint 'master2.localdomain' for zone 'master' because we have an authoritative version of the zone's config.

That’s information level, so totally valid to ignore it. Whenever some of your satellites logs that, that’s suspicious.

Agreed, this is hard to debug, but also hard to write a troubleshooting entry.

I also have seen users which manually rsync/copy /var/lib/icinga2/api/zones* from the config master and then run into this behaviour.