Some objects were missing on new satellite

rsx · September 27, 2019, 7:49am

I’ve a well running multi tier setup with one master and two satellites. Adding a third satellite in the same way (means adding an new zone, same configuration etc.) result in missing some service objects for host objects in that new zone.

It took me some hours to find the root cause: a empty file named .authoritative was in /var/lib/icinga2/api/zones/windows-commands/

Were does this file comes from? Might that happen again?

All nodes having icinga r2.10.5-1 installed.

dnsmichi · September 27, 2019, 8:22am

Hi,

it seems that there is a bug with syncing this file to other endpoints. I might have fixed this with 2.11 and the stages.

github.com

Icinga/icinga2/blob/master/lib/remote/apilistener-filesync.cpp#L774


/**
 * Read the given file and store it in the config information structure.
 * Callback function for Glob().
 *
 * @param config Reference to the config information object.
 * @param path File path.
 * @param file Full file name.
 */
void ApiListener::ConfigGlobHandler(ConfigDirInformation& config, const String& path, const String& file)
{
	// Avoid loading the authoritative marker for syncs at all cost.
	if (Utility::BaseName(file) == ".authoritative")
		return;


	CONTEXT("Creating config update for file '" + file + "'");


	Log(LogNotice, "ApiListener")
		<< "Creating config update for file '" << file << "'.";


	std::ifstream fp(file.CStr(), std::ifstream::binary);
	if (!fp)

The main intention of this file is the following:

As you know, there’s only one config master with /etc/icinga2/zones.d
This endpoint must not receive configuration from the secondary master, this would result in a loop
Therefore the configuration is copied from /etc/icinga2/zones.d to /var/lib/icinga2/api/zones*
- The .authoritative file marker is added in there
- On reconnect, everything except for the .authoritative file marker is synced to the connecting endpoints (master in the same zone, satellites in the child zone).
- Whenever this config master receives a configuration from the other master, it checks for having the config authority for this zone. If that’s true, you’ll see a message like this:

Ignoring config update from endpoint 'master2.localdomain' for zone 'master' because we have an authoritative version of the zone's config.

That’s information level, so totally valid to ignore it. Whenever some of your satellites logs that, that’s suspicious.

Agreed, this is hard to debug, but also hard to write a troubleshooting entry.

I also have seen users which manually rsync/copy /var/lib/icinga2/api/zones* from the config master and then run into this behaviour.

Cheers,
Michael