Icinga Agent is reloading configuration in a loop

Hello,

after we migrated one of the Icinga Master server to another site, all Linux Agents were reloading the configuration in a loop.

[2020-08-19 09:21:21 +0700] information/ApiListener: Restarting after configuration change.
[2020-08-19 09:21:22 +0700] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2020-08-19 09:21:22 +0700] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2020-08-19 09:21:23 +0700] information/Application: Got reload command: Starting new instance.
[2020-08-19 09:21:23 +0700] information/Application: Reload requested, letting new process take over.
[2020-08-19 09:21:23 +0700] information/ApiListener: 'api' stopped.
[2020-08-19 09:21:23 +0700] information/CheckerComponent: 'checker' stopped.
[2020-08-19 09:21:23 +0700] information/FileLogger: 'main-log' started.
[2020-08-19 09:21:23 +0700] information/ApiListener: 'api' started.
[2020-08-19 09:21:23 +0700] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-08-19 09:21:23 +0700] information/CheckerComponent: 'checker' started.
[2020-08-19 09:21:23 +0700] information/ConfigItem: Activated all objects.
[2020-08-19 09:21:30 +0700] information/ApiListener: New client connection for identity 'master.domain.tld' from [10.2.1.161]:42928
[2020-08-19 09:21:30 +0700] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'master.domain.tld'.
[2020-08-19 09:21:30 +0700] information/ApiListener: Sending config updates for endpoint 'master.domain.tld' in zone 'master'.
[2020-08-19 09:21:30 +0700] information/ApiListener: Finished sending config file updates for endpoint 'master.domain.tld' in zone 'master'.
[2020-08-19 09:21:30 +0700] information/ApiListener: Syncing runtime objects to endpoint 'master.domain.tld'.
[2020-08-19 09:21:30 +0700] information/ApiListener: Finished syncing runtime objects to endpoint 'master.domain.tld'.
[2020-08-19 09:21:30 +0700] information/ApiListener: Finished sending runtime config updates for endpoint 'master.domain.tld' in zone 'master'.
[2020-08-19 09:21:30 +0700] information/ApiListener: Sending replay log for endpoint 'master.domain.tld' in zone 'master'.
[2020-08-19 09:21:31 +0700] information/ApiListener: Finished sending replay log for endpoint 'master.domain.tld' in zone 'master'.
[2020-08-19 09:21:31 +0700] information/ApiListener: Finished syncing endpoint 'master.domain.tld' in zone 'master'.
[2020-08-19 09:21:31 +0700] information/ApiListener: Applying config update from endpoint 'master.domain.tld' of zone 'master'.
[2020-08-19 09:21:31 +0700] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones/director-global' (0 Bytes). Received timestamp '2020-08-19 09:21:31 +0700' (1597803691.941806), Current timestamp '2020-08-19 09:21:21 +0700' (1597803681.928448).
[2020-08-19 09:21:31 +0700] information/ApiListener: Restarting after configuration change.

To my understanding it tries to reload the director-global zone, which isn’t used and also has 0 Byte. For now I have removed the director-global zone from the zones file so the Agent don’t reload every 10s, but I want to understand why this happened.

Hi,

which version of Icinga 2 you are using on your main Icinga 2 node and your agents?

Best regards
Michael

Hi,

the masters are on 2.11.4. The agents are on different versions between 2.10.3 and 2.11.2. But the problem just occured after the migration of one master, where we only changed the IP, the hostname, renewed the certificate and changed the zones files on the directly to the master zone connected hosts.

Best regards,
Martin

Hi,

I know that there was a bug in older versions with the sync mechanism that empty zones directories got synced and another bug with the timestamp comparison of the remote and local config. Unfortunately I can’t find the issue on GitHub right now.

In the past I workaround this problem and created an empty file called empty.conf inside the global zones and just put a comment in it.

# vim /etc/icinga2/zones.d/director-global/empty.conf

// this file is empty

I don’t know if this is the issue in question since you faced this issue after your migration of the Icinga 2 main server.

Does this behavior occur on all connected agents/satellites or only specific ones?

Best regards
Michael

Hi Michael,

this happened on all Linux agents. The Windows agents don’t have the problem.

I will try the workaround with the empty file after my vacation. Thanks for the tip.

Best regards,
Martin