Trouble with Icinga Satellite not getting Host Templates from Master

Icinga Web Modules Icinga Director
1

I have a problem where configurations aren’t getting synced to my satellite node.

Environment:
Using a master with mysql backend and director, and a satellite, both version: 2.11.2-1

Problem: I cannot get objects synced to the satellite.

In the satellite icinga2.log:

[2020-01-20 10:46:14 -0600] information/ApiListener: Received configuration updates (1) from endpoint 'master.redacted' are different to production, triggering validation and reload. [2020-01-20 10:46:17 -0600] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Abor ting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log' [2020-01-20 10:46:28 -0600] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Abor ting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log'

In the aforementioned startup.log, there are several host templates not being imported that hosts rely on (last few lines only, as it occurs for 2778 hosts):

[2020-01-20 10:46:28 -0600] critical/config: Error: Import references unknown template: 'HT-OPS2-HOST-CHECK'
Location: in /var/lib/icinga2/api/zones-stage//Zone-REDACTED/director/hosts.conf: 51009:5-51009:31 /var/lib/icinga2/api/zones-stage//Zone-REDACTED/director/hosts.conf(51007): /var/lib/icinga2/api/zones-stage//Zone-REDACTED/director/hosts.conf(51008): object Host "some.host.name" { /var/lib/icinga2/api/zones-stage//Zone-REDACTED/director/hosts.conf(51009): import "HT-OPS2-HOST-CHECK" ^^^^^^^^^^^^^^^^^^^^^^^^^^^ /var/lib/icinga2/api/zones-stage//Zone-REDACTED/director/hosts.conf(51010): /var/lib/icinga2/api/zones-stage//Zone-REDACTED/director/hosts.conf(51011): display_name = "some.host.name - CID 3761 (3717)"

[2020-01-20 10:46:28 -0600] critical/config: 2778 errors
[2020-01-20 10:46:28 -0600] critical/cli: Config validation failed. Re-run with ‘icinga2 daemon -C’ after fixing the config.`

In both debug logs, I see the following:

[2020-01-20 10:31:49 -0600] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'satellite.redactedPreformatted text': Error: short read (0) icinga2: icinga::JsonRpc::ReadMessage(std::shared_ptr<icinga::AsioTlsStream> const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) (+0x95) [0x92e165] (1) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0xf4) [0xbc26e4] [2020-01-20 10:31:49 -0600] warning/JsonRpcConnection: API client disconnected for identity 'satellite.redacted' [2020-01-20 10:31:49 -0600] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'satellite.redacted'. '1', state_type = '1', status_update_time = FROM_UNIXTIME(1579538756) WHERE service_object_id = 18400

I can provide any other information as needed. I’ve been chasing this for a few days, and can’t seem to find anything else on it.

Edit: I’m 99% certain it has something to do with zones, but I’m too fuzzled and probably overlooking it.

2

did you run:

icinga2 daemon -C

already?

1 Like
3

Hello

can you provide the following :
satellite constants.conf
Host template with zone definition
Master zones.conf

4

Hmmm the short read would indicate that the TLS connection was not properly closed by the other side.
Do you see anything in regard to the config sync / reload with closing connections too early? Likewise, how big is your /var/lib/icinga2/api/zones directory, and are you transferring data on a slow network connection maybe?

Also, if you do a tcpdump, can you see the correct TLS exchanges and data packets sent?

Cheers,
Michael

5

In order:

@rsx yes, while telling it to check the zones-stage folder it has the same output as the startup log. If I use the “base” command where it checks the /etc/icinga2/ directory, it passes.

@aflatto

Satellite constants.conf

/**
 * This file defines global constants which can be used in
 * the other configuration files.
 */

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = "/usr/lib64/nagios/plugins"

/* The directory which contains the Manubulon plugins.
 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
 */
const ManubulonPluginDir = "/usr/lib64/nagios/plugins"

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
 * Check the documentation, chapter "Plugins Contribution", for details.
 */
const PluginContribDir = "/usr/lib64/nagios/plugins"

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "satellite.redacted"

/* Our local zone name. */
const ZoneName = "Zone-Redacted"

/* Secret key for remote node tickets */
const TicketSalt = ""

When looking for the host templates in /var/lib/icinga2/api/zones-stage/Zone-Redacted/director/, I find that the host-templates aren’t getting there at all (hosts.conf and services.conf are the only 2 files in this directory and the host-templates aren’t defined in either file, but are referenced in hosts with the import statement).

Keep in mind that I’m using director, here is my master's zone.conf:

object Zone "master" {
	endpoints = [ "master.redacted" ]
}


object Zone "global-templates" {
	global = true
}

Oddly enough, the endpoint isn’t defined in this file, but is defined in director.

@dnsmichi
Other than the short read “error” in the debug log, all connections seem normal:
Sample from /var/log/icinga2.log on the master

[2020-01-21 07:56:34 -0600] information/ApiListener: New client connection for identity 'satellite.redacted' from [172.27.0.189]:41336
[2020-01-21 07:56:34 -0600] information/ApiListener: Sending config updates for endpoint 'satellite.redacted' in zone 'Zone-Redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Syncing configuration files for zone 'Zone-Redacted' to endpoint 'satellite.redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Finished sending config file updates for endpoint 'satellite.redacted' in zone 'Zone-Redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Syncing runtime objects to endpoint 'satellite.redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Finished syncing runtime objects to endpoint 'satellite.redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Finished sending runtime config updates for endpoint 'satellite.redacted' in zone 'Zone-Redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Sending replay log for endpoint 'satellite.redacted' in zone 'Zone-Redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Finished sending replay log for endpoint 'satellite.redacted' in zone 'Zone-Redacted'.
[2020-01-21 07:56:34 -0600] information/ApiListener: Finished syncing endpoint 'satellite.redacted' in zone 'Zone-Redacted'.
[2020-01-21 07:56:35 -0600] information/JsonRpcConnection: Received certificate request for CN 'satellite.redacted' signed by our CA.
[2020-01-21 07:56:35 -0600] information/JsonRpcConnection: The certificate for CN 'satellite.redacted' is valid and uptodate. Skipping automated renewal.
[2020-01-21 07:56:51 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:56:51 -0600] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 0, rate: 2.73333/s (164/min 850/5min 2564/15min);
[2020-01-21 07:56:51 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.2/s (12/min 74/5min 231/15min);
[2020-01-21 07:56:53 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'

Here is the size of the directory:
du -sch /var/lib/icinga2/api/zones
8.8M /var/lib/icinga2/api/zones
8.8M total

In this instance, the master and satellite exist on the same KVM cluster with a 10Gb backend.

While performing tcpdump on the satellite and restarting icinga2 on the master, I am able confirm a successful TLS exchange. I also see the in debug logs that they are connected to each other, and in director, I am able to inspect the endpoint successfully.

EDIT:

I’ve found that the host-templates are defined on the master in /var/lib/icinga2/api/zones/director-global/director/host_templates.conf. It seems like the director-global zone isn’t getting synced like it should maybe? I reckon that even if I just moved host-templates over, there would be a number of other dependancies that aren’t met that are defined in director-global.

6

Since this zone is not defined in your master’s zones.conf, this likely is the culprit. Keep in mind that the agents, satellites also need this zone being configured.

Cheers,
Michael

7

director-global currently exists in director and is defined as a global zone.

When I add this to my master’s zone.conf and reload icinga2:

critical/config: Error: Object 'director-global' of type 'Zone' re-defined: in /var/lib/icinga2/api/packages/director/c617fba2-2856-4afc--e8a45133fe4c/zones.d/director-global/zones.conf: 1:0-1:28; previous definition: in /etc/icinga2/zones.conf: 19:1-19:29

Same thing happens on my satellite node.

8

That’s not a good idea, Zones must not be managed in the Director, especially not global ones. They need to exist in Icinga 2’s zones.conf and thus be imported via Director kickstart allowing to use them. See these instructions.

1 Like
9

I guess I could rename the existing director-global zone to something, introduce the local one, go into the DB and lookup the new ID and change references to the old ID in the objects to look at the new ID. Director won’t let me delete the old one until no other objects reference it, so it should be clean.

I’ll report back if this works or not.

10

I did the following:

  • Defined the zone in the zone.conf file on the master and satellite.
  • Logged into the database for director (in my case, mysql/mariadb) and toggled the zone to be externally_defined in the object type field:
    update icinga_zone set object_type = external_object where object_name = "director-global";
  • Rendered the config in director and pushed the configs even though there were no changes.

I now see the zone getting deployed to the satellite, but for the past 10 minutes or so, my satellite is in a loop according to the logs (this repeats over and over again):

[2020-01-21 09:59:12 -0600] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2020-01-21 09:59:12 -0600] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'Zone-redacted//.checksums' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'Zone-redacted//.timestamp' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'Zone-redacted//director/hosts.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'Zone-redacted//director/services.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//.checksums' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//.timestamp' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/001-director-basics.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/command_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/commands.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/host_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/hostgroups.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/service_apply.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/service_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/servicegroups.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/timeperiod_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/timeperiods.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:12 -0600] information/ApiListener: Copying file 'director-global//director/user_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:15 -0600] information/CheckerComponent: 'checker' stopped.
[2020-01-21 09:59:15 -0600] information/ExternalCommandListener: 'command' stopped.
[2020-01-21 09:59:15 -0600] information/ApiListener: 'api' stopped.
[2020-01-21 09:59:16 -0600] information/FileLogger: 'main-log' started.
[2020-01-21 09:59:16 -0600] information/ApiListener: 'api' started.
[2020-01-21 09:59:16 -0600] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-01-21 09:59:16 -0600] information/ExternalCommandListener: 'command' started.
[2020-01-21 09:59:16 -0600] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
	(0) Activating object 'command' of type 'ExternalCommandListener'

[2020-01-21 09:59:16 -0600] information/CheckerComponent: 'checker' started.
[2020-01-21 09:59:16 -0600] information/ApiListener: Reconnecting to endpoint 'satellite2.redacted' via host 'satellite2.redacted' and port '5665'
[2020-01-21 09:59:16 -0600] information/ApiListener: Reconnecting to endpoint 'satellite.redacted' via host 'satellite.redacted' and port '5665'
[2020-01-21 09:59:16 -0600] information/ConfigItem: Activated all objects.
[2020-01-21 09:59:16 -0600] information/ApiListener: Reconnecting to endpoint 'satellite2.redacted' via host 'satellite2.redacted' and port '5665'
[2020-01-21 09:59:16 -0600] information/ApiListener: Reconnecting to endpoint 'master.redacted' via host 'master.redacted' and port '5665'
[2020-01-21 09:59:16 -0600] information/ApiListener: New client connection for identity 'master.redacted' to [172.27.0.188]:5665
[2020-01-21 09:59:16 -0600] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'master.redacted'.
[2020-01-21 03:50:56 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 03:51:48 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 03:51:48 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 03:55:56 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 03:56:58 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 03:56:58 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:00:56 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:02:08 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:02:08 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:05:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:07:18 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:07:18 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:10:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:12:28 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:12:28 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:15:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:17:38 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:17:38 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:20:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:22:48 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:22:48 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:25:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:27:58 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:27:58 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:30:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:33:08 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:33:08 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:35:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:38:18 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:38:18 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:40:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:43:28 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:43:28 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:45:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:48:38 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:48:38 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:50:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:53:48 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:53:48 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:55:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 04:58:58 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 04:58:58 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:00:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:04:08 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:04:08 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:05:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:09:18 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:09:18 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:10:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:14:28 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:14:28 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:15:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:19:38 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:19:38 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:20:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:24:48 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:24:48 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:25:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:29:57 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:29:57 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:30:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:35:07 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:35:07 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:35:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:40:17 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:40:17 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:40:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:45:27 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:45:27 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:45:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:50:37 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:50:37 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:50:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 05:55:47 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:55:47 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 05:55:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:00:57 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:00:57 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:00:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:05:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:06:07 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:06:07 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:10:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:11:17 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:11:17 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:15:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:16:27 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:16:27 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:20:57 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:21:37 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:21:37 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:25:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:26:47 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:26:47 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:30:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:31:57 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:31:57 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:35:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:37:07 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:37:07 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:40:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:42:17 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:42:17 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:45:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:47:27 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:47:27 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:50:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:52:36 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:52:36 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:55:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 06:57:46 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 06:57:46 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:00:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 07:02:56 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:02:56 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:05:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 07:08:06 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:08:06 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:10:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-01-21 07:13:16 -0600] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:13:16 -0600] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-01-21 07:15:58 -0600] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
...skipping...
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending runtime config updates for endpoint 'satellite2.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Sending replay log for endpoint 'satellite2.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending replay log for endpoint 'satellite2.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished syncing endpoint 'satellite2.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished reconnecting to endpoint 'satellite2.redacted' via host 'satellite2.redacted' and port '5665'
[2020-01-21 09:59:19 -0600] information/ApiListener: New client connection for identity 'master.redacted' to [172.27.0.188]:5665
[2020-01-21 09:59:19 -0600] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'master.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Sending config updates for endpoint 'master.redacted' in zone 'master'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending config file updates for endpoint 'master.redacted' in zone 'master'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Syncing runtime objects to endpoint 'master.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished syncing runtime objects to endpoint 'master.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending runtime config updates for endpoint 'master.redacted' in zone 'master'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Sending replay log for endpoint 'master.redacted' in zone 'master'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending replay log for endpoint 'master.redacted' in zone 'master'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished syncing endpoint 'master.redacted' in zone 'master'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished reconnecting to endpoint 'master.redacted' via host 'master.redacted' and port '5665'
[2020-01-21 09:59:19 -0600] information/ApiListener: Applying config update from endpoint 'satellite2.redacted' of zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Received configuration for zone 'Zone-redacted' from endpoint 'satellite2.redacted'. Comparing the timestamp and checksums.
[2020-01-21 09:59:19 -0600] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Zone-redacted' (0 Bytes).
[2020-01-21 09:59:19 -0600] information/ApiListener: Received configuration updates (1) from endpoint 'satellite2.redacted' do not qualify for production, not triggering reload.
[2020-01-21 09:59:19 -0600] information/ApiListener: New client connection for identity 'satellite.redacted' to [172.27.128.132]:5665
[2020-01-21 09:59:19 -0600] information/ApiListener: Sending config updates for endpoint 'satellite.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Syncing configuration files for zone 'Zone-redacted' to endpoint 'satellite.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'satellite.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending config file updates for endpoint 'satellite.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Syncing runtime objects to endpoint 'satellite.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished syncing runtime objects to endpoint 'satellite.redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending runtime config updates for endpoint 'satellite.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Sending replay log for endpoint 'satellite.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished sending replay log for endpoint 'satellite.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished syncing endpoint 'satellite.redacted' in zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Finished reconnecting to endpoint 'satellite.redacted' via host 'satellite.redacted' and port '5665'
[2020-01-21 09:59:19 -0600] information/ApiListener: Applying config update from endpoint 'satellite.redacted' of zone 'Zone-redacted'.
[2020-01-21 09:59:19 -0600] information/ApiListener: Received configuration for zone 'Zone-redacted' from endpoint 'satellite.redacted'. Comparing the timestamp and checksums.
[2020-01-21 09:59:19 -0600] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Zone-redacted' (0 Bytes).
[2020-01-21 09:59:19 -0600] information/ApiListener: Received configuration updates (1) from endpoint 'satellite.redacted' do not qualify for production, not triggering reload.
[2020-01-21 09:59:20 -0600] information/ApiListener: Applying config update from endpoint 'master.redacted' of zone 'master'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Received configuration for zone 'Zone-redacted' from endpoint 'master.redacted'. Comparing the timestamp and checksums.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Zone-redacted//director/hosts.conf' for zone 'Zone-redacted'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Zone-redacted//director/services.conf' for zone 'Zone-redacted'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Zone-redacted' (4338974 Bytes).
[2020-01-21 09:59:20 -0600] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'master.redacted'. Comparing the timestamp and checksums.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/command_templates.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/commands.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/hostgroups.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_apply.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicegroups.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/timeperiod_templates.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/timeperiods.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/user_templates.conf' for zone 'director-global'.
[2020-01-21 09:59:20 -0600] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (93215 Bytes).
[2020-01-21 09:59:20 -0600] information/ApiListener: Received configuration updates (2) from endpoint 'master.redacted' are different to production, triggering validation and reload.
[2020-01-21 09:59:27 -0600] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'Zone-redacted//.checksums' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'Zone-redacted//.timestamp' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'Zone-redacted//director/hosts.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'Zone-redacted//director/services.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//.checksums' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//.timestamp' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/001-director-basics.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/command_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/commands.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/host_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/hostgroups.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/service_apply.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-global//director/service_templates.conf' from config sync staging to production zones directory.
[2020-01-21 09:59:27 -0600] information/ApiListener: Copying file 'director-

What I’m seeing is the satellites getting the config, reloading, getting the config, reloading, and so on and so forth with no end in sight.

EDIT: I think this was due to me only having one of the satellites in the zone configured correctly. After getting all of them configured correctly, they eventually stopped reloading and are functioning normally.

Thanks for the help!

1 Like