Since upgrade to 2.11 host changes at the director are no longer synced to satellite

icinga2.log master:

[2019-10-18 08:38:45 +0200] information/ApiListener: New client connection for identity 'mon1.site1.em.lan' from [192.168.53.211]:49114
[2019-10-18 08:38:45 +0200] information/ApiListener: Sending config updates for endpoint 'mon1.site1.em.lan' in zone 'site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished sending config file updates for endpoint 'mon1.site1.em.lan' in zone 'site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Syncing runtime objects to endpoint 'mon1.site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'mon1.site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'mon1.site1.em.lan' in zone 'site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Sending replay log for endpoint 'mon1.site1.em.lan' in zone 'site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished sending replay log for endpoint 'mon1.site1.em.lan' in zone 'site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished syncing endpoint 'mon1.site1.em.lan' in zone 'site1.em.lan'.
[2019-10-18 08:38:45 +0200] information/JsonRpcConnection: Received certificate request for CN 'mon1.site1.em.lan' signed by our CA.
[2019-10-18 08:38:45 +0200] information/JsonRpcConnection: The certificate for CN 'mon1.site1.em.lan' is valid and uptodate. Skipping automated renewal.

icinga2.log satellite:

[2019-10-18 08:38:45 +0200] information/ApiListener: Reconnecting to endpoint 'main.em.lan' via host '192.168.53.210' and port '5665'
[2019-10-18 08:38:45 +0200] information/ApiListener: New client connection for identity 'main.em.lan' to [192.168.53.210]:5665
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished reconnecting to endpoint 'main.em.lan' via host '192.168.53.210' and port '5665'
[2019-10-18 08:38:45 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'main.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Sending config updates for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished sending config file updates for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Syncing runtime objects to endpoint 'main.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'main.em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Sending replay log for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Replayed 9 messages.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished sending replay log for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 08:38:45 +0200] information/ApiListener: Finished syncing endpoint 'main.em.lan' in zone 'em.lan'.

but at satellite:

ll /var/lib/icinga2/api/zones/site1.em.lan/director/hosts.conf
-rw-rw---- 1 nagios nagios 757 Aug 20 11:33 /var/lib/icinga2/api/zones/site1.em.lan/director/hosts.conf

versions:

  • master: r2.11.1-1 (upgraded from r2.10.5-1 to 2.11.0-2 on Oct 1st and to 2.11.1-1 on Oct 17th)
  • satellite: r2.10.5-1

Is this a bug and I should open a new issue at gitlab? If so, at icinga2 (core) or director?

The logs don’t provide any sorts of syncing the zone site1.em.lan as they should. Try enabling the debug log on both instances to gather more insights about the synced files.

Cheers,
Michael

Master:

[2019-10-18 09:51:39 +0200] notice/ApiListener: Updated meta data for cluster config sync. Checksum: '/var/lib/icinga2/api/zones/windows-commands/.checksums'
, timestamp: '/var/lib/icinga2/api/zones/windows-commands/.timestamp', auth: '/var/lib/icinga2/api/zones/windows-commands/.authoritative'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/etc/icinga2/zones.d/site1.em.lan/t1'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/
zones.d/site1.em.lan/agent_endpoints.conf'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/
zones.d/site1.em.lan/agent_zones.conf'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/
zones.d/site1.em.lan/host_templates.conf'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/
zones.d/site1.em.lan/hosts.conf'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Copying 5 zone configuration files for zone 'site1.em.lan' to '/var/lib/icinga2/api/zones/site1.em.lan'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/site1.em.lan//_etc/t1
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/site1.em.lan//director/agent_endpoints.conf
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/site1.em.lan//director/agent_zones.conf
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/site1.em.lan//director/host_templates.conf
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/site1.em.lan//director/hosts.conf
[2019-10-18 09:51:39 +0200] notice/ApiListener: Updated meta data for cluster config sync. Checksum: '/var/lib/icinga2/api/zones/site1.em.lan/.checksums', timestamp: '/var/lib/icinga2/api/zones/site1.em.lan/.timestamp', auth: '/var/lib/icinga2/api/zones/site1.em.lan/.authoritative'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/zones.d/director-global/001-director-basics.conf'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/zones.d/director-global/host_templates.conf'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Copying 2 zone configuration files for zone 'director-global' to '/var/lib/icinga2/api/zones/director-global'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/001-director-basics.conf
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/host_templates.conf
[2019-10-18 09:51:39 +0200] notice/ApiListener: Updated meta data for cluster config sync. Checksum: '/var/lib/icinga2/api/zones/director-global/.checksums', timestamp: '/var/lib/icinga2/api/zones/director-global/.timestamp', auth: '/var/lib/icinga2/api/zones/director-global/.authoritative'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/zones.d/em.lan/agent_endpoints.conf'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/zones.d/em.lan/agent_zones.conf'.
[2019-10-18 09:51:39 +0200] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/packages/director/90593777-4f3b-4980-bb74-a26c7b227b00/zones.d/em.lan/hosts.conf'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Copying 3 zone configuration files for zone 'em.lan' to '/var/lib/icinga2/api/zones/em.lan'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/em.lan//director/agent_endpoints.conf
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/em.lan//director/agent_zones.conf
[2019-10-18 09:51:39 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/em.lan//director/hosts.conf
[2019-10-18 09:51:39 +0200] notice/ApiListener: Updated meta data for cluster config sync. Checksum: '/var/lib/icinga2/api/zones/em.lan/.checksums', timestamp: '/var/lib/icinga2/api/zones/em.lan/.timestamp', auth: '/var/lib/icinga2/api/zones/em.lan/.authoritative'.
[2019-10-18 09:51:39 +0200] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2019-10-18 09:51:39 +0200] debug/ApiListener: Not connecting to Zone 'site1.em.lan' because it's not in the same zone, a parent or a child zone.

satellite:

[2019-10-18 09:51:39 +0200] warning/JsonRpcConnection: API client disconnected for identity 'main.em.lan'
[2019-10-18 09:51:39 +0200] warning/ApiListener: Removing API client for endpoint 'main.em.lan'. 0 API clients left.
[2019-10-18 09:51:39 +0200] debug/EndpointDbObject: update is_connected=0 for endpoint 'main.em.lan'
...
[2019-10-18 09:51:46 +0200] information/ApiListener: Reconnecting to endpoint 'main.em.lan' via host '192.168.53.210' and port '5665'
[2019-10-18 09:51:46 +0200] notice/ApiListener: Current zone master: mon1.site1.em.lan
[2019-10-18 09:51:46 +0200] notice/ApiListener: Connected endpoints: plaza150k.site1.em.lan (1)
[2019-10-18 09:51:46 +0200] debug/HttpRequest: line: HTTP/1.1 204 No Content, tokens: 4
[2019-10-18 09:51:46 +0200] notice/ThreadPool: Thread pool; current: 2; adjustment: -1
[2019-10-18 09:51:46 +0200] debug/ThreadPool: Killing worker thread.
[2019-10-18 09:51:46 +0200] information/ApiListener: New client connection for identity 'main.em.lan' to [192.168.53.210]:5665
[2019-10-18 09:51:46 +0200] notice/ApiListener: New JSON-RPC client
[2019-10-18 09:51:46 +0200] debug/EndpointDbObject: update is_connected=1 for endpoint 'main.em.lan'
[2019-10-18 09:51:46 +0200] information/ApiListener: Finished reconnecting to endpoint 'main.em.lan' via host '192.168.53.210' and port '5665'
[2019-10-18 09:51:46 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'main.em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Sending config updates for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Finished sending config file updates for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Syncing runtime objects to endpoint 'main.em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'main.em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Sending replay log for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 09:51:46 +0200] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/current
[2019-10-18 09:51:46 +0200] information/ApiListener: Replayed 7 messages.
[2019-10-18 09:51:46 +0200] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/current
[2019-10-18 09:51:46 +0200] notice/ApiListener: Replayed 0 messages.
[2019-10-18 09:51:46 +0200] information/ApiListener: Finished sending replay log for endpoint 'main.em.lan' in zone 'em.lan'.
[2019-10-18 09:51:46 +0200] information/ApiListener: Finished syncing endpoint 'main.em.lan' in zone 'em.lan'.

master: cat /etc/icinga2/zones.conf

object Endpoint "main.em.lan" {
}

object Zone "em.lan" {
   endpoints = [ "main.em.lan" ]
}

object Zone "global-templates" {
   global = true
}

object Zone "director-global" {
   global = true
}

object Zone "windows-commands" {
   global = true
}

object Zone "site1.em.lan" {
   endpoints = [ "mon1.site1.em.lan" ]
}

object Endpoint "mon1.site1.em.lan" {
   host = "192.168.53.211"
   port = "5665"
}

Any progress on this or shall I head over to github?

I don’t see a bug yet, but I am really busy with other things right before OSMC.

One thought on this - the satellite logs still don’t tell anything about the satellite zone. Is that zone configured in zones.conf on the satellite?

Also, on a second look, the parent attribute is missing for object Zone "site1.em.lan" - so the master doesn’t know anything about this child zone and as such, no trust relationship is established and no config is synced.

Cheers,
Michael

I’ve moved the satellites from director to zones.conf close to the 2.11 update (and have missed to add parent). That’s why I thought this update might be the root cause (but was also thinking, no one else complains a similar issue - so there must be something wrong with my setup).

Thank you so much!

1 Like