Based on the documentation I should be looking in /var/log/icinga2/
for the log entries. I made a fake host for Site A and then deployed the configuration. Here are the 150 or so log entries created on IcingaMaster the were generated during that.
[2020-12-09 13:18:51 +0000] information/ApiListener: Copying 1 zone configuration files for zone 'Site-A' to '/var/lib/icinga2/api/zones/Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/Site-A//director/hosts.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Copying 3 zone configuration files for zone 'master' to '/var/lib/icinga2/api/zones/master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/hosts.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/services.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-12-09 13:18:51 +0000] information/ExternalCommandListener: 'command' started.
[2020-12-09 13:18:51 +0000] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
(0) Activating object 'command' of type 'ExternalCommandListener'
[2020-12-09 13:18:51 +0000] information/DbConnection: 'ido-mysql' started.
[2020-12-09 13:18:51 +0000] information/NotificationComponent: 'notification' started.
[2020-12-09 13:18:51 +0000] information/CheckerComponent: 'checker' started.
[2020-12-09 13:18:51 +0000] information/ApiListener: Reconnecting to endpoint 'Site-B-Satellite.picnicsecurity.com' via host '10.50.0.20' and port '5665'
[2020-12-09 13:18:51 +0000] information/ConfigItem: Activated all objects.
[2020-12-09 13:18:51 +0000] information/ApiListener: Reconnecting to endpoint 'Site-A-Satellite.picnicsecurity.com' via host '10.40.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/ApiListener: Reconnecting to endpoint 'Site-C-Satellite.picnicsecurity.com' via host '10.60.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2020-12-09 13:18:51 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
[2020-12-09 13:18:51 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'Site-C-Satellite.picnicsecurity.com' to [10.60.0.10]:5665
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for zone 'Site-C' to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished reconnecting to endpoint 'Site-C-Satellite.picnicsecurity.com' via host '10.60.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-C-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-C-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'Site-B-Satellite.picnicsecurity.com' to [10.50.0.20]:5665
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished reconnecting to endpoint 'Site-B-Satellite.picnicsecurity.com' via host '10.50.0.20' and port '5665'
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for zone 'Site-B' to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-B-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-B-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'Site-A-Satellite.picnicsecurity.com' to [10.40.0.10]:5665
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished reconnecting to endpoint 'Site-A-Satellite.picnicsecurity.com' via host '10.40.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for zone 'Site-A' to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-A-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-A-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga2' in 0.154311 second(s).
[2020-12-09 13:18:53 +0000] information/ApiListener: New client connection from [10.50.0.10]:41670 (no client certificate)
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/stages/director/bcd05b3c-8f05-4076-9617-bd8a6f1b5b4c (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/files/director/bcd05b3c-8f05-4076-9617-bd8a6f1b5b4c/status (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/files/director/bcd05b3c-8f05-4076-9617-bd8a6f1b5b4c/startup.log (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41670)
[2020-12-09 13:18:57 +0000] information/ApiListener: New client connection from [10.50.0.10]:41672 (no client certificate)
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41672), user: director, agent: , status: OK).
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41672), user: director, agent: , status: OK).
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41672)
[2020-12-09 13:19:00 +0000] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.116667/s (7/min 7/5min 7/15min);
[2020-12-09 13:19:00 +0000] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-12-09 13:19:01 +0000] information/IdoMysqlConnection: Pending queries: 9 (Input: 3/s; Output: 4/s)
[2020-12-09 13:19:11 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:19:17 +0000] information/ApiListener: New client connection from [10.50.0.10]:41674 (no client certificate)
[2020-12-09 13:19:17 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41674), user: director, agent: , status: OK).
[2020-12-09 13:19:17 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41674), user: director, agent: , status: OK).
[2020-12-09 13:19:17 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41674)
[2020-12-09 13:19:21 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:19:31 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:19:38 +0000] information/ApiListener: New client connection from [10.50.0.10]:41676 (no client certificate)
[2020-12-09 13:19:38 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41676), user: director, agent: , status: OK).
[2020-12-09 13:19:38 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41676), user: director, agent: , status: OK).
[2020-12-09 13:19:38 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41676)
[2020-12-09 13:19:58 +0000] information/ApiListener: New client connection from [10.50.0.10]:41680 (no client certificate)
[2020-12-09 13:19:58 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41680), user: director, agent: , status: OK).
[2020-12-09 13:19:58 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41680), user: director, agent: , status: OK).
[2020-12-09 13:19:58 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41680)
[2020-12-09 13:20:01 +0000] information/IdoMysqlConnection: Pending queries: 10 (Input: 3/s; Output: 2/s)
[2020-12-09 13:20:11 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:20:19 +0000] information/ApiListener: New client connection from [10.50.0.10]:41682 (no client certificate)
[2020-12-09 13:20:19 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41682), user: director, agent: , status: OK).
[2020-12-09 13:20:19 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41682), user: director, agent: , status: OK).
[2020-12-09 13:20:19 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41682)
[2020-12-09 13:20:39 +0000] information/ApiListener: New client connection from [10.50.0.10]:41684 (no client certificate)
[2020-12-09 13:20:39 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41684), user: director, agent: , status: OK).
[2020-12-09 13:20:39 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41684), user: director, agent: , status: OK).
[2020-12-09 13:20:39 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41684)
[2020-12-09 13:20:41 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
The things that are standing out to me are
[2020-12-09 13:18:51 +0000] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
(0) Activating object 'command' of type 'ExternalCommandListener'
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-C-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-C-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-B-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-B-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-A-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-A-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
2020-12-09 13:18:53 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41670)
[2020-12-09 13:18:57 +0000] information/ApiListener: New client connection from [10.50.0.10]:41672 (no client certificate)
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41672), user: director, agent: , status: OK).
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41672), user: director, agent: , status: OK).
This log entry seems to repeat a lot.
So as far as the master is concerned it is able to connect to the satellites and push configuration to them.
Looking at Site-A-Satellite most recent log entries I see this
[2020-12-09 13:09:55 +0000] information/ApiListener: New client connection for identity 'IcingaMaster.picnicsecurity.com' from [10.50.0.10]:34390
[2020-12-09 13:09:55 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Sending config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished sending config file updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished syncing endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Applying config update from endpoint 'IcingaMaster.picnicsecurity.com' of zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Received configuration for zone 'Site-A' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Site-A//director/hosts.conf' for zone 'Site-A'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A' (242 Bytes).
[2020-12-09 13:09:55 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (4632 Bytes).
[2020-12-09 13:09:55 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'IcingaMaster.picnicsecurity.com' are equal to production, skipping validation and reload.
[2020-12-09 13:10:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:11:16 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 12/5min 13/15min);
[2020-12-09 13:11:16 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-12-09 13:12:14 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:13:12 +0000] information/ApiListener: New client connection from [10.50.0.10]:34438 (no client certificate)
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/types (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:34438)
[2020-12-09 13:15:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:16:26 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-12-09 13:16:26 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 16/5min 29/15min);
[2020-12-09 13:17:04 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:18:51 +0000] warning/JsonRpcConnection: API client disconnected for identity 'IcingaMaster.picnicsecurity.com'
[2020-12-09 13:18:51 +0000] warning/ApiListener: Removing API client for endpoint 'IcingaMaster.picnicsecurity.com'. 0 API clients left.
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'IcingaMaster.picnicsecurity.com' from [10.50.0.10]:34490
[2020-12-09 13:18:51 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Applying config update from endpoint 'IcingaMaster.picnicsecurity.com' of zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Received configuration for zone 'Site-A' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Site-A//director/hosts.conf' for zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A' (242 Bytes).
[2020-12-09 13:18:51 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (4632 Bytes).
[2020-12-09 13:18:51 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'IcingaMaster.picnicsecurity.com' are equal to production, skipping validation and reload.
[2020-12-09 13:19:44 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:20:44 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:20:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:21:36 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-12-09 13:21:36 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 12/5min 40/15min);
[2020-12-09 13:22:04 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:25:34 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:25:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:26:46 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 12/5min 36/15min);
[2020-12-09 13:26:46 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-12-09 13:27:04 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
The things that stand out to me here are
[2020-12-09 13:05:57 +0000] information/ApiListener: Applying config update from endpoint 'IcingaMaster.picnicsecurity.com' of zone 'master'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A'. Current timestamp '2020-12-09 13:05:53 +0000' (1607519153.953211) >= received timestamp '2020-12-09 13:05:53 +0000' (1607519153.953211).
[2020-12-09 13:05:57 +0000] information/ApiListener: Received configuration for zone 'Site-A' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Site-A//director/hosts.conf' for zone 'Site-A'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A' (242 Bytes).
[2020-12-09 13:05:57 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global'. Current timestamp '2020-12-09 13:05:53 +0000' (1607519153.955098) >= received timestamp '2020-12-09 13:05:53 +0000' (1607519153.955098).
[2020-12-09 13:05:57 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (4632 Bytes).
[2020-12-09 13:05:57 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'IcingaMaster.picnicsecurity.com' are equal to production, skipping validation and reload.
[2020-12-09 13:06:06 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2020-12-09 13:06:06 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-12-09 13:07:24 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:09:22 +0000] warning/JsonRpcConnection: API client disconnected for identity 'IcingaMaster.picnicsecurity.com'
[2020-12-09 13:09:22 +0000] warning/ApiListener: Removing API client for endpoint 'IcingaMaster.picnicsecurity.com'. 0 API clients left.
These logs seem to indicate that Site-A is not applying configuration updates because it thinks it is ahead? But then it seems to apply it anyways so I am not sure.
[2020-12-09 13:13:12 +0000] information/ApiListener: New client connection from [10.50.0.10]:34438 (no client certificate)
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/types (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:34438)
These are the same entries as ones found on IcingaMaster. Not sure what they mean though