Icinga2 Distributed monitoring checks by satellites are stuck in"pending"

Satellite

[root@icinga-qa-satellite-01 features-enabled]# tail -f n70 /var/log/icinga2/icinga2.log
tail: cannot open ‘n70’ for reading: No such file or directory
==> /var/log/icinga2/icinga2.log <==
[2021-03-26 13:38:57 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:43:49 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:44:07 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:44:07 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:48:49 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:49:17 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:49:17 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:53:49 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:54:27 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:54:27 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:56:54 +0100] information/Application: Received request to shut down.
[2021-03-26 13:56:54 +0100] information/Application: Shutting down...
[2021-03-26 13:56:54 +0100] information/CheckerComponent: 'checker' stopped.
[2021-03-26 13:56:54 +0100] information/ApiListener: 'api' stopped.
[2021-03-26 13:56:54 +0100] information/FileLogger: 'main-log' started.
[2021-03-26 13:56:54 +0100] information/ApiListener: 'api' started.
[2021-03-26 13:56:54 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-03-26 13:56:54 +0100] information/CheckerComponent: 'checker' started.
[2021-03-26 13:56:54 +0100] information/ConfigItem: Activated all objects.
[2021-03-26 13:56:54 +0100] information/ApiListener: Reconnecting to endpoint 'icinga-qa.domain' via host 'icinga-qa.domain' and port '5665'
[2021-03-26 13:56:55 +0100] information/ApiListener: New client connection for identity 'icinga-qa.domain' from [10.116.219.2]:50366
[2021-03-26 13:56:55 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icinga-qa.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Sending config updates for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga-qa.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-qa.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Sending replay log for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished syncing endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Applying config update from endpoint 'icinga-qa.domain' of zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Received configuration for zone 'global-templates' from endpoint 'icinga-qa.domain'. Comparing the timestamp and checksums.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/commands.conf' for zone 'global-templates'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/services.conf' for zone 'global-templates'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/templates.conf' for zone 'global-templates'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/global-templates' (39012 Bytes).
[2021-03-26 13:56:55 +0100] information/ApiListener: Received configuration for zone 'icinga-qa-satellite-01.domain' from endpoint 'icinga-qa.domain'. Comparing the timestamp and checksums.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain//_etc/health.conf' for zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain//_etc/hosts.conf' for zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain//_etc/pet.conf' for zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain' (1753 Bytes).
[2021-03-26 13:56:55 +0100] information/ApiListener: Received configuration updates (2) from endpoint 'icinga-qa.domain' are different to production, triggering validation and reload.
[2021-03-26 13:56:55 +0100] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log'
[2021-03-26 13:56:59 +0100] information/ApiListener: New client connection for identity 'icinga-qa.domain' to [10.116.219.2]:5665
[2021-03-26 13:56:59 +0100] information/ApiListener: Finished reconnecting to endpoint 'icinga-qa.domain' via host 'icinga-qa.domain' and port '5665'
[2021-03-26 13:57:04 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:57:04 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);

After cleaning the satellite, the services still stay in “pending”.