Satellite stuck in PENDING state

gkasKaR · March 7, 2022, 10:44am

Hello communitiy,

as the caption describes, I have trouble getting a satellite up and running.

I installed an Icinga instance on an Ubuntu Server 20.04.
After setting everything up, I ran successfully the node wizard for satellites.

Everything works fine except the services - even the hostalive.

pending

My master is also installed on an Ubuntu Server 20.04.
Both the master and satellite are running the Version r2.13.2-1.

Here is the zones.conf from the master:

Here is the zones.conf from the satellite:

Do you have an idea, why it doesn’t work as expected?

Maybe this is a clue: When I inspect the satellite via director I receive on the zones tab a suspicious “The configuration for this object has not been rendered by Icinga Director. You can find it on line 18 in /etc/icinga2/zones.conf”

Greetings and Thanks!
Gabriel

rsx · March 7, 2022, 11:57am

Certificate is signed? Time synchronization is ok? Any hints in the icinga2.log both master and satellite?

gkasKaR · March 7, 2022, 12:12pm

Hello Roland,

I assume, that the certificate is signed, because I generated a token from the master a passed it in to node wizard on the satellite.

A “netstat taupen” outputs an established connection between the master an satellite.

Time synchronization was not right. I just corrected it, but no change.

This is the icinga2.log of the master (with grep IP-Address of the satellite):

[2022-03-07 10:05:26 +0100] information/ApiListener: Reconnecting to endpoint 'sv1005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:05:26 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv1005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:05:36 +0100] information/ApiListener: Reconnecting to endpoint 'sv1005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:05:36 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv1005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:05:39 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:05:39 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:05:39 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:07:20 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:07:20 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:07:20 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:08:48 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:08:48 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:08:48 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:09:35 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33064
[2022-03-07 10:10:01 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:10:01 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:10:01 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:20:37 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33070
[2022-03-07 10:23:00 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:23:00 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:23:00 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:25:06 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:25:06 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:25:06 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:32:46 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33074
[2022-03-07 10:45:10 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:45:10 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:45:10 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:47:10 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 10:47:10 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 10:47:10 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 11:00:28 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 11:00:28 +0100] information/ApiListener: New client connection for identity 'sv2005' to [IP-Address of satllite]:5665
[2022-03-07 11:00:28 +0100] information/ApiListener: Finished reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 11:04:39 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33076
[2022-03-07 11:04:45 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33078
[2022-03-07 11:11:23 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33080
[2022-03-07 11:11:29 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:33082
[2022-03-07 12:59:28 +0100] information/ApiListener: Reconnecting to endpoint 'sv2005' via host 'IP-Address of satllite' and port '5665'
[2022-03-07 12:59:35 +0100] critical/ApiListener: Cannot connect to host 'IP-Address of satllite' on port '5665': Connection refused
[2022-03-07 12:59:37 +0100] information/ApiListener: New client connection for identity 'sv2005' from [::ffff:IP-Address of satllite]:42562

rsx · March 7, 2022, 12:40pm

Sounds weird since your config shows that the satellite shall connect to the master. Anyway, connect is not allowed. API is enabled?

gkasKaR · March 8, 2022, 7:42am

The API ist enabled, on both sides.

I can deploy successfully configs to the satellite via director.

The icinga2.log of the satellite:

[2022-03-08 08:34:01 +0100] warning/JsonRpcConnection: API client disconnected for identity 'sv0002'
[2022-03-08 08:34:01 +0100] warning/ApiListener: Removing API client for endpoint 'sv0002'. 0 API clients left.
[2022-03-08 08:34:01 +0100] information/ApiListener: New client connection for identity 'sv0002' from [::ffff:IP-AddressOfMaster]:47682
[2022-03-08 08:34:01 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'sv0002'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Sending config updates for endpoint 'sv0002' in zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Finished sending config file updates for endpoint 'sv0002' in zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Syncing runtime objects to endpoint 'sv0002'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'sv0002'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'sv0002' in zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Sending replay log for endpoint 'sv0002' in zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Finished sending replay log for endpoint 'sv0002' in zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Finished syncing endpoint 'sv0002' in zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Applying config update from endpoint 'sv0002' of zone 'master'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'sv0002'. Comparing the timestamp and checksums.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/commands.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/dependency_apply.conf' for
zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/dependency_templates.conf'
for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/hostgroups.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_apply.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone
'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/user_templates.conf' for zone 'director-global'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (13236 Bytes).
[2022-03-08 08:34:01 +0100] information/ApiListener: Received configuration for zone 'linux-agents' from endpoint 'sv0002'. Comparing the timestamp and checksums.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/linux-agents//director/service_templates.conf' for zone 'linux-agents'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/linux-agents//director/services.conf' for zone 'linux-agents'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/linux-agents' (303 Bytes).
[2022-03-08 08:34:01 +0100] information/ApiListener: Received configuration for zone 'sv2005' from endpoint 'sv0002'. Comparing the timestamp and checksums.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/sv2005//director/hosts.conf' for zone 'sv2005'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/sv2005' (324 Bytes).
[2022-03-08 08:34:01 +0100] information/ApiListener: Received configuration for zone 'windows-agents' from endpoint 'sv0002'. Comparing the timestamp and checksums.
[2022-03-08 08:34:01 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/windows-agents//director/commands.conf' for zone 'windows-agents'.
[2022-03-08 08:34:01 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/windows-agents' (90719 Bytes).
[2022-03-08 08:34:01 +0100] information/ApiListener: Received configuration updates (4) from endpoint 'sv0002' are different to production, triggering validation and reload.
[2022-03-08 08:34:01 +0100] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log'

The startup.log contains an error too, but I dont really know how to fix this redefinition properly.

[2022-03-08 08:34:01 +0100] information/cli: Icinga application loader (version: r2.13.2-1)
[2022-03-08 08:34:01 +0100] information/cli: Loading configuration file(s).
[2022-03-08 08:34:01 +0100] critical/config: Error: Object 'Linux Disk' of type 'Service' re-defined: in /var/lib/icinga2/api/zones-stage//linux-agents/director/service_templates.conf: 1:0-1:28; previous definition: in /var/lib/icinga2/api/zones-stage//director-global/director/service_templates.conf: 6:1-6:29
Location: in /var/lib/icinga2/api/zones-stage//linux-agents/director/service_templates.conf: 1:0-1:28
/var/lib/icinga2/api/zones-stage//linux-agents/director/service_templates.conf(1): template Service "Linux Disk" {
                                                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//linux-agents/director/service_templates.conf(2):     import "Agent Service"
/var/lib/icinga2/api/zones-stage//linux-agents/director/service_templates.conf(3):
[2022-03-08 08:34:01 +0100] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

Greetings and thanks

rsx · March 8, 2022, 8:15am

gkasKaR:

[2022-03-08 08:34:01 +0100] critical/config: Error: Object 'Linux Disk' of type 'Service' re-defined: in /var/lib/icinga2/api/zones-stage//linux-agents/director/service_templates.conf: 1:0-1:28; previous definition: in /var/lib/icinga2/api/zones-stage//director-global/director/service_templates.conf: 6:1-6:29

The service Linux Disk is defined twice. One in linux-agents and another one in director-global, but I have no idea how this is possible (since we don’t use the director for services). Hopefully, someelse can jump in and explain.

gkasKaR · March 8, 2022, 12:36pm

Thanks for your opinion.

I think that almost all configuration config is done via director-global

So I ask myself rather, why this service template is deployed over the linux-agents zone.
The GUI displays it as an object of director-global:

Can I simply delete the template definition out of “/var/lib/icinga2/api/zones/linux-agents/director/service_templates.conf” from the master?
Or is this the wrong place?

rsx · March 8, 2022, 12:50pm

You can delete it. But most properly it will be recreated with the next deploy run.

gkasKaR · March 8, 2022, 1:31pm

After I deleted the template, it surprisingly wasn’t recreated again.
I assume that the definition in linux-agents was a leftover.

After I restarted and re-deployed everything, the satellite responds valid values.

services

But I really don’t know, why it says that the satellite zone is not connected

rsx · March 8, 2022, 1:46pm

To me it looks like your satellite checks if it is connected to itself.