Satellite "No data received on new API" Error on Master

Hello Community,

i have a problem with the communication between my icinga master and one of my satellites.
All checks working but sometimes delayed and the “next check time” moves to a negative value (only on this satellite).

the following log is from the master (/var/log/icinga2/icinga2.log) and shows what happen if i restart the satellite.

[2020-02-05 11:16:54 +0100] information/ApiListener: New client connection for identity '**SATELLITE**' from [IP_SATELLITE]:51538
[2020-02-05 11:16:54 +0100] information/ApiListener: Sending config updates for endpoint '**SATELLITE**' in zone '**SATELLITE.ZONE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Syncing configuration files for zone  '**SATELLITE.ZONE**' to endpoint '**SATELLITE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Syncing configuration files for global zone 'GLOBALZONE' to endpoint '**SATELLITE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Finished sending config file updates for endpoint '**SATELLITE**' in zone '**SATELLITE.ZONE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Syncing runtime objects to endpoint '**SATELLITE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Finished syncing runtime objects to endpoint '**SATELLITE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Finished sending runtime config updates for endpoint '**SATELLITE**' in zone '**SATELLITE.ZONE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Sending replay log for endpoint '**SATELLITE**' in zone '**SATELLITE.ZONE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Finished sending replay log for endpoint '**SATELLITE**' in zone '**SATELLITE.ZONE**'.
[2020-02-05 11:16:54 +0100] information/ApiListener: Finished syncing endpoint '**SATELLITE**' in zone '**SATELLITE.ZONE**'.
[2020-02-05 11:16:54 +0100] information/JsonRpcConnection: Received certificate request for CN '**SATELLITE**' signed by our CA.
[2020-02-05 11:16:54 +0100] information/JsonRpcConnection: The certificate for CN '**SATELLITE**' is valid and uptodate. Skipping automated renewal.

[2020-02-05 11:17:01 +0100] information/ApiListener: New client connection for identity '**SATELLITE**' from [IP_SATELLITE]:51540
[2020-02-05 11:17:01 +0100] warning/ApiListener: No data received on new API connection from [IP_SATELLITE]:51540 for identity '**SATELLITE**'. Ensure that the remote endpoints are properly configured in a cluster setup.
[2020-02-05 11:17:11 +0100] information/ApiListener: New client connection for identity '**SATELLITE**' from [IP_SATELLITE]:51548
[2020-02-05 11:17:11 +0100] warning/ApiListener: No data received on new API connection from [IP_SATELLITE]:51548 for identity '**SATELLITE**'. Ensure that the remote endpoints are properly configured in a cluster setup.
[2020-02-05 11:17:21 +0100] information/ApiListener: New client connection for identity '**SATELLITE**' from [IP_SATELLITE]:51550

Hi,

sometimes we had this also. After searching arround we found out that “active checks” were disabled on the host in icingaweb2.

image

We don’t know why, but we probably clicked incorrectly once.

Hello Stevie,

thanks for your answer.
Active Checks are still activated.

Hi,

please share the zones.conf from both the master and the satellite. Also, include the output of icinga2 --version.

Cheers,
Michael

I have an other idea which we had in the past: check your date and time from your servers, if it’s not the same it could also be a problem.

I used the Icinga Director for the Master and Satellites config.

/etc/icinga2/zones.conf (Master)

object Endpoint "**MASTER**" {
	host ="**MASTER**"
}

object Zone "**MASTER-ZONE**" {
	 endpoints = [ "**MASTER**" ]
}

/var/lib/icinga2/api/packages/director/*****/zones.d/MASTER/zones.conf

object Zone "**GLOBAL-ZONE**" {
    parent = "**MASTER**"
    global = true
}

object Zone "**SATELLITE-ZONE**" {
    parent = "**MASTER**"
    endpoints = [ "**SATELLITE**" ]
}
... other Satellites

/var/lib/icinga2/api/packages/director/*****/zones.d/MASTER/endpoints.conf

object Endpoint "**SATELLITE-1**" {
    host = "SATELLITE-1**"
}

object Endpoint "**SATELLITE-2**" {
    host = "SATELLITE-2**"
}
...

/etc/icinga2/zones.conf (Satellite)

object Endpoint "**MASTER**" {
	host ="**MASTER**"
}
object Zone "**MASTER-ZONE**" {
	 endpoints = [ "**MASTER**" ]
}
object Endpoint "**SATELLITE**" {
}
object Zone "**SATELLITE-ZONE**" {
	endpoints = [ "**SATELLITE**" ]
	parent = "**MASTER**"
}
object Zone "**GLOBAL-ZONE**" {
global = true
}

icinga2 - The Icinga 2 network monitoring daemon (version: r2.11.2-1)

Copyright (c) 2012-2020 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 9 (stretch)
  Kernel: Linux
  Kernel version: 4.9.0-8-amd64
  Architecture: x86_64

@stevie-sy good idea but the times are correct.

So you are using the Infrastructure tab in the Director to manage zones and endpoints?

at first i used the icinga2 node wizard for the master + satellites and than i imported with the kickstart assistant the zone+endpoint config.
After this step i had problems with re-definied errors in the zones.conf and zones.d files.
I removed the satellite configs on the master at the /etc/icinga2/zones.conf and the problem was fixed.

All checks on the satellites working and i have no problems on the other satellites.
The delayed check problem is not the whole time.

icinga2 daemon -C output (satellite)

[2020-02-06 15:41:48 +0100] information/cli: Icinga application loader (version: r2.10.3-1)
[2020-02-06 15:41:48 +0100] information/cli: Loading configuration file(s).
[2020-02-06 15:41:49 +0100] information/ConfigItem: Committing config item(s).
[2020-02-06 15:41:49 +0100] information/ApiListener: My API identity: **SATELLITE**
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 SyslogLogger.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 User.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 3 Zones.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 373 Services.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 23 Hosts.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 746 Notifications.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 2 FileLoggers.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 225 CheckCommands.
[2020-02-06 15:41:50 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2020-02-06 15:41:50 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2020-02-06 15:41:50 +0100] information/cli: Finished validating the configuration file(s).

Satellite output (icinga2 object list --type Endpoint)

Object '**MASTER**' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 1:0-1:33
  * __name = "**MASTER**"
  * host = "**MASTER**"
    % = modified in '/etc/icinga2/zones.conf', lines 2:2-2:26
  * log_duration = 86400
  * name = "**MASTER**"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 33
    * last_line = 1
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "**MASTER**" ]
    % = modified in '/etc/icinga2/zones.conf', lines 1:0-1:33
  * type = "Endpoint"
  * zone = ""

Object '**SATELLITE**' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 9:1-9:34
  * __name = "**SATELLITE**"
  * host = ""
  * log_duration = 86400
  * name = "**SATELLITE**"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 34
    * last_line = 9
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "**SATELLITE**" ]
    % = modified in '/etc/icinga2/zones.conf', lines 9:1-9:34
  * type = "Endpoint"
  * zone = ""

Master output (icinga2 object list --type Endpoint)

Object '**MASTER**' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 1:0-1:33
  * __name = "**MASTER**"
  * host = "**MASTER**"
    % = modified in '/etc/icinga2/zones.conf', lines 2:2-2:25
  * log_duration = 86400
  * name = "**MASTER**"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 33
    * last_line = 1
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "**MASTER**" ]
    % = modified in '/etc/icinga2/zones.conf', lines 1:0-1:33
  * type = "Endpoint"
  * zone = ""

Object '**SATELLITE**' of type 'Endpoint':
  % declared in '/var/lib/icinga2/api/packages/director/1f938237-cf69-4092-946d-cd9cfa041fa4/zones.d    /**MASTER**/endpoints.conf', lines 1:0-1:33
  * __name = "**SATELLITE**"
  * host = "**SATELLITE**"
    % = modified in '/var/lib/icinga2/api/packages/director/1f938237-cf69-4092-946d-cd9cfa041fa4    /zones.d/**MASTER**/endpoints.conf', lines 2:5-2:29
  * log_duration = 86400
  * name = "**SATELLITE**"
  * package = "director"
  * port = "5665"
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 33
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/director/1f938237-cf69-4092-946d-cd9cfa041fa4/zones.d    /**MASTER**/endpoints.conf"
  * templates = [ "**SATELLITE**" ]
    % = modified in '/var/lib/icinga2/api/packages/director/1f938237-cf69-4092-946d-cd9cfa041fa4/zones.d/**MASTER**/endpoints.conf', lines 1:0-1:33
  * type = "Endpoint"
  * zone = "**MASTER-ZONE**"

Imho the master/satellite details should be stored in zones.conf and outside the infrastructure tab inside the Director. Their usage there is experimental and won’t necessarily work with deployments and cluster config syncs.

See the troubleshooting entry here.

Cheers,
Michael

1 Like

Hi Michael,

thanks for the link.
my cluster config is now stored at the local zone.conf.
the endpoints and zones were changed to external objects at the director.
but the problem was still there.
i reinstalled the satellite and now it works without problems or warnings at the log.

thanks for your help.

1 Like