Satellite doesn´t check clients

Service Monitoring
, , ,
1

Hello everybody,

in my configuration is one master and two satellites, each satellite check one place.
When I check something with my master everything is ok, but when i check something with one of the satellite, it doesn’t check something… what can I do?

OS: Ubuntu 20.04.1 LTS
Version Icinga-Master: r.2.11.4-1
Version Icinga-Satellite: r.2.11.4-1

Config Master:

object Endpoint "icinga2-master01" {\
}

object Zone "master" {\
endpoints = [ "icinga2-master01" ]\
}

object Endpoint "icinga2-satellite01.localdomain" {\
host = "icinga2-satellite01.localdomain"\
port = "5665"\
}

object Endpoint "icinga2-satellite51.localdomain" {\
host = "icinga2-satellite51.localdomain"\
port = "5665"\
}

object Zone "satellite-sme" {\
endpoints = [ "icinga2-satellite01.localdomain" ]\
parent = "master"\
}

object Zone "satellite-lue" {\
endpoints = [ "icinga2-satellite51.localdomain" ]\
parent = "master"\
}

object Zone "global-templates" {\
global = true\
}

object Zone "director-global" {\
global = true\
}

Config Satellite01:

object Endpoint "icinga2-master01.localdomain" {\
host = "icinga2-master01.localdomain"\
port = "5665"\
}

object Zone "master" {\
endpoints = [ "icinga2-master01.localdomain" ]\
}

object Endpoint "icinga2-satellite01.localdomain" {\
host = "icinga2-satellite01.localdomain"\
port = "5665"\
}

object Zone "satellite-sme" {\
endpoints = [ "icinga2-satellite01.localdomain" ]\
parent = "master"\
}

object Zone "global-templates" {\
global = true\
}

object Zone "director-global" {\
global = true\
}

Config Satellite51:

object Endpoint "icinga2-master01.localdomain" {\
host = "icinga2-master01.localdomain"\
port = "5665"\
}

object Zone "master" {\
endpoints = [ "icinga2-master01.localdomain" ]\
}

object Endpoint "icinga2-satellite51.localdomain" {\
host = "icinga-satellite51.localdomain"\
port = "5665"\
}

object Zone "satellite-lue" {\
endpoints = [ "icinga2-satellite51.localdomain" ]\
parent = "master"\
}

object Zone "global-templates" {\
global = true\
}

object Zone "director-global" {\
global = true\
}

I hope that somebody can help me…

Greetings

2

Your zones.conf are looking ok (having a quick look). Are your certificates ok? Any hints in icinga.logs? In general, it’s best practice to monitor if zones are connected using cluster and/or cluster-zone.

BTW: Please use markdown to format your text as described here.

3

The icinga2.log only says:

warning/ApiListener: No data received on new API connection from [*******]:41800 for identity ‘icinga2-satellite01.localdomain’. Ensure that the remote endpoints are properly configured in a cluster setup.

[2020-07-27 13:54:47 +0000] information/ApiListener: New client connection for identity ‘icinga2-satellite51.localdomain’ from [*******]:54356

[2020-07-27 13:54:47 +0000] warning/ApiListener: No data received on new API connection from [*******]:54356 for identity ‘icinga2-satellite51.localdomain’. Ensure that the remote endpoints are properly configured in a cluster setup.

[2020-07-27 13:54:52 +0000] information/ApiListener: New client connection for identity ‘icinga2-satellite01.localdomain’ from [*******]:41802

[2020-07-27 13:54:52 +0000] warning/ApiListener: No data received on new API connection from [*******]:41802 for identity ‘icinga2-satellite01.localdomain’. Ensure that the remote endpoints are properly configured in a cluster setup.

[2020-07-27 13:54:57 +0000] information/ApiListener: New client connection for identity ‘icinga2-satellite51.localdomain’ from [*******]:54358

[2020-07-27 13:54:57 +0000] warning/ApiListener: No data received on new API connection from [*******]:54358 for identity ‘icinga2-satellite51.localdomain’. Ensure that the remote endpoints are properly configured in a cluster setup.

[2020-07-27 13:54:59 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2, rate: 3.3/s (198/min 1024/5min 3085/15min);

[2020-07-27 13:55:02 +0000] information/ApiListener: New client connection for identity ‘icinga2-satellite01.localdomain’ from [*******]:41804

[2020-07-27 13:55:02 +0000] warning/ApiListener: No data received on new API connection from [*******]:41804 for identity ‘icinga2-satellite01.localdomain’. Ensure that the remote endpoints are properly configured in a cluster setup.

How can i check the certificates?

I use cluster-zones in my checks, when i give the master-zone (cluster) then the check is ok, but when i use the satellite-zone it doesn’t do…

I hope that the markdown now works.

4

I found something else, when i check my endpoint (in the satellite-zone) with a service (in the satellite-zone) i dont have a check source.

5

The check source will be filled once a check was scheduled.

You’ve defined icinga2-master01.localdomain as endpoint at your satellites, but icinga2-master01 at your master. Is this a copy and paste error?

6

I changed the config, it was a copy paste error, now i have other problems, my master only communicate with only one satellite and on this satellite i have a certificate problem… the second satellite does not work or communicate with the master…

Log icinga2-master01:

[2020-07-28 07:08:23 +0000] information/ApiListener: New client connection for identity ‘icinga2-satellite51.localdomain’ from [***************]:38534

[2020-07-28 07:08:23 +0000] information/ApiListener: Sending config updates for endpoint ‘icinga2-satellite51.localdomain’ in zone ‘satellite-lue’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Syncing configuration files for zone ‘satellite-lue’ to endpoint ‘icinga2-satellite51.localdomain’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Syncing configuration files for global zone ‘global-templates’ to endpoint ‘icinga2-satellite51.localdomain’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘icinga2-satellite51.localdomain’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Finished sending config file updates for endpoint ‘icinga2-satellite51.localdomain’ in zone ‘satellite-lue’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Syncing runtime objects to endpoint ‘icinga2-satellite51.localdomain’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Finished syncing runtime objects to endpoint ‘icinga2-satellite51.localdomain’.

[2020-07-28 07:08:23 +0000] information/ApiListener: Finished sending runtime config updates for endpoint ‘icinga2-satellite51.localdomain’ in zone ‘satellite-lue’.

Log icinga2-satellite01:

[2020-07-28 08:03:14 +0200] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’

[2020-07-28 08:03:24 +0200] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);

[2020-07-28 08:03:24 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);

Log icinga2-satellite51:

[2020-04-02 16:51:22 +0200] information/ApiListener: Reconnecting to endpoint ‘icinga2-master01’ via host ‘icinga2-master01’ and port ‘5665’

[2020-04-02 16:51:22 +0200] warning/ApiListener: Certificate validation failed for endpoint ‘icinga2-master01’: code 9: certificate is not yet valid

[2020-04-02 16:51:22 +0200] information/ApiListener: New client connection for identity ‘icinga2-master01’ to [172.22.120.164]:5665 (certificate validation failed: code 9: certificate is not yet valid)

[2020-04-02 16:51:22 +0200] information/ApiListener: Finished reconnecting to endpoint ‘icinga2-master01’ via host ‘icinga2-master01’ and port ‘5665’

7

Hi.

To this:

from the second log:
Could you check your time-settings?
There seems to be a difference between the machines.

You can also check your certificate dates by:

openssl s_client -connect 127.0.0.1:5665 | openssl x509 -noout -dates
# (exchange 127.0.0.1 by the master-IP if you don't execute this directly on the master)

Greetings.

8

After I configured the time on all servers with a ntp, i dont have certificate problems anymore. Now i have another problem:

Log icinga2-satellite51:

[2020-07-28 10:19:42 +0200] information/ApiListener: Received configuration for zone ‘satellite-lue’ from endpoint ‘icinga2-master01’. Comparing the timestamp and checksums.

[2020-07-28 10:19:42 +0200] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/satellite-lue//director/host_templates.conf’ for zone ‘satellite-lue’.

[2020-07-28 10:19:42 +0200] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/satellite-lue//director/hosts.conf’ for zone ‘satellite-lue’.

[2020-07-28 10:19:42 +0200] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones-stage/satellite-lue’ (798 Bytes).

[2020-07-28 10:19:42 +0200] information/ApiListener: Received configuration updates (3) from endpoint ‘icinga2-master01’ are different to production, triggering validation and reload.

[2020-07-28 10:19:42 +0200] critical/ApiListener: Config validation failed for staged cluster config sync in ‘/var/lib/icinga2/api/zones-stage/’. Aborting. Logs: ‘/var/lib/icinga2/api/zones-stage//startup.log’

[2020-07-28 10:20:25 +0200] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’

[2020-07-28 10:20:34 +0200] critical/ApiListener: Cannot connect to host ‘icinga2-master01’ on port ‘5665’: Connection timed out

[2020-07-28 10:20:34 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);

[2020-07-28 10:20:34 +0200] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);

9
  • Is icinga2 running at icinga2-master01?
  • Is icinga2 listening on port 5665 at icinga2-master01?
  • Can you connect from icinga2-satellite51 to icinga2-master01 port 5665 manually?
    E.g. by:
nc icinga2-master01 5665 -v
10
  • root@icinga2-master01:~# systemctl status icinga2
    ● icinga2.service - Icinga host/service/network monitoring system
    Loaded: loaded (/lib/systemd/system/icinga2.service; disabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/icinga2.service.d
    └─limits.conf
    Active: active (running) since Tue 2020-07-28 10:19:56 CEST; 1h 33min ago
    Process: 1260 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
    Main PID: 1276 (icinga2)
    Tasks: 65
    Memory: 97.9M
    CGroup: /system.slice/icinga2.service
    ├─1276 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
    ├─1332 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
    ├─1944 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
    ├─5429 /usr/lib/nagios/plugins/check_ping -H icinga2-satellite51.localdomain -c 200,15% -w 100,5%
    ├─5430 /bin/ping -4 -n -U -w 10 -c 5 icinga2-satellite51.localdomain
    ├─5431 /usr/lib/nagios/plugins/check_apt
    └─5433 /usr/bin/apt-get -o Debug::NoLocking=true -s -qq upgrade

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 3 Endpoints.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 2 ApiUsers.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 235 CheckCommands.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 2 UserGroups.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 466 Users.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ConfigItem: Instantiated 10 Services.

Jul 28 10:19:56 icinga2-master01 icinga2[1333]: [2020-07-28 10:19:56 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’

Jul 28 10:19:56 icinga2-master01 icinga2[1276]: [2020-07-28 10:19:56 +0200] information/cli: Closing console log.

  • root@icinga2-satellite51:~# nc icinga2-master01 5665 -v
    Connection to icinga2-master01 5665 port [tcp/*] succeeded!

  • icinga2 858 nagios 18u IPv4 27475 0t0 TCP *:5665 (LISTEN)

11

I think i dont have any certificate problems. My only problem is that the satellites dosen´t check anything… maybe you have some solutions for this problem?

Greetings

12

Hi.

In general it is advisable to check the log files on both sides (in this case the master and satellite) for corresponding errors and possibly check the zones.conf on master and satellite again.

As long as this error occours, the communication will not work.

Greetings.

13

I found the error, now all is working!

14

It would have been helpful to others if you shared your findings.