Icinga2 Client could not connect to Master

Hello everyone,

I have a client setup problem (Client could not connect to Master) after Icinga2 Node Wizard has been run on the client

The following configuration is available: Master - Master Setup

Master 1

Icinga2 Version 2.10.4

Icingaweb 2.6.3

Icinga Director Master

IP 10.0.0.1

RedHat Enterprise 7

Firewall disabled - SELinux disabled

Master 2

Icinga2 Version 2.10.4

Icingaweb 2.6.3

The Icinga Director is only running on Master 1

IP 10.0.0.2

RedHat Enterprise 7

Firewall disabled - SELinux disabled

Zones.cfg Master 1

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-04-13 18:17:26 +0200
 */

object Endpoint "master1.test.net" {
}

object Endpoint "master2.test.net" {
        host = "10.0.0.2"
}

object Zone "master" {
        endpoints = [ "master1.test.net", "master2.test.net" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Zones.cfg Master 2

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-04-13 18:22:13 +0200
 */

object Endpoint "master2.test.net" {
}

object Endpoint "master1.test.net" {
        host = "10.0.0.1"
}

object Zone "master" {
        endpoints = [ "master1.test.net", "master2.test.net" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

When you run the Icinga2 Node Wizard, enter:

icinga2 feature list
Disabled features: checker command compatlog elasticsearch gelf graphite influxdb livestatus mainlog notification opentsdb perfdata statusdata syslog
Enabled features: api debuglog
icinga2 node wizard
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]:

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [mndemuclxr099.mn-man.biz]: client1.test.net

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): master1.test.net

Do you want to establish a connection to the parent node from this node? [Y/n]:
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 10.0.0.1
Master/Satellite endpoint port [5665]: 5665

Add more master/satellite endpoints? [y/N]:
Parent certificate information:

 Subject:     CN = master1.test.net
 Issuer:      CN = Icinga CA
 Valid From:  Apr 13 16:17:18 2019 GMT
 Valid Until: Apr  9 16:17:18 2034 GMT
 Fingerprint: AE DD 48 28 EA EE 7B 8E 23 FE 22 0C D5 D7 A5 FC 8D 0E F0 FB

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'mndemuclxr099.mn-man.biz'):

No ticket was specified. Please approve the certificate signing request manually
on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...

Local zone name [client1.test.net]:
Parent zone name [master]:

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]:

Do you want to disable the inclusion of the conf.d directory [Y/n]:
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

systemctl restart icinga2

Zones.cfg Client 1

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-05-01 22:08:24 +0200
 */

object Endpoint "master1.test.net" {
        host = "10.0.0.1"
        port = "5665"
}

object Endpoint "master2.test.net" {
        host = "10.0.0.2"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "master1.test.net", "master2.test.net" ]
}

object Endpoint "client1.test.net" {
}

object Zone "client1.test.net" {
        endpoints = [ "client1.test.net" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

When running icinga2 daemon -C

icinga2 daemon -C
[2019-05-01 22:20:09 +0200] information/cli: Icinga application loader (version: r2.10.4-1)
[2019-05-01 22:20:09 +0200] information/cli: Loading configuration file(s).
[2019-05-01 22:20:09 +0200] information/ConfigItem: Committing config item(s).
[2019-05-01 22:20:09 +0200] information/ApiListener: My API identity: client1.test.net
[2019-05-01 22:20:09 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-05-01 22:20:09 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2019-05-01 22:20:09 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2019-05-01 22:20:09 +0200] information/ConfigItem: Instantiated 4 Zones.
[2019-05-01 22:20:09 +0200] information/ConfigItem: Instantiated 3 Endpoints.
[2019-05-01 22:20:09 +0200] information/ConfigItem: Instantiated 218 CheckCommands.
[2019-05-01 22:20:09 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-05-01 22:20:09 +0200] information/cli: Finished validating the configuration file(s).

The result of icinga-diagnostics.sh

./icinga-diagnostics.sh
### Icinga 2 Diagnostics ###
# Version: 0.1.0
# Run on client1.test.net at Wed May  1 22:37:19 CEST 2019

Running as root

## OS ##

OS Version: Red Hat Enterprise Linux Server release 7.6 (Maipo)
Hypervisor: Running virtually on a vmware hypervisor
CPU cores: 2
RAM: 7.6G

### Top output ###

top - 22:37:19 up 13 days,  7:00,  1 user,  load average: 0.18, 0.09, 0.07
Tasks: 156 total,   1 running, 155 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.5 us, 10.0 sy,  0.0 ni, 87.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8009564 total,  5029836 free,   796352 used,  2183376 buff/cache
KiB Swap:  4194300 total,  4194300 free,        0 used.  6851448 avail Mem

SELinux: Permissive
Firewall: active

# Icinga 2 #

## Packages: ##

Icinga 2  Version     : 2.10.4

Done checking packages. See Anomaly section if something odd was found.

Features:
Disabled features: checker command compatlog elasticsearch gelf graphite influxdb livestatus mainlog notification opentsdb perfdata statusdata syslog
Enabled features: api debuglog

Check intervals:

Used commands (numbers are relative to each other, not showing configured objects):

[2019-05-01 22:37:21 +0200] information/cli: Icinga application loader (version: r2.10.4-1)
[2019-05-01 22:37:21 +0200] information/cli: Loading configuration file(s).
[2019-05-01 22:37:22 +0200] information/ConfigItem: Committing config item(s).
[2019-05-01 22:37:22 +0200] information/ApiListener: My API identity: mndemuclxr099.mn-man.biz
[2019-05-01 22:37:22 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-05-01 22:37:22 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2019-05-01 22:37:22 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2019-05-01 22:37:22 +0200] information/ConfigItem: Instantiated 4 Zones.
[2019-05-01 22:37:22 +0200] information/ConfigItem: Instantiated 3 Endpoints.
[2019-05-01 22:37:22 +0200] information/ConfigItem: Instantiated 218 CheckCommands.
[2019-05-01 22:37:22 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-05-01 22:37:22 +0200] information/cli: Finished validating the configuration file(s).

# Icinga Web 2 #

Icinga Web 2 is not installed

# Anomalies found #

* At least one php.ini file has no valid timezone setting
* NTP is not synchronized

Total count of detected anomalies: 2

The firewall daemon is running but all TCP and UDP ports are open.

firewall-cmd --zone = public --permanent --add-port = 0-65535 / tcp
firewall-cmd - reload
firewall-cmd --zone = public --permanent --add-port = 0-65535 / udp
firewall-cmd - reload

Maybe someone has a tip what goes wrong here, sometimes it works after the third reinstallation of Icinga2 and the 10 times Icinga2 Node Wizard sometimes but not at all. The error only occurs with checks that are executed via port 5665, such as check_load or check_disk …
Port 5665 is open on all servers, check_ping and hostalive are working. The Icinga2 client also runs on some systems, all are installed exactly the same and all have the same software installed.

It also does not work during setup (Icinga Node Wizard) on the client with the input of 2 master servers.

Looking at your trip through the wizard, it looks like you’re not pasting a ticket from the master in. Are you signing the request manually? If so, the node wizard can be discarded with a small script tuned for your env. For a basic first setup though, that is the easiest way to make sure your certs are setup right. If you set the log level to debug on your master, you’ll see if it’s screaming about cert errors trying to connect to the client.

You’ll also want to make sure you’re revising the client’s zones.conf to pull out the host and port entries of the parent nodes. The only real point anymore to telling it that you want it to connect to the parent is so it can do that ticket exchange. I recall reading this in the documentation somewhere, but at the moment can’t find it. So like:

Zones.cfg Client 1

/*

    Generated by Icinga 2 node setup commands
    on 2019-05-01 22:08:24 +0200
    */

object Endpoint “master1.test.net” {
}

object Endpoint “master2.test.net” {
}

object Zone “master” {
endpoints = [ “master1.test.net”, “master2.test.net” ]
}

object Endpoint “client1.test.net” {
}

object Zone “client1.test.net” {
endpoints = [ “client1.test.net” ]
parent = “master”
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

The standard setup these days is the masters connect to the satellites connect to the clients (or masters direct to clients as in this case). If you leave that host address on both sides, they’ll sometimes both try to play boss. I also don’t see the checker feature enabled, that could be an issue, as far as I can see reviewing some of my clients here (only looking at my home network atm) seems that was on by default.

I don’t see the host/zone/endpoint entries for the client in the master’s config, just its own. Did you add that after the fact, or was that skipped?

Sorry if that’s an avalanche of questions, you provided a lot of details. If you’re handling any of that with Director, hopefully someone else pops in, I’m not much help with it.

On all clients only the modules Api and Debug are switched on. The client should also automatically receive its certificate from the master. This will be handed over to the Node Wizard. Here is a short excerpt from the debug log of client 1:

[2019-05-02 08:00:18 +0200] debug/EndpointDbObject: update is_connected=1 for endpoint 'master2.test.net'
[2019-05-02 08:00:18 +0200] notice/JsonRpcConnection: Received 'icinga::Hello' message from 'master2.test.net'
[2019-05-02 08:00:18 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'master2.test.net'.
[2019-05-02 08:00:18 +0200] information/ApiListener: Sending config updates for endpoint 'master2.test.net' in zone 'master'.
[2019-05-02 08:00:18 +0200] information/ApiListener: Finished sending config file updates for endpoint 'master2.test.net' in zone 'master'.
[2019-05-02 08:00:18 +0200] information/ApiListener: Syncing runtime objects to endpoint 'master2.test.net'.
[2019-05-02 08:00:18 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'master2.test.net'.
[2019-05-02 08:00:18 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'master2.test.net' in zone 'master'.
[2019-05-02 08:00:20 +0200] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2019-05-02 08:00:20 +0200] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0.15/s (9/min 33/5min 35/15min);
[2019-05-02 08:00:20 +0200] information/WorkQueue: #9 (JsonRpcConnection, #0) items: 0, rate: 0.0833333/s (5/min 16/5min 16/15min);
[2019-05-02 08:00:20 +0200] debug/ApiListener: Not connecting to Endpoint 'master2.test.net' because we're already connected to it.
[2019-05-02 08:00:20 +0200] debug/ApiListener: Not connecting to Endpoint 'master1.test.net' because we're already connected to it.
[2019-05-02 08:00:20 +0200] debug/ApiListener: Not connecting to Endpoint 'client1.test.net' because that's us.
[2019-05-02 08:00:20 +0200] notice/ApiListener: Current zone master: client1.test.net
[2019-05-02 08:00:20 +0200] notice/ApiListener: Connected endpoints: master2.test.net (3) and master1.test.net (7)
[2019-05-02 08:00:20 +0200] information/WorkQueue: #10 (JsonRpcConnection, #1) items: 0, rate: 0.0666667/s (4/min 15/5min 15/15min);
[2019-05-02 08:00:27 +0200] debug/ThreadPool: Spawning worker thread.
[2019-05-02 08:00:27 +0200] information/ApiListener: New client connection for identity 'master1.test.net' from [10.0.0.3]:37552
[2019-05-02 08:00:27 +0200] notice/ApiListener: New JSON-RPC client
[2019-05-02 08:00:27 +0200] debug/EndpointDbObject: update is_connected=1 for endpoint 'master1.test.net'
[2019-05-02 08:00:27 +0200] notice/JsonRpcConnection: Received 'icinga::Hello' message from 'master1.test.net'
[2019-05-02 08:00:27 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'master1.test.net'.
[2019-05-02 08:00:27 +0200] information/ApiListener: Sending config updates for endpoint 'master1.test.net' in zone 'master'.
[2019-05-02 08:00:27 +0200] information/ApiListener: Finished sending config file updates for endpoint 'master1.test.net' in zone 'master'.
[2019-05-02 08:00:27 +0200] information/ApiListener: Syncing runtime objects to endpoint 'master1.test.net'.
[2019-05-02 08:00:27 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'master1.test.net'.
[2019-05-02 08:00:27 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'master1.test.net' in zone 'master'.
[2019-05-02 08:00:28 +0200] notice/ThreadPool: Thread pool; current: 2; adjustment: -1
[2019-05-02 08:00:28 +0200] debug/ThreadPool: Killing worker thread.
[2019-05-02 08:00:30 +0200] debug/ApiListener: Not connecting to Endpoint 'master2.test.net' because we're already connected to it.
[2019-05-02 08:00:30 +0200] debug/ApiListener: Not connecting to Endpoint 'master1.test.net' because we're already connected to it.
[2019-05-02 08:00:30 +0200] debug/ApiListener: Not connecting to Endpoint 'client1.test.net' because that's us.
[2019-05-02 08:00:30 +0200] notice/ApiListener: Current zone master: client1.test.net
[2019-05-02 08:00:30 +0200] notice/ApiListener: Connected endpoints: master2.test.net (3) and master1.test.net (8)
[2019-05-02 08:00:31 +0200] notice/ThreadPool: Pool #1: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.0333693%
[2019-05-02 08:00:31 +0200] notice/ThreadPool: Pool #2: Pending tasks: 0; Average latency: 0ms; Threads: 4; Pool utilization: 0.122666%
[2019-05-02 08:00:31 +0200] information/JsonRpcConnection: No messages for identity 'master1.test.net' have been received in the last 60 seconds.
[2019-05-02 08:00:31 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master1.test.net'
[2019-05-02 08:00:31 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master1.test.net'
[2019-05-02 08:00:31 +0200] warning/ApiListener: Removing API client for endpoint 'master1.test.net'. 7 API clients left.
[2019-05-02 08:00:31 +0200] debug/EndpointDbObject: update is_connected=1 for endpoint 'master1.test.net'
[2019-05-02 08:00:31 +0200] warning/ApiListener: Removing API client for endpoint 'master1.test.net'. 7 API clients left.
[2019-05-02 08:00:31 +0200] debug/EndpointDbObject: update is_connected=1 for endpoint 'master1.test.net'
[2019-05-02 08:00:31 +0200] information/JsonRpcConnection: No messages for identity 'master1.test.net' have been received in the last 60 seconds.
[2019-05-02 08:00:31 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master1.test.net'
[2019-05-02 08:00:31 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master1.test.net'


Here is an excerpt from the logfile of master 1 with the data from client 1:

tail -f /var/log/icinga2/debug.log | grep client1
[2019-05-02 08:06:27 +0200] information/ApiListener: Reconnecting to endpoint 'client1.test.net' via host 'client1.test.net' and port '5665'
[2019-05-02 08:06:27 +0200] warning/ApiListener: Certificate validation failed for endpoint 'client1.test.net': code 18: self signed certificate
[2019-05-02 08:06:27 +0200] information/ApiListener: New client connection for identity 'client1.test.net' to [10.0.0.3]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-05-02 08:06:27 +0200] information/ApiListener: Finished reconnecting to endpoint 'client1.test.net' via host 'client1.test.net' and port '5665'
[2019-05-02 08:06:27 +0200] notice/JsonRpcConnection: Received 'pki::RequestCertificate' message from 'client1.test.net'
[2019-05-02 08:06:27 +0200] information/JsonRpcConnection: Received certificate request for CN 'client1.test.net' not signed by our CA.
[2019-05-02 08:06:27 +0200] information/JsonRpcConnection: Certificate request for CN 'client1.test.net' is pending. Waiting for approval.
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:30 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:33 +0200] debug/CheckerComponent: Scheduling info for checkable 'client1.test.net' (2019-05-02 08:06:33 +0200): Object 'client1.test.net', Next Check: 2019-05-02 08:06:33 +0200(1.55678e+09).
[2019-05-02 08:06:33 +0200] debug/CheckerComponent: Executing check for 'client1.test.net'
[2019-05-02 08:06:33 +0200] debug/Checkable: Update checkable 'client1.test.net' with check interval '30' from last check time at 2019-05-02 08:06:05 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:01 +0200(1.55678e+09).
[2019-05-02 08:06:33 +0200] notice/Process: Running command '/usr/lib64/nagios/plugins/check_ping' '-H' 'client1.test.net' '-c' '5000,100%' '-w' '3000,80%': PID 5978
[2019-05-02 08:06:33 +0200] debug/CheckerComponent: Check finished for object 'client1.test.net'
[2019-05-02 08:06:34 +0200] warning/JsonRpcConnection: API client disconnected for identity 'client1.test.net'
[2019-05-02 08:06:37 +0200] notice/Process: PID 5978 ('/usr/lib64/nagios/plugins/check_ping' '-H' 'client1.test.net' '-c' '5000,100%' '-w' '3000,80%') terminated with exit code 0
[2019-05-02 08:06:37 +0200] debug/Checkable: Update checkable 'client1.test.net' with check interval '30' from last check time at 2019-05-02 08:06:37 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:05 +0200(1.55678e+09).
[2019-05-02 08:06:37 +0200] debug/DbEvents: add checkable check history for 'client1.test.net'
[2019-05-02 08:06:37 +0200] information/ApiListener: Reconnecting to endpoint 'client1.test.net' via host 'client1.test.net' and port '5665'
[2019-05-02 08:06:37 +0200] warning/ApiListener: Certificate validation failed for endpoint 'client1.test.net': code 18: self signed certificate
[2019-05-02 08:06:37 +0200] information/ApiListener: New client connection for identity 'client1.test.net' to [10.0.0.3]:5665 (certificate validation failed: code 18: self signed certificate)
[2019-05-02 08:06:37 +0200] information/ApiListener: Finished reconnecting to endpoint 'client1.test.net' via host 'client1.test.net' and port '5665'
[2019-05-02 08:06:37 +0200] notice/JsonRpcConnection: Received 'pki::RequestCertificate' message from 'client1.test.net'
[2019-05-02 08:06:37 +0200] information/JsonRpcConnection: Received certificate request for CN 'client1.test.net' not signed by our CA.
[2019-05-02 08:06:37 +0200] information/JsonRpcConnection: Certificate request for CN 'client1.test.net' is pending. Waiting for approval.
[2019-05-02 08:06:39 +0200] information/JsonRpcConnection: Received certificate request for CN 'client1.test.net' not signed by our CA.
[2019-05-02 08:06:39 +0200] information/JsonRpcConnection: Certificate request for CN 'client1.test.net' is pending. Waiting for approval.
[2019-05-02 08:06:40 +0200] debug/CheckerComponent: Scheduling info for checkable 'client1.test.net!Check Memory' (2019-05-02 08:06:40 +0200): Object 'client1.test.net!Check Memory', Next Check: 2019-05-02 08:06:40 +0200(1.55678e+09).
[2019-05-02 08:06:40 +0200] debug/CheckerComponent: Executing check for 'client1.test.net!Check Memory'
[2019-05-02 08:06:40 +0200] debug/Checkable: Update checkable 'client1.test.net!Check Memory' with check interval '60' from last check time at 2019-05-02 08:05:42 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:37 +0200(1.55678e+09).
[2019-05-02 08:06:40 +0200] debug/Checkable: Update checkable 'client1.test.net!Check Memory' with check interval '60' from last check time at 2019-05-02 08:06:40 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:37 +0200(1.55678e+09).
[2019-05-02 08:06:40 +0200] debug/DbEvents: add checkable check history for 'client1.test.net!Check Memory'
[2019-05-02 08:06:40 +0200] debug/CheckerComponent: Check finished for object 'client1.test.net!Check Memory'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:40 +0200] notice/JsonRpcConnection: Received 'event::Heartbeat' message from 'client1.test.net'
[2019-05-02 08:06:43 +0200] debug/Checkable: Update checkable 'client1.test.net!Check Load' with check interval '60' from last check time at 2019-05-02 08:06:43 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:41 +0200(1.55678e+09).
[2019-05-02 08:06:43 +0200] debug/DbEvents: add checkable check history for 'client1.test.net!Check Load'
[2019-05-02 08:06:43 +0200] debug/CheckerComponent: Scheduling info for checkable 'client1.test.net!Check Procs' (2019-05-02 08:06:43 +0200): Object 'client1.test.net!Check Procs', Next Check: 2019-05-02 08:06:43 +0200(1.55678e+09).
[2019-05-02 08:06:43 +0200] debug/CheckerComponent: Executing check for 'client1.test.net!Check Procs'
[2019-05-02 08:06:43 +0200] debug/Checkable: Update checkable 'client1.test.net!Check Procs' with check interval '60' from last check time at 2019-05-02 08:05:46 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:40 +0200(1.55678e+09).
[2019-05-02 08:06:43 +0200] debug/Checkable: Update checkable 'client1.test.net!Check Procs' with check interval '60' from last check time at 2019-05-02 08:06:43 +0200 (1.55678e+09) to next check time at 2019-05-02 08:07:40 +0200(1.55678e+09).
[2019-05-02 08:06:43 +0200] debug/DbEvents: add checkable check history for 'client1.test.net!Check Procs'
[2019-05-02 08:06:43 +0200] debug/CheckerComponent: Check finished for object 'client1.test.net!Check Procs'

Nevertheless, the following is always displayed in the frontend of Icingaweb2:

Remote Icinga instance ‘client1.test.net’ is not connected to ‘master1.test.net

Maybe someone has a tip what goes wrong here

Okay, so yeah, this is what I expected. When you do the node wizard, that line that says to paste into the master to fetch the ticket, put that into the wizard and it’ll obtain its certificate. After that, go back to the client and remove the “host” ip address entries from its zones.conf.

Hi,

thanks for the many details. I’ve taken the liberty to edit your posts and enclose all configuration and log blocks with 3 backticks to enhance readability (otherwise I could not read and reply fast enough). Please keep this in mind for future posts of yours, a guide for Markdown formatting can be found here.

Reading the node wizard output, you’re using the on-demand signing method and as such, you’ll need to sign the certificate request on your master node, what’s the output of icinga2 ca list - e.g. did you already sign the request? The logs from your client tell a different story, it always updates the request but does not receive any.

Read more about this specific signing method here.

Cheers,
Michael

Hallo Zusammen,

das ist ist die Ausgabe von Master1 mit folgendem Command:


[root@master1 ssl]# icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
174472dfeb80d2d0d8f79ab7bfa9bd212ad30968cf06533ec9e04cf4a36d00ca | Apr 26 15:57:02 2019 GMT |        | CN = client02.test.net
2d83bc335015257cabab62bd203cac5afd84b346ef9018af37596bb8ce650240 | Apr 27 06:23:18 2019 GMT |        | CN = client02.test.net
36fff50e20726cb221decb1263ce570f3c00b49e4ba61bec9722079c8e8a3065 | May  1 09:15:29 2019 GMT |        | CN = client01.test.net
388eba17ee5995e7288a565a7443e18b3d28e7644d615301e735e2387d505a5e | May  1 09:10:45 2019 GMT |        | CN = client01.test.net
3930f8a2437e4774670405d30ae0cd02c00e5a2894c52a89a9da8743b37dcbe1 | Apr 27 05:40:41 2019 GMT |        | CN = client02.test.net
3c6add570eb433d31b1805728029d2e2d4eb1048f09e148c4fb5d854e6698094 | Apr 27 04:53:51 2019 GMT |        | CN = client02.test.net
4210372fb6c7656adfe005fc89fa5d81a914ab9a801e7959132512fcadfeda08 | May  1 07:56:51 2019 GMT |        | CN = client01.test.net
4d9e65797ae017670eab371dc56b2c09fa995728dedb307686bbfd6e8d146745 | May  1 09:25:05 2019 GMT |        | CN = client01.test.net
727e0a62c6bf56f31f9d0eb3c9d089334fe1bd74163229342d8078ce8f5f7567 | May  2 05:53:53 2019 GMT |        | CN = client01.test.net
77f2723bfa043e8b58149f140993e873271f9c30a477c4b660c1cce05d9196be | Apr 27 06:25:41 2019 GMT |        | CN = client02.test.net
a25539b96368aa039bd396bbb86060e724a8af366f605368011729edbba7ef08 | Apr 28 15:33:24 2019 GMT | *      | CN = client02.test.net
a726b438bb7d0466acd12f2f7794e69e112ebb704f36f695d915dfd7db476991 | Apr 26 06:02:20 2019 GMT |        | CN = client02.test.net
a819b626c68d5ae8fda86ba86988f0ab3cb59778b444f3f1ba3b4ef4fb8bef01 | Apr 27 05:13:57 2019 GMT |        | CN = client02.test.net
b464ad55fab69287bebc5b39090fc09e790bf8f7603372c4e2c3b577efebef47 | Apr 30 17:15:22 2019 GMT |        | CN = client01.test.net
b73fd5146fb92c9fb91f9a7c8cf63318c98f33da5e5954ae22035f6c87628e5e | May  1 09:04:13 2019 GMT |        | CN = client01.test.net
bf7d41b2dfd1dce937eb32eec02e80f7f00cff871804fc3a3bd50678e29d3cfc | May  1 08:03:28 2019 GMT |        | CN = client01.test.net
c0d9de1f2dab3e5c8e07f04ad083bfa6405b25b71c47ad350dd66c12a94b17cf | Apr 30 17:06:50 2019 GMT |        | CN = client01.test.net
c5c49cc3eea432fa6ea59437b4f957558c2f07838038ab62b77eccb0539097ef | Apr 26 15:39:20 2019 GMT |        | CN = client02.test.net
d877aae456577b28392c6862a1304cd70bb1e5d52321bd83759adc7637b8dab8 | May  1 20:08:01 2019 GMT |        | CN = client01.test.net
df6114851365fa193c175141359ea0f1823ee00b69c8496a86eefa41886ae199 | May  1 08:56:15 2019 GMT |        | CN = client01.test.net
ecefeb8be667e73cd26e4c67764eac5ea180c90c2e29a7820d71beace3b3b71a | Apr 26 16:06:19 2019 GMT |        | CN = client02.test.net
f17ea14ac24cadca8cf72b0f4d996e48dd1a99a34ca1fb18a53e6082d959a720 | May  1 08:16:40 2019 GMT |        | CN = client01.test.net
f860a881e38a859d00dae5cfb1f0c7858cee5c8b444765279863838374134e59 | Apr 28 20:53:33 2019 GMT |        | CN = client03.test.net

Hallo Zusammen,

das ist ist die Ausgabe von Master2 mit folgendem Command:


[root@master2 root]# icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
174472dfeb80d2d0d8f79ab7bfa9bd212ad30968cf06533ec9e04cf4a36d00ca | Apr 26 15:57:02 2019 GMT |        | CN = client02.test.net
2d83bc335015257cabab62bd203cac5afd84b346ef9018af37596bb8ce650240 | Apr 27 06:23:18 2019 GMT |        | CN = client02.test.net
36fff50e20726cb221decb1263ce570f3c00b49e4ba61bec9722079c8e8a3065 | May  1 09:15:29 2019 GMT |        | CN = client01.test.net
388eba17ee5995e7288a565a7443e18b3d28e7644d615301e735e2387d505a5e | May  1 09:10:45 2019 GMT |        | CN = client01.test.net
3930f8a2437e4774670405d30ae0cd02c00e5a2894c52a89a9da8743b37dcbe1 | Apr 27 05:40:41 2019 GMT |        | CN = client02.test.net
3c6add570eb433d31b1805728029d2e2d4eb1048f09e148c4fb5d854e6698094 | Apr 27 04:53:51 2019 GMT |        | CN = client02.test.net
4210372fb6c7656adfe005fc89fa5d81a914ab9a801e7959132512fcadfeda08 | May  1 07:56:51 2019 GMT |        | CN = client01.test.net
4d9e65797ae017670eab371dc56b2c09fa995728dedb307686bbfd6e8d146745 | May  1 09:25:05 2019 GMT |        | CN = client01.test.net
727e0a62c6bf56f31f9d0eb3c9d089334fe1bd74163229342d8078ce8f5f7567 | May  2 05:53:53 2019 GMT |        | CN = client01.test.net
77f2723bfa043e8b58149f140993e873271f9c30a477c4b660c1cce05d9196be | Apr 27 06:25:41 2019 GMT |        | CN = client02.test.net
a25539b96368aa039bd396bbb86060e724a8af366f605368011729edbba7ef08 | Apr 28 15:33:24 2019 GMT |        | CN = client02.test.net
a726b438bb7d0466acd12f2f7794e69e112ebb704f36f695d915dfd7db476991 | Apr 26 06:02:20 2019 GMT |        | CN = client02.test.net
a819b626c68d5ae8fda86ba86988f0ab3cb59778b444f3f1ba3b4ef4fb8bef01 | Apr 27 05:13:57 2019 GMT |        | CN = client02.test.net
b464ad55fab69287bebc5b39090fc09e790bf8f7603372c4e2c3b577efebef47 | Apr 30 17:15:22 2019 GMT |        | CN = client01.test.net
b73fd5146fb92c9fb91f9a7c8cf63318c98f33da5e5954ae22035f6c87628e5e | May  1 09:04:13 2019 GMT |        | CN = client01.test.net
bf7d41b2dfd1dce937eb32eec02e80f7f00cff871804fc3a3bd50678e29d3cfc | May  1 08:03:28 2019 GMT |        | CN = client01.test.net
c0d9de1f2dab3e5c8e07f04ad083bfa6405b25b71c47ad350dd66c12a94b17cf | Apr 30 17:06:50 2019 GMT |        | CN = client01.test.net
c5c49cc3eea432fa6ea59437b4f957558c2f07838038ab62b77eccb0539097ef | Apr 26 15:39:20 2019 GMT |        | CN = client02.test.net
d63e35bd59e7fca1c56aab83e1cbde9df409e61dd81066d73d4e6733cf2253a9 | Apr 14 15:45:50 2019 GMT |        | CN = client03.test.net
d877aae456577b28392c6862a1304cd70bb1e5d52321bd83759adc7637b8dab8 | May  1 20:08:01 2019 GMT |        | CN = client01.test.net
df6114851365fa193c175141359ea0f1823ee00b69c8496a86eefa41886ae199 | May  1 08:56:15 2019 GMT |        | CN = client01.test.net
df8173bac88faeffa1b8c976d89d2eb4b685052d7110e01f8c97d797a7ed0695 | Apr 14 15:53:38 2019 GMT |        | CN = client04.test.net
ecefeb8be667e73cd26e4c67764eac5ea180c90c2e29a7820d71beace3b3b71a | Apr 26 16:06:19 2019 GMT |        | CN = client02.test.net
f17ea14ac24cadca8cf72b0f4d996e48dd1a99a34ca1fb18a53e6082d959a720 | May  1 08:16:40 2019 GMT |        | CN = client01.test.net
f860a881e38a859d00dae5cfb1f0c7858cee5c8b444765279863838374134e59 | Apr 28 20:53:33 2019 GMT |        | CN = client03.test.net

This is the content of constants.conf of Master1


/**
 * This file defines global constants which can be used in
 * the other configuration files.
 */

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = "/usr/lib64/nagios/plugins"

/* The directory which contains the Manubulon plugins.
 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
 */
const ManubulonPluginDir = "/usr/lib64/nagios/plugins"

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
 * Check the documentation, chapter "Plugins Contribution", for details.
 */
const PluginContribDir = "/usr/lib64/nagios/plugins"

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "master1.test.net"

/* Our local zone name. */
const ZoneName = "master1.test.net"

/* Secret key for remote node tickets */
const TicketSalt = "4eb26a6f4a8e37ef7a67d3aba4d3c94e"

and this is the Content from constants.conf of Master2


/**
 * This file defines global constants which can be used in
 * the other configuration files.
 */

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = "/usr/lib64/nagios/plugins"

/* The directory which contains the Manubulon plugins.
 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
 */
const ManubulonPluginDir = "/usr/lib64/nagios/plugins"

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
 * Check the documentation, chapter "Plugins Contribution", for details.
 */
const PluginContribDir = "/usr/lib64/nagios/plugins"

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "master2.test.net"

/* Our local zone name. */
const ZoneName = "master2.test.net"

/* Secret key for remote node tickets */
const TicketSalt = ""

This is the Content form api.conf from Master 1 (api feature enabled)


object ApiListener "api" {
  accept_config = true
  accept_commands = true

  ticket_salt = TicketSalt
}

and this the Content from Master 2 - api.conf - here is include_recursive “conf.d” disabled


/**
 * The API listener is used for distributed monitoring setups.
 */
object ApiListener "api" {
  accept_config = true
  accept_commands = true
}

I hope that Helps you. Thanks