Hmm… something is still not working when I try to add a new client. The node wizard does not succeed and the master log reports:
[2019-08-09 12:44:42 +0000] information/ApiListener: New client connection for identity 'ldue-clickhouse-grafana-ds-01' from [x.x.x.x]:43120 (certificate validation failed: code 18: self signed certificate)
[2019-08-09 12:44:52 +0000] warning/ApiListener: No data received on new API connection for identity 'ldue-clickhouse-grafana-ds-01'. Ensure that the remote endpoints are properly configured in a cluster setup.
First, I created the host in the Icinga2 director. The rendered configuration:
zones.d/do-ffm-icinga-master/hosts.conf
object Host "ldue-clickhouse-grafana-ds-01" {
import "linux-host"
address = "x.x.x.x"
vars.customer = "SSC"
}
zones.d/do-ffm-icinga-master/agent_endpoints.conf
object Endpoint "ldue-clickhouse-grafana-ds-01" {
log_duration = 0s
}
zones.d/do-ffm-icinga-master/agent_zones.conf
object Zone "ldue-clickhouse-grafana-ds-01" {
parent = "do-ffm-icinga-master"
endpoints = [ "ldue-clickhouse-grafana-ds-01" ]
}
zones.conf on the client:
object Endpoint "do-ffm-icinga-01" {
host = "x.x.x.165"
port = "5665"
}
object Endpoint "do-ffm-icinga-02" {
host = "x.x.x.188"
port = "5665"
}
object Zone "do-ffm-icinga-master" {
endpoints = ["do-ffm-icinga-01", "do-ffm-icinga-02"]
}
object Endpoint "ldue-clickhouse-grafana-ds-01" {
host = "x.x.x.x"
port = "5665"
}
object Zone "ldue-clickhouse-grafana-ds-01" {
endpoints = ["ldue-clickhouse-grafana-ds-01"]
parent = "do-ffm-icinga-master"}
object Zone "global-templates" {
global = true
}
object Zone "director-global" {
global = true
}
Next step is to setup the pki (the PKI on the master is fine and we have more than 70 hosts connected).
So I run on the client:
icinga2 pki new-cert
--cn ldue-clickhouse-grafana-ds-01
--key /var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.key
--csr /var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.csr
--cert /var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.crt
information/base: Writing private key to '/var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.crt'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.csr'.
icinga2 pki save-cert --host x.x.x.165 --key /var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.key --cert /var/lib/icinga2/certs/ldue-clickhouse-grafana-ds-01.crt --trustedcert /var/lib/icinga2/certs/do-ffm-icinga-01.crt
information/cli: Retrieving X.509 certificate for 'x.x.x.165:5665'.
Subject: CN = do-ffm-icinga-01
Issuer: CN = Icinga CA
Valid From: Dec 28 01:10:51 2018 GMT
Valid Until: Dec 24 01:10:51 2033 GMT
Fingerprint: 99 D8 5E 49 ED 02 47 82 06 12 9F A9 AD 66 C6 ED F3 96 F9 8E
***
*** You have to ensure that this certificate actually matches the parent
*** instance's certificate in order to avoid man-in-the-middle attacks.
***
information/pki: Writing certificate to file '/var/lib/icinga2/certs/do-ffm-icinga-01.crt'.
Next step is the setup using the node wizard:
icinga2 node setup
--cn ldue-clickhouse-grafana-ds-01
--endpoint do-ffm-icinga-01,x.x.x.165,5665
--zone ldue-clickhouse-grafana-ds-01
--parent_zone do-ffm-icinga-master
--parent_host x.x.x.165
--trustedcert /var/lib/icinga2/certs/do-ffm-icinga-01.crt
--accept-commands
--disable-confd
--accept-config
--ticket c9934524a18744f73c64d41abb19d31e78d3a27f
information/cli: Requesting certificate with ticket 'c9934524a18744f73c64d41abb19d31e78d3a27f'.
information/cli: Verifying parent host connection information: host 'x.x.x.165', port '5665'.
information/cli: Using the following CN (defaults to FQDN): 'ldue-clickhouse-grafana-ds-01'.
information/cli: Backup file '/var/lib/icinga2/certs//ldue-clickhouse-grafana-ds-01.key.orig' already exists. Skipping backup.
information/cli: Backup file '/var/lib/icinga2/certs//ldue-clickhouse-grafana-ds-01.crt.orig' already exists. Skipping backup.
information/base: Writing private key to '/var/lib/icinga2/certs//ldue-clickhouse-grafana-ds-01.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs//ldue-clickhouse-grafana-ds-01.crt'.
information/cli: Verifying trusted certificate file '/var/lib/icinga2/certs/do-ffm-icinga-01.crt'.
information/cli: Requesting a signed certificate from the parent Icinga node.
critical/cli: Could not fetch valid response. Please check the master log.
critical/cli: Failed to fetch signed certificate from parent Icinga node 'x.x.x.165, 5665'. Please try again.
The log on the master shows just two events:
[2019-08-09 13:14:43 +0000] information/ApiListener: New client connection for identity 'ldue-clickhouse-grafana-ds-01' from [x.x.x.x]:43154 (certificate validation failed: code 18: self signed certificate)
[2019-08-09 13:14:53 +0000] warning/ApiListener: No data received on new API connection for identity 'ldue-clickhouse-grafana-ds-01'. Ensure that the remote endpoints are properly configured in a cluster setup.
Context:
(0) Handling new API client connection
So I came across this thread: CSR auto-signing fails silently if no ticket_salt is set in the ApiListener feature configuration and I did not have a TicketSalt defined as well. So I created it with openssl rand -base64 30 and added it to the constants.conf. Did the same for the 2nd master node.
The api-feature is enabled and the configuration looks like this:
/**
* The API listener is used for distributed monitoring setups.
*/
object ApiListener "api" {
accept_config = true
accept_commands = true
ticket_salt = TicketSalt
}
The constants.conf of that node:
/**
* This file defines global constants which can be used in
* the other configuration files.
*/
/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = "/usr/lib/nagios/plugins"
/* The directory which contains the Manubulon plugins.
* Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
*/
const ManubulonPluginDir = "/usr/lib/nagios/plugins"
/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
* Check the documentation, chapter "Plugins Contribution", for details.
*/
const PluginContribDir = "/usr/lib/nagios/plugins"
/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
* This should be the common name from the API certificate.
*/
const NodeName = "do-ffm-icinga-01"
/* Our local zone name. */
const ZoneName = "do-ffm-icinga-master"
/* Secret key for remote node tickets */
const TicketSalt = "2xMm1JgsEQcUjJ/nve5xWPPA47zXIIXKU6boyK4n"
Any hint appreciated again. I don’t get why this topic confuses me that much. 
