Greetings everyone!
Just for the purpose of documentation I would like to present you an issue I recently experienced with the certificate on a satellite and how it could be solved.
We run a distributed Icinga 2 setup with 2 masters and several satellites. On may 22 the masters were no longer able to talk to the icinga instance running on satellite “i2sat”:
Remote Icinga instance 'i2sat.ourdomain' is not connected to 'i2master-a.ourdomain'
In the logs of the satellite we repeatedly saw these messages:
[2023-05-24 11:21:24 +0200] information/JsonRpcConnection: Requesting new certificate for this Icinga instance from endpoint 'i2master-a'.
[2023-05-24 11:21:27 +0200] information/JsonRpcConnection: Requesting new certificate for this Icinga instance from endpoint 'i2master-b'.
In the logs of the masters we repeatedly saw these messages:
[2023-05-24 11:26:24 +0200] warning/ApiListener: Certificate validation failed for endpoint 'i2sat.ourdomain': code 7: certificate signature failure
[2023-05-24 11:26:24 +0200] information/ApiListener: New client connection for identity 'i2sat' to [1.2.3.4]:5665 (certificate validation failed: code 7: certificate signature failure)
[2023-05-24 11:26:24 +0200] information/JsonRpcConnection: Received certificate request for CN 'i2sat.ourdomain' signed by our CA.
[2023-05-24 11:26:24 +0200] information/JsonRpcConnection: The certificate for CN 'i2sat.ourdomain' is valid and uptodate. Skipping automated renewal.
A verification of the certificate revealed the following:
$ cd /var/lib/icinga2/certs/
$ icinga2 pki verify --cert i2sat.ourdomain.crt
information/cli: Printing certificate 'i2sat.ourdomain.crt'
Version: 3
Subject: CN = i2sat.ourdomain
Issuer: CN = Icinga CA
Valid From: May 22 11:50:18 2023 GMT
Valid Until: Jun 22 11:50:18 2024 GMT
Serial: 6a:44:96:3f:8c:1a:49:6a:3b:58:4a:cc:23:bc:27:bd:44:08:3d:bd
Signature Algorithm: sha256WithRSAEncryption
Subject Alt Names: i2sat.ourdomain
Fingerprint: BC 08 24 55 45 D5 D2 AA 3E 3B 33 6C 16 68 67 D4 F1 F7 4A 36 FC A7 C5 AE 0F FA D8 E0 F5 2B 67 8B
In fact, communication between masters and satellite stopped on may 22. And I do not find the reason why this should have happened like this. Should the Icinga PKI certificates not be valid for a period of 15 years?
Anyway, here is an article that helped me with the fix:
# Manual Icinga2 PKI certificate creation / distribution
My steps to create a new certificate for the satellite on the master server with the CA:
$ cd /tmp
$ icinga2 pki new-cert --cn i2sat.ourdomain --key i2sat.ourdomain.key --csr i2sat.ourdomain.csr
$ icinga2 pki sign-csr --csr i2sat.ourdomain.csr --cert i2sat.ourdomain.crt
Then copy over the files to the directory /var/lib/icinga2/certs/ on the satellite so that it looks like this in the end:
$ cd /var/lib/icinga2/certs
$ ls -al
total 28
drwx------ 4 nagios nagios 4096 May 24 14:58 .
drwxr-x--- 4 nagios nagios 4096 May 24 15:23 ..
-rw-r--r-- 1 nagios nagios 1720 Dec 13 2020 ca.crt
-rw-r--r-- 1 nagios nagios 1797 May 24 14:56 i2sat.ourdomain.crt
-rw------- 1 nagios nagios 3243 May 24 14:56 i2sat.ourdomain.key
After restarting icinga on the satellite everything was good again.
Can someone explain why the certificate on the satellite was only valid for 1 year instead of 15 years? We used the node wizard to set up the satellite.