Icinga certificate validation failed: code 18: self signed certificate

Hi

I’ve a problem with ca certs.
I’ve one master01 which is working with many satellites.
I’ve added one another master but i’ve a ca signature problem.

"certificate validation failed: code 18: self signed certificate"

master02[users:3] /var/lib/icinga2/certs # echo | openssl s_client -connect master01:5665 -showcerts 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /var/lib/icinga2/certs/ca.crt

master02[users:3] /var/lib/icinga2/certs # echo | openssl s_client -connect master01:5665 -showcerts 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /var/lib/icinga2/certs/ca.crt

master02[users:3] /var/lib/icinga2/certs # icinga2 pki new-cert --cn master02 --key /var/lib/icinga2/certs/master02.key --cert /var/lib/icinga2/certs/master02.crt
information/base: Writing private key to '/var/lib/icinga2/certs/master02.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/certs/master02.crt'.

master02[users:3] /var/lib/icinga2/certs # icinga2 pki request --host master01 --port 5665 --key /var/lib/icinga2/certs/master02.key --cert /var/lib/icinga2/certs/master02.crt --trustedcert /var/lib/icinga2/certs/ca.crt --ca /var/lib/icinga2/certs/ca.key --ticket `cat /var/lib/icinga2/certs/pki.ticket`
information/cli: Writing CA certificate to file '/var/lib/icinga2/certs/ca.key'.
information/cli: !!!!!!
information/cli: !!! Certificate request for CN 'master02' is pending. Waiting for approval from the parent Icinga instance.
information/cli: !!!!!!


master02[users:3] /var/lib/icinga2/certs # ll
total 20
-rw-r--r-- 1 xxx yyy 3529 Aug 20 16:52 ca.crt
-rw-r--r-- 1 xxx yyy 1720 Aug 20 16:52 ca.key
-rw-r----- 1 xxx yyy 1842 Aug 20 16:52 master02.crt
-rw------- 1 xxx yyy 3243 Aug 20 16:52 master02.key
-rw-r--r-- 1 xxx yyy   41 Aug 20 16:34 pki.ticket
     
 

 
master01[users:1] ~ # icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
09878ae29f6365aab0b67ff09641a297debb1d61b23b49b94e4e948ce49ecb03 | Aug 20 14:52:05 2021 GMT |        | CN = master02

master01[users:1] ~ # icinga2 ca sign 09878ae29f6365aab0b67ff09641a297debb1d61b23b49b94e4e948ce49ecb03
information/cli: Signed certificate for 'CN = master02'.
 
 
master01[users:1] ~ # tail -f /var/log/icinga2/icinga2.log | grep dev02
[2021-08-20 16:53:31 +0200] information/ApiListener: Reconnecting to endpoint 'master02' via host 'master02' and port '5665'
[2021-08-20 16:53:31 +0200] warning/ApiListener: Certificate validation failed for endpoint 'master02': code 18: self signed certificate
[2021-08-20 16:53:31 +0200] information/ApiListener: New client connection for identity 'master02' to [141.95.39.129]:5665 (certificate validation failed: code 18: self signed certificate)
[2021-08-20 16:53:31 +0200] information/ApiListener: Finished reconnecting to endpoint 'master02' via host 'master02' and port '5665'
[2021-08-20 16:53:31 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master02'
[2021-08-20 16:53:34 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master02'
[2021-08-20 16:53:41 +0200] information/ApiListener: Reconnecting to endpoint 'master02' via host 'master02' and port '5665'
[2021-08-20 16:53:41 +0200] warning/ApiListener: Certificate validation failed for endpoint 'master02': code 18: self signed certificate
[2021-08-20 16:53:41 +0200] information/ApiListener: New client connection for identity 'master02' to [141.95.39.129]:5665 (certificate validation failed: code 18: self signed certificate)
[2021-08-20 16:53:41 +0200] information/ApiListener: Finished reconnecting to endpoint 'master02' via host 'master02' and port '5665'
 
 
 
master02[users:3] /var/lib/icinga2/certs # icinga2 pki verify --cert master02.crt --cacert ca.crt
information/cli: Verifying certificate 'master02.crt'
 
 Version:             3
 Subject:             CN = master02
 Issuer:              CN = master02     Valid From:          Aug 20 14:52:05 2021 GMT
 Valid Until:         Aug 16 14:52:05 2036 GMT
 Serial:              89:84:1f:0b:a2:7e:0a:d4:eb:74:d6:54:8f:ed:91:c3:05:75:e3:a4
 
 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   master02
 Fingerprint:         09 87 8A E2 9F 63 65 AA B0 B6 7F F0 96 41 A2 97 DE BB 1D 61 B2 3B 49 B9 4E 4E 94 8C E4 9E CB 03
 
information/cli:  with CA certificate 'ca.crt'.
 
 Version:             3
 Subject:             CN = master01     Issuer:              CN = Icinga CA
 Valid From:          Feb 26 08:28:40 2020 GMT
 Valid Until:         Feb 22 08:28:40 2035 GMT
 Serial:              01:53:ea:8e:fd:82:bb:c2:6c:e0:6a:d5:f8:07:a7:ad:db:88:82:0f
 
 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   master01
 Fingerprint:         60 3B A0 82 25 5A 75 13 D8 E9 E8 05 26 7D E8 1A 51 64 92 6E A9 83 0B D2 F3 37 AA 0B E4 2A 3C 39
 
critical/cli: CRITICAL: Certificate with CN 'master02' is NOT signed by CA: self signed certificate (code 18)




master02[users:3] icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.1-1)


master01][users:1] ~ # icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.4-1)

No problem with my satellites, only with this second master.
i’ve tried to remove rm -rf /var/lib/icinga2/certs/ and retry but same problem.

If someone can help me .

Thanks

1 Like

Hello @enigma619!

Have you approved it via icinga2 ca sign?

https://icinga.com/docs/icinga-2/latest/doc/11-cli-commands/

Best,
AK

1 Like

Yes I’ve done it (you can see it on my last message it seems you have to scroll to see all informations on it)

master01[users:1] ~ # icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
09878ae29f6365aab0b67ff09641a297debb1d61b23b49b94e4e948ce49ecb03 | Aug 20 14:52:05 2021 GMT |        | CN = master02

master01[users:1] ~ # icinga2 ca sign 09878ae29f6365aab0b67ff09641a297debb1d61b23b49b94e4e948ce49ecb03
information/cli: Signed certificate for 'CN = master02'.

To complete my last message, yes certificate has been signed but no more success.

master01[users:1] ~ # icinga2 ca sign 09878ae29f6365aab0b67ff09641a297debb1d61b23b49b94e4e948ce49ecb03
information/cli: Signed certificate for 'CN = master02'.

icinga restarted.

master01[users:1] ~ # icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------

Logs:

[2021-08-20 16:53:31 +0200] information/ApiListener: Reconnecting to endpoint 'master02' via host 'master02' and port '5665'
[2021-08-20 16:53:31 +0200] warning/ApiListener: Certificate validation failed for endpoint 'master02': code 18: self signed certificate
[2021-08-20 16:53:31 +0200] information/ApiListener: New client connection for identity 'master02' to [141.95.39.129]:5665 (certificate validation failed: code 18: self signed certificate)
[2021-08-20 16:53:31 +0200] information/ApiListener: Finished reconnecting to endpoint 'master02' via host 'master02' and port '5665'
[2021-08-20 16:53:31 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master02'
[2021-08-20 16:53:34 +0200] warning/JsonRpcConnection: API client disconnected for identity 'master02'
[2021-08-20 16:53:41 +0200] information/ApiListener: Reconnecting to endpoint 'master02' via host 'master02' and port '5665'
[2021-08-20 16:53:41 +0200] warning/ApiListener: Certificate validation failed for endpoint 'master02': code 18: self signed certificate
[2021-08-20 16:53:41 +0200] information/ApiListener: New client connection for identity 'master02' to [141.95.39.129]:5665 (certificate validation failed: code 18: self signed certificate)
[2021-08-20 16:53:41 +0200] information/ApiListener: Finished reconnecting to endpoint 'master02' via host 'master02' and port '5665'

I don’t know if it’s linked but I’ve enabled debug log on my master02 and I have some errors like:

[2021-08-23 09:50:47 +0200] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'master01': Error: End of file


	(0) icinga2: icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) (+0x3d1) [0x562ba5d6ccb1]
	(1) icinga2: icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) (+0x49) [0x562ba5c5ff19]
	(2) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x119) [0x562ba5c6cc09]
	(3) icinga2: <unknown function> (+0x67cdcd) [0x562ba5c4fdcd]
	(4) icinga2: <unknown function> (+0x648ef2) [0x562ba5c1bef2]
	(5) libboost_context.so.1.67.0: make_fcontext (+0x2f) [0x7f4cc9ace1ef]

Another strange thing:

I had a similar error with one satellite. With this satellite, I’ve removed /var/lib/icinga2/certs/, relaunch pki signature, signed certificate on master01, restarted icinga and all is OK with this satellite.

So the problem is only located on master02.
icinga2 versions were differents on my 2 masters, i’ve fixed it (all on r2.12.4-1) but problem is always present.

On which of all nodes find /var/lib/icinga2 -name ca.key gives you a result?

master01[users:1] ~ # find /var/lib/icinga2 -name ca.key
/var/lib/icinga2/ca/ca.key
master02[users:1] ~ # find /var/lib/icinga2 -name ca.key
/var/lib/icinga2/certs/ca.key

And on every satellites too

Please compare the output of md5sum /var/lib/icinga2/ca/ca.key on all nodes.

master01[users:1] ~ # md5sum /var/lib/icinga2/ca/ca.key
cfda8368ad62a15825d645aa13b3f0c6  /var/lib/icinga2/ca/ca.key

master01[users:1] ~ # cd /var/lib/icinga2/ca/
master01[users:1] ~ #/var/lib/icinga2/ca # ll
total 8
-rw------- 1 nagios nagios 1720 Feb 26  2020 ca.crt
-rw------- 1 nagios nagios 3247 Feb 26  2020 ca.key

master02[users:1] ~ # md5sum /var/lib/icinga2/ca/ca.key
md5sum: /var/lib/icinga2/ca/ca.key: No such file or directory

master02[users:1] ~ # cd /var/lib/icinga2/ca/
-bash: cd: /var/lib/icinga2/ca/: No such file or directory


But:

master02[users:1] ~ # md5sum /var/lib/icinga2/certs/ca.key
ef47058d1187b283da2197f9ffe89213  /var/lib/icinga2/certs/ca.key

And for all my satellites: same thing “md5sum: /var/lib/icinga2/ca/ca.key: No such file or directory”
Example for one of my satellite (where signature was working):

satellite01: md5sum /var/lib/icinga2/ca/ca.key
md5sum: /var/lib/icinga2/ca/ca.key: No such file or directory

satellite01: md5sum /var/lib/icinga2/certs/ca.key
ef47058d1187b283da2197f9ffe89213  /var/lib/icinga2/certs/ca.key

Remove the key from all nodes ex. master01. Also remove /var/lib/icinga2/certs and /var/lib/icinga2/ca on that nodes (where the key existed ex. master01) and re-run the node wizard.

Not better :frowning:
I’ve made some clean on my zone.conf, I’ve just now master01 and master02.
I’ve deleted all ca certs on master02 and recreate but no success.

master02 [users:1] /var/lib/icinga2/certs # ll
total 16
-rw-r--r-- 1 nagios nagios 3529 Aug 24 15:19 ca.crt
-rw-r--r-- 1 nagios nagios 1720 Aug 24 15:19 ca.key
-rw-r--r-- 1 nagios nagios 1842 Aug 24 15:19 master02.crt
-rw------- 1 nagios nagios 3243 Aug 24 15:19 master02.key

md5sum ca.crt
67d3cdbbdaba8891cecef77edecf37ef  ca.crt

(On master01: md5sum ca.crt  ef47058d1187b283da2197f9ffe89213  ca.crt)

icinga2 pki verify --cert master02.crt --cacert ca.crt
information/cli: Verifying certificate 'master02.crt'

 Version:             3
 Subject:             CN = master02
 Issuer:              CN = master02
 Valid From:          Aug 24 13:19:52 2021 GMT
 Valid Until:         Aug 20 13:19:52 2036 GMT
 Serial:              b9:00:90:89:6e:a9:48:fc:7c:ee:11:9d:1f:78:2c:0f:f0:ee:98:3e

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   master02
 Fingerprint:         33 11 4E 19 E0 2D DE 5D C2 F7 01 51 98 B3 1C BA A9 EB E6 3A C6 6F AD D3 84 9F D6 EE B4 A9 0D F4 

information/cli:  with CA certificate 'ca.crt'.

 Version:             3
 Subject:             CN = master01
 Issuer:              CN = Icinga CA
 Valid From:          Feb 26 08:28:40 2020 GMT
 Valid Until:         Feb 22 08:28:40 2035 GMT
 Serial:              01:53:ea:8e:fd:82:bb:c2:6c:e0:6a:d5:f8:07:a7:ad:db:88:82:0f

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   master01
 Fingerprint:         60 3B A0 82 25 5A 75 13 D8 E9 E8 05 26 7D E8 1A 51 64 92 6E A9 83 0B D2 F3 37 AA 0B E4 2A 3C 39 

critical/cli: CRITICAL: Certificate with CN 'master02' is NOT signed by CA: self signed certificate (code 18)

My commands when trying manually…

master02:
echo | openssl s_client -connect master01:5665 -showcerts 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /var/lib/icinga2/certs/ca.crt

icinga2 pki new-cert --cn "master02" --key /var/lib/icinga2/certs/master02.key --cert /var/lib/icinga2/certs/master02.crt

icinga2 pki request --host master01 --port 5665 --key /var/lib/icinga2/certs/master02.key --cert /var/lib/icinga2/certs/master02.crt --trustedcert /var/lib/icinga2/certs/ca.crt --ca /var/lib/icinga2/certs/ca.key

Then icinga2 ca sign on master01

Problem is only present with masters (so with master02)

cat /etc/icinga2/zones.conf

/* configuration for zone master */
object Endpoint "master01" {
  host = "master01"
}

object Endpoint "master02" {
  host = "master02"
}

object Zone "master" {
  endpoints = ["master01", "master02"]
  
}

/* global templates for top down sync config */
object Zone "global-templates" {
  global = true
}

This file is the same on master01 and 02.

icinga2 pki new-cert, icinga2 pki request, … why not just the node wizard?

Due to my automation (with many satellites, I’ve done it with icinga2 pki…)
But I’ve tried to do it with wizard and same problem.
What is strange is that with my satellites i’ve no problem, only when I want to add a second master.

Version: 3
Subject: CN = master02
Issuer: CN = master02

The Issuer here is wrong. It should always be the Icinga CA and not the NodeName.

In order to fix this, you have to delete the following directories from master02.

/var/lib/icinga2/ca
/var/lib/icinga2/certificate-requests
/var/lib/icinga2/certs

Then perform the following step in master01.

Transfer master01's CA pub key into master02 using scp or whatever.
scp /var/lib/icinga2/ca/ca.crt master02Host:/var/lib/icinga2/certs/

Then create a signing request for master02. Note Now, we aren’t on master01 anymore.

icinga2 pki new-cert --cn master02 \
--key /var/lib/icinga2/certs/master02.key \
--csr /var/lib/icinga2/certificate-requests/master02.csr

Sign the newly create CR. Still on master02

icinga2 pki sign-csr --csr /var/lib/icinga2/certificate-requests/master02.csr \
--cert /var/lib/icinga2/certs/master02.crt

Now check the ownership of the newly created files and you’re done.

1 Like

Thank you for your help !
I’ve deleted directories on master02

Then transferring /var/lib/icinga2/ca/ca.crt from master01 to /var/lib/icinga2/certs/ on master02:
master02][users:1] /var/lib/icinga2/certs # ll ca.crt 
-rw-r--r-- 1 nagios nagios 3529 Aug 25 09:43 ca.crt
On master02:
icinga2 pki new-cert --cn master02 \
--key /var/lib/icinga2/certs/master02.key \
--csr /var/lib/icinga2/certificate-requests/master02.csr

Result:
information/base: Writing private key to '/var/lib/icinga2/certs/master02.key'.
information/base: Writing certificate signing request to '/var/lib/icinga2/certificate-requests/master02.csr'.

master02[users:1] /var/lib/icinga2/certificate-requests # ll
total 4
-rw-r--r-- 1 nagios nagios 1700 Aug 25 09:49 master02.csr

Last step:

master02[users:1] /var/lib/icinga2 # icinga2 pki sign-csr --csr /var/lib/icinga2/certificate-requests/master02.csr --cert /var/lib/icinga2/certs/master02.crt
critical/SSL: Could not open CA key file '/var/lib/icinga2/ca/ca.key': 33558530, "error:02001002:system library:fopen:No such file or directory"
information/pki: Writing certificate to file '/var/lib/icinga2/certs/master02.crt'.


master02[users:1] /var/lib/icinga2/certs # ll
total 12
-rw-r--r-- 1 nagios nagios 3529 Aug 25 09:43 ca.crt
-rw-r--r-- 1 nagios nagios   54 Aug 25 09:53 master02.crt
-rw------- 1 nagios nagios 3247 Aug 25 09:49 master02.key

master02.crt has so been created but empty:
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----


master02[users:1] /var/lib/icinga2 # ll ca
ls: cannot access 'ca': No such file or directory

no "ca" repository on master02

Thanks for ideas

Restarted again, I’ve exported ca.crt from master01, ca.key too.
Made a symbolic link: ln -sf /var/lib/icinga2/certs/ca.crt /var/lib/icinga2/ca/ca.crt
Then doing your steps again and restarted icinga.

All is OK now !!

Master01 and 02 are UP and running, and are syncho

Thanks for your great help!

Ohh :open_mouth:, of course, because the CA private key does not exist in master02. All steps except (removing the directories) should be performed on master01 and transfer all files belonging to master02 and the master01's ca.crt to the respective directories. My mistake 🤦. But you shouldn’t copy the ca.key of master01 anywhere, because it should always be only in the primary master (master01). So you can delete it directly from master02 now.

Done now.

Thanks for your help :slight_smile:

1 Like