Reconnect loop for self-signed certificates in icinga2 2.14

Icinga 2
1

I believe I may be seeing an instance of Icinga 2 reconnects in a loop for self-signed certificates · Issue #7680 · Icinga/icinga2 · GitHub with icinga2 version r2.14.2-1. After a recent rebuild of my organization’s icinga2 master, several clients are unable to connect due to certificate issues. When I run icinga2 ca sign for the satellites on the master, after a few minutes a new request shows up in icinga2 ca list as if the old request did nothing. How can I prevent this loop and finish establishing a connection between the satellite and the master?

  • Version used (icinga2 --version)
    • master: r2.14.2-1
    • satellite: r2.14.2-1
  • Operating System and version
    • master: Debian 12.5
    • satellite: Debian 12.5
  • Enabled features (icinga2 feature list)
    • master: Disabled features: command compatlog debuglog elasticsearch gelf icingadb influxdb influxdb2 journald livestatus opentsdb perfdata syslog Enabled features: api checker graphite ido-mysql mainlog notification
    • satellite: Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 journald livestatus notification opentsdb perfdata syslog Enabled features: api checker mainlog
master log 
[2024-08-11 00:00:07 +0000] warning/ApiListener: Certificate validation failed for endpoint 'test-redis02.transformativeworks.org': code 7: certificate signature failure
[2024-08-11 00:00:07 +0000] information/ApiListener: New client connection for identity 'test-redis02.transformativeworks.org' to [10.10.8.64]:5665 (certificate validation failed: code 7: certificate signature failure)
[2024-08-11 00:00:07 +0000] information/ApiListener: Finished reconnecting to endpoint 'test-redis02.transformativeworks.org' via host '10.10.8.64' and port '5665'
satellite logs 
[2024-08-10 16:51:04 +0000] information/ApiListener: New client connection for identity 'icinga-02.transformativeworks.org' from [::ffff:10.11.0.57]:29580
[2024-08-10 16:51:04 +0000] information/JsonRpcConnection: Requesting new certificate for this Icinga instance from endpoint 'icinga-02.transformativeworks.org'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Sending config updates for endpoint 'icinga-02.transformativeworks.org' in zone 'master'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Finished sending config file updates for endpoint 'icinga-02.transformativeworks.org' in zone 'master'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Syncing runtime objects to endpoint 'icinga-02.transformativeworks.org'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-02.transformativeworks.org'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-02.transformativeworks.org' in zone 'master'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Sending replay log for endpoint 'icinga-02.transformativeworks.org' in zone 'master'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Finished sending replay log for endpoint 'icinga-02.transformativeworks.org' in zone 'master'.
[2024-08-10 16:51:04 +0000] information/ApiListener: Finished syncing endpoint 'icinga-02.transformativeworks.org' in zone 'master'.
[2024-08-10 16:51:04 +0000] warning/JsonRpcConnection: API client disconnected for identity 'icinga-02.transformativeworks.org'
[2024-08-10 16:51:04 +0000] warning/ApiListener: Removing API client for endpoint 'icinga-02.transformativeworks.org'. 0 API clients left.
signing requests 
07d1cec275d3f8ac84f0f8b903f37fd5cbf06c4419221697c6366c1191de3a36 | Aug 10 00:51:16 2024 GMT | *      | CN = test-redis02.transformativeworks.org
20dd62c6fd62ad2685757f4640cf1638c43e48bfb08831bb704e8d11757ad8aa | Aug 11 17:41:42 2024 GMT | *      | CN = test-redis02.transformativeworks.org
2c286df167fdf675da1f35a91171295e127fc7ae501dac6caeef723dded7726a | Aug 10 16:50:21 2024 GMT | *      | CN = test-redis02.transformativeworks.org
437fdb5f49e576694c423f37b57b44056cf798a32a4cfa35e3853d2687e34d88 | Aug 11 17:42:27 2024 GMT |        | CN = test-redis02.transformativeworks.org
70ec7bdbd863d88c25946fdb1ee58b6ca82d9cfdfe7fe3b6af03fee905c08f5a | Aug 10 16:50:51 2024 GMT |        | CN = test-redis02.transformativeworks.org
c0ffa08bc08c608986db96c6cb9314186bd467e0f8794bcffe20da6e689103f8 | Aug 10 16:24:50 2024 GMT | *      | CN = test-redis02.transformativeworks.org
c9ab6d683055db2fee24664eeb69f8f5972331b3755f4416783b126011aa1e6a | Aug 10 00:51:15 2024 GMT | *      | CN = test-redis02.transformativeworks.org
2

When you say “rebuild”, do you mean a replacement of the master instance?
Especially a replacement of the CA?

3

Yes; due issues surrounding an OS upgrade I had to create a new VM and migrate the configuration from the one we’d used previously.

4

Did you also migrate /var/lib/icinga2?

Edit: I originally mentioned /usr/share/icinga2 here, which is the “wrong” directory in this context and should NOT be fiddled with.

1 Like
5

I did not (the existing host was in such a bad state that we could not log in); however, some of the satellite hosts that were monitored before can still connect fine.

It looks like I do still have access to the old disk, so if there’s something to copy over I can try that, but it’s non-trivial.

6

So, icinga2 stores runtime data in /var/lib/icinga2 including the certificates and keys for it’s own CA (all connections between icinga2 nodes are wrapped in TLS.
I wonder how you get any connections at all without resetting the satellites/agents too.
The critical files would be in /var/lib/icinga2/ca and /var/lib/icinga2/certs.

7

It looks like that was it! I had copied a handful of files from /var/lib/icinga2/certs (I believe just ca.crt and maybe 1 other). Grabbing the whole directory fixed the issue. Thanks!