CA issue - new agents not able to commuicate / certificate signature failure

Icinga 2
1

I switched everything master to containers. Not sure if it started with that change, but I cannot connect new agents to the master anymore. I see certificate errors and have no more ideas what to check.

Master seems to not accept the certificate signature sent by Agent, Agent keeps sending requests for signing. When I sign, request pops up again after a minute.

icinga-podman@<master-hostname>:~ $ podman ps
CONTAINER ID  IMAGE                               COMMAND               CREATED       STATUS           PORTS                   NAMES
d906725fab1d  localhost/influxdb:1.8              influxd               8 months ago  Up 3 weeks ago   0.0.0.0:8086->8086/tcp  influxdb
5de58a7728d3  localhost/grafana/grafana:9.1.8                           8 months ago  Up 3 weeks ago   0.0.0.0:3000->3000/tcp  grafana
4084dc975210  localhost/mariadb:10.9.3            mariadbd              8 months ago  Up 5 weeks ago                           mariadb-icingaweb2
c3346af680b1  localhost/icinga/icingaweb2:2.11.3  bash -eo pipefail...  8 months ago  Up 5 weeks ago   0.0.0.0:8080->8080/tcp  icingaweb2
6b4bd51840e5  localhost/mariadb:10.9.3            mariadbd              8 months ago  Up 3 weeks ago   0.0.0.0:3306->3306/tcp  mariadb-icingadb
1c0c7effc819  localhost/icinga/icinga2:2.13.6     icinga2 daemon        8 months ago  Up 19 hours ago  0.0.0.0:5665->5665/tcp  icinga-master
ff6fbf6284f4  localhost/icinga/icingadb:1.1.0     /entrypoint           5 weeks ago   Up 47 hours ago                          icingadb
0ad84de9d51e  localhost/redis:7.0.5               redis-server          5 weeks ago   Up 2 days ago    0.0.0.0:6379->6379/tcp  icinga-redis


[2023-09-09 09:17:48 +0000] information/ApiListener: Reconnecting to endpoint '<agent-hostname>' via host '<agent-hostname>' and port '5665'
[2023-09-09 09:17:51 +0000] information/ApiListener: New client connection for identity '<agent-hostname>' from [::ffff:10.89.0.80]:34870 (certificate validation failed: code 7: certificate signature failure)
[2023-09-09 09:17:51 +0000] information/JsonRpcConnection: Received certificate request for CN '<agent-hostname>' not signed by our CA: certificate signature failure (code 7)
[2023-09-09 09:17:51 +0000] information/JsonRpcConnection: Certificate request for CN '<agent-hostname>' is pending. Waiting for approval.
[2023-09-09 09:17:51 +0000] warning/JsonRpcConnection: API client disconnected for identity '<agent-hostname>'

Zones.conf

  1 /*
  2  * Generated by Icinga 2 node setup commands
  3  * on 2023-01-09 11:56:41 +0100
  4  */
  5
  6 object Endpoint "<master-hostname>" {
  7    host = "<master-hostname>"
  8 }
  9
 10 object Zone "master" {
 11    endpoints = [ "<master-hostname>" ]
 12 }
 13
 14 object Zone "global-templates" {
 15    global = true
 16 }
 17
 18 object Zone "director-global" {
 19    global = true
 20 }
 21
 22 object Zone "linux-commands" {
 23    global = true
 24 }
 25
 26 object Zone "windows" {
 27    global = true
 28 }
 29
 30

Hostconfig Agent

  1 object Endpoint "<agent-hostname>" {
  2    host = "<agent-hostname>"
  3 }
  4
  5 object Zone "<agent-hostname>" {
  6    endpoints = [ "<agent-hostname>" ]
  7    parent = "master"
  8 }
  9
 10 object Host "<agent-hostname>" {
 11    import "linux-servers-tpl"
 12    address = "<agent-hostname>"
 13    zone = "master"
 14         groups = [ "linux-servers", "repo-servers" ]
.
.
.
2

Hello Daneil!

How exactly do you sign them?

Best,
A/K

3

“Both” ways. Tried to create a ticket before the agent node wizard and to sign the request on the master after agent node wizard as well.

This started after switching to rootless podman (I think…). Will try podman running as root as soon as I have time to test. All the other certificates from before the switch running fine, but I fear they stop working once they expire.

4

Is the hostname of your master is correct? I guess you know podman exec -it icinga-master hostname.

Have you compared old and new signed certificates e.g. openssl x509 ...?

5

Folks, sorry I havent answered for so long. Just to much work.

So, certs are the same on master and client in /var/lib/icinga2/certs/ca.crt

It is somehow the signing in the master - no idea whats wrong.

Clients are installed locally, master in a container. All the old connections work, only new connections will not connect. Spent another hour there and still no clue.

Oh, ps: I have no idea what “podman exec -it icinga-master hostname” does…

6

Please share the master debug log being written while connecting a completely new agent via ticket.

1 Like
7

I finally got it done and we can close this discussion :slight_smile:

Issue was I was only paying attention to the ca files, not to the master host certificate file itself. I thought, the ca files matter and was comparing them btw master and client and they always matched.
Somehow the host certificate file and the ca file did not match anymore after the move. I re-created the whole chain with ca and host certificates and had to re-run the node wizard on all clients. Certificate request signing works again now.

Lessons learned:

  • master / slave signing depends on the master host certificate
  • I need some automation for client certificate signing