CA issue - new agents not able to commuicate / certificate signature failure

I switched everything master to containers. Not sure if it started with that change, but I cannot connect new agents to the master anymore. I see certificate errors and have no more ideas what to check.

Master seems to not accept the certificate signature sent by Agent, Agent keeps sending requests for signing. When I sign, request pops up again after a minute.

icinga-podman@<master-hostname>:~ $ podman ps
CONTAINER ID  IMAGE                               COMMAND               CREATED       STATUS           PORTS                   NAMES
d906725fab1d  localhost/influxdb:1.8              influxd               8 months ago  Up 3 weeks ago   0.0.0.0:8086->8086/tcp  influxdb
5de58a7728d3  localhost/grafana/grafana:9.1.8                           8 months ago  Up 3 weeks ago   0.0.0.0:3000->3000/tcp  grafana
4084dc975210  localhost/mariadb:10.9.3            mariadbd              8 months ago  Up 5 weeks ago                           mariadb-icingaweb2
c3346af680b1  localhost/icinga/icingaweb2:2.11.3  bash -eo pipefail...  8 months ago  Up 5 weeks ago   0.0.0.0:8080->8080/tcp  icingaweb2
6b4bd51840e5  localhost/mariadb:10.9.3            mariadbd              8 months ago  Up 3 weeks ago   0.0.0.0:3306->3306/tcp  mariadb-icingadb
1c0c7effc819  localhost/icinga/icinga2:2.13.6     icinga2 daemon        8 months ago  Up 19 hours ago  0.0.0.0:5665->5665/tcp  icinga-master
ff6fbf6284f4  localhost/icinga/icingadb:1.1.0     /entrypoint           5 weeks ago   Up 47 hours ago                          icingadb
0ad84de9d51e  localhost/redis:7.0.5               redis-server          5 weeks ago   Up 2 days ago    0.0.0.0:6379->6379/tcp  icinga-redis

[2023-09-09 09:17:48 +0000] information/ApiListener: Reconnecting to endpoint '<agent-hostname>' via host '<agent-hostname>' and port '5665'
[2023-09-09 09:17:51 +0000] information/ApiListener: New client connection for identity '<agent-hostname>' from [::ffff:10.89.0.80]:34870 (certificate validation failed: code 7: certificate signature failure)
[2023-09-09 09:17:51 +0000] information/JsonRpcConnection: Received certificate request for CN '<agent-hostname>' not signed by our CA: certificate signature failure (code 7)
[2023-09-09 09:17:51 +0000] information/JsonRpcConnection: Certificate request for CN '<agent-hostname>' is pending. Waiting for approval.
[2023-09-09 09:17:51 +0000] warning/JsonRpcConnection: API client disconnected for identity '<agent-hostname>'

Zones.conf

  1 /*
  2  * Generated by Icinga 2 node setup commands
  3  * on 2023-01-09 11:56:41 +0100
  4  */
  5
  6 object Endpoint "<master-hostname>" {
  7    host = "<master-hostname>"
  8 }
  9
 10 object Zone "master" {
 11    endpoints = [ "<master-hostname>" ]
 12 }
 13
 14 object Zone "global-templates" {
 15    global = true
 16 }
 17
 18 object Zone "director-global" {
 19    global = true
 20 }
 21
 22 object Zone "linux-commands" {
 23    global = true
 24 }
 25
 26 object Zone "windows" {
 27    global = true
 28 }
 29
 30

Hostconfig Agent

  1 object Endpoint "<agent-hostname>" {
  2    host = "<agent-hostname>"
  3 }
  4
  5 object Zone "<agent-hostname>" {
  6    endpoints = [ "<agent-hostname>" ]
  7    parent = "master"
  8 }
  9
 10 object Host "<agent-hostname>" {
 11    import "linux-servers-tpl"
 12    address = "<agent-hostname>"
 13    zone = "master"
 14         groups = [ "linux-servers", "repo-servers" ]
.
.
.

Hello Daneil!

How exactly do you sign them?

Best,
A/K

“Both” ways. Tried to create a ticket before the agent node wizard and to sign the request on the master after agent node wizard as well.

This started after switching to rootless podman (I think…). Will try podman running as root as soon as I have time to test. All the other certificates from before the switch running fine, but I fear they stop working once they expire.

Is the hostname of your master is correct? I guess you know podman exec -it icinga-master hostname.

Have you compared old and new signed certificates e.g. openssl x509 ...?