Windows Agent needs to be restarted after ca sign on the master to connect

unic · October 21, 2019, 12:41pm

Hello,

I’am deploying Icinga Agents with the icinga2 Powershell Module and use the ca proxy feature. Everything works fine, but after I sign the certificate on the master, the agent is still not coming up. I need to restart the Windows service to get it up and running.

My Icinga Version is Icinga 2.11.1

The log on the client tells me endlessly that he is updating the certificate until I restart the agent.

[2019-10-21 14:25:37 +0200] information/JsonRpcConnection: Received certificate update message for CN 'testvm.dom.local'
[2019-10-21 14:25:37 +0200] information/JsonRpcConnection: Updating CA certificate in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//ca.crt'.
[2019-10-21 14:25:37 +0200] information/JsonRpcConnection: Updating client certificate for CN 'testvm.dom.local' in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//testvm.dom.local.crt'.
[2019-10-21 14:25:37 +0200] information/JsonRpcConnection: Updating the client certificate for CN 'testvm.dom.local' at runtime and reconnecting the endpoints.
[2019-10-21 14:25:37 +0200] warning/JsonRpcConnection: API client disconnected for identity 'sat03.dom.local'
[2019-10-21 14:25:38 +0200] warning/ApiListener: Removing API client for endpoint 'sat03.dom.local'. 0 API clients left.
[2019-10-21 14:25:47 +0200] information/ApiListener: New client connection for identity 'sat03.dom.local' from [::ffff:192.168.0.52]:45188
[2019-10-21 14:25:47 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Sending config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Finished sending config file updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Finished sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/ApiListener: Finished syncing endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:47 +0200] information/JsonRpcConnection: Received certificate update message for CN 'testvm.dom.local'
[2019-10-21 14:25:47 +0200] information/JsonRpcConnection: Updating CA certificate in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//ca.crt'.
[2019-10-21 14:25:47 +0200] information/JsonRpcConnection: Updating client certificate for CN 'testvm.dom.local' in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//testvm.dom.local.crt'.
[2019-10-21 14:25:47 +0200] information/JsonRpcConnection: Updating the client certificate for CN 'testvm.dom.local' at runtime and reconnecting the endpoints.
[2019-10-21 14:25:47 +0200] warning/JsonRpcConnection: API client disconnected for identity 'sat03.dom.local'
[2019-10-21 14:25:47 +0200] warning/ApiListener: Removing API client for endpoint 'sat03.dom.local'. 0 API clients left.
[2019-10-21 14:25:57 +0200] information/ApiListener: New client connection for identity 'sat03.dom.local' from [::ffff:192.168.0.52]:45192
[2019-10-21 14:25:57 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Sending config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Finished sending config file updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Finished sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/ApiListener: Finished syncing endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:25:57 +0200] information/JsonRpcConnection: Received certificate update message for CN 'testvm.dom.local'
[2019-10-21 14:25:57 +0200] information/JsonRpcConnection: Updating CA certificate in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//ca.crt'.
[2019-10-21 14:25:57 +0200] information/JsonRpcConnection: Updating client certificate for CN 'testvm.dom.local' in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//testvm.dom.local.crt'.
[2019-10-21 14:25:57 +0200] information/JsonRpcConnection: Updating the client certificate for CN 'testvm.dom.local' at runtime and reconnecting the endpoints.
[2019-10-21 14:25:57 +0200] warning/JsonRpcConnection: API client disconnected for identity 'sat03.dom.local'
[2019-10-21 14:25:57 +0200] warning/ApiListener: Removing API client for endpoint 'sat03.dom.local'. 0 API clients left.
[2019-10-21 14:26:07 +0200] information/ApiListener: New client connection for identity 'sat03.dom.local' from [::ffff:192.168.0.52]:45194
[2019-10-21 14:26:07 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Sending config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Finished sending config file updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Finished sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/ApiListener: Finished syncing endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:07 +0200] information/JsonRpcConnection: Received certificate update message for CN 'testvm.dom.local'
[2019-10-21 14:26:07 +0200] information/JsonRpcConnection: Updating CA certificate in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//ca.crt'.
[2019-10-21 14:26:07 +0200] information/JsonRpcConnection: Updating client certi14:27 21.10.2019ficate for CN 'testvm.dom.local' in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//testvm.dom.local.crt'.
[2019-10-21 14:26:07 +0200] information/JsonRpcConnection: Updating the client certificate for CN 'testvm.dom.local' at runtime and reconnecting the endpoints.
[2019-10-21 14:26:07 +0200] warning/JsonRpcConnection: API client disconnected for identity 'sat03.dom.local'
[2019-10-21 14:26:07 +0200] warning/ApiListener: Removing API client for endpoint 'sat03.dom.local'. 0 API clients left.
[2019-10-21 14:26:16 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2019-10-21 14:26:17 +0200] information/ApiListener: New client connection for identity 'sat03.dom.local' from [::ffff:192.168.0.52]:45198
[2019-10-21 14:26:17 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Sending config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Finished sending config file updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Finished sending replay log for endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/ApiListener: Finished syncing endpoint 'sat03.dom.local' in zone 'sat03.dom.local'.
[2019-10-21 14:26:17 +0200] information/JsonRpcConnection: Received certificate update message for CN 'testvm.dom.local'
[2019-10-21 14:26:17 +0200] information/JsonRpcConnection: Updating CA certificate in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//ca.crt'.
[2019-10-21 14:26:17 +0200] information/JsonRpcConnection: Updating client certificate for CN 'testvm.dom.local' in 'C:\ProgramData\icinga2\var\lib\icinga2/certs//testvm.dom.local.crt'.
[2019-10-21 14:26:17 +0200] information/JsonRpcConnection: Updating the client certificate for CN 'testvm.dom.local' at runtime and reconnecting the endpoints.
[2019-10-21 14:26:17 +0200] warning/JsonRpcConnection: API client disconnected for identity 'sat03.dom.local'
[2019-10-21 14:26:17 +0200] warning/ApiListener: Removing API client for endpoint 'sat03.dom.local'. 0 API clients left.

Best Regards,
Rafael

dnsmichi · October 21, 2019, 1:39pm

Can you verify that the signed certificate is written to %ProgramData%\var\lib\icinga\certs? This may be a permission problem.

Cheers,
Michael

unic · October 28, 2019, 8:25am

Hi Michael,

Sorry for my late response, I was not in the office to test it.

Yes, the certificate is there and I also checked so folder permissions, all looks fine. I will test it on a fresh vm again to make sure that the certs are there after setup and not after the second restart.

Greets,
Rafael

unic · November 11, 2019, 8:21am

Hi @dnsmichi,

I now had the possibility to check the folders. Everything looks fine:

After signing the certificate on the master, the agent still does not connect. I have tested it on 4 servers now. A simple restart of the service after the certificate is signed solves the problem. So after auto deployment and signing, someone still needs to restart the agent once.

unic · November 22, 2019, 1:18pm

After moving from Powershell Module to the new Powershell framework its still the same problem. Without restarting the Service on the Windowsservers after signing the cert on the master, the agent will not connect.

To clarify: The agents have no connection to the Master/Satellite because of Firewall rules, so they are waiting for a connection from the master. But this connection is not successfull until i restart the windowsservice.

Zone ‘agentzone’ is not connected. Log lag: less than 1 millisecond