Remote Icinga instance 'Windows Agent' is not connected to 'Icinga master'

Figment · August 11, 2021, 5:43pm

Hey Team,

I’ve looked through similar threads but I’m not making much headway with this issue.

I’ve deployed the Icinga agent to most windows machines in my environment using the directors self service API however it no longer works with any new machines.

I use the below PowerShell script to register with my master server.

Use-Icinga;
Start-IcingaAgentInstallWizard `
    -DirectorUrl            'https://icinga.domain.com/icingaweb2/director/' `
    -SelfServiceAPIKey      'APIKEY' `
    -UseDirectorSelfService 1 `
    -OverrideDirectorVars   0 `
    -Reconfigure `
	-CAFile '' `
	-EmptyCA 1 `
	-RunInstaller

This works correctly and the host appears in the deploy section of the director.

However non of the services work, they all state

“Remote Icinga instance ‘Windows Agent’ is not connected to ‘Icinga master’”

My zones.conf appears to be the same as other working hosts.

object Endpoint "hostname" {
}

object Endpoint "icinga.domain.com" {
    host = "icinga.domain.com";
    port = "5665";
}

object Zone "master" {
    endpoints = [ "icinga.domain.com" ];
}

object Zone "hostname" {
    parent = "master";
    endpoints = [ "hostname" ];
}

object Zone "director-global" {
    global = true;
}

The logs aren’t giving me much to go off either.

Icinga2.txt

[2021-08-11 17:59:22 +0100] information/FileLogger: 'main-log' started.
[2021-08-11 17:59:22 +0100] information/NotificationComponent: 'notification' started.
[2021-08-11 17:59:22 +0100] information/CheckerComponent: 'checker' started.
[2021-08-11 17:59:22 +0100] information/ConfigItem: Activated all objects.
[2021-08-11 17:59:39 +0100] information/FileLogger: 'main-log' started.
[2021-08-11 17:59:39 +0100] information/ConfigItem: Activated all objects.
[2021-08-11 18:04:39 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-08-11 18:09:39 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-08-11 18:16:01 +0100] information/FileLogger: 'main-log' started.
[2021-08-11 18:16:01 +0100] information/ConfigItem: Activated all objects.
[2021-08-11 18:21:01 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-08-11 18:23:05 +0100] information/FileLogger: 'main-log' started.
[2021-08-11 18:23:05 +0100] information/FileLogger: 'debug-file' started.
[2021-08-11 18:23:05 +0100] information/ConfigItem: Activated all objects.
[2021-08-11 18:28:05 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-08-11 18:33:05 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'

debug.txt

[2021-08-11 18:23:05 +0100] information/FileLogger: 'debug-file' started.
[2021-08-11 18:23:05 +0100] information/ConfigItem: Activated all objects.
[2021-08-11 18:23:05 +0100] notice/WorkQueue: Stopped WorkQueue threads for 'DaemonCommand::Run'
[2021-08-11 18:23:05 +0100] notice/ApiListener: Updating object authority for local objects.
[2021-08-11 18:23:05 +0100] debug/IcingaApplication: In IcingaApplication::Main()
[2021-08-11 18:28:05 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-08-11 18:33:05 +0100] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'

Lastly here’s the rest of the details of my setup.

Icinga Version 2.13.0-1
Icinga Web Version 2.9.2
Director Version 1.8.0
Icinga for Windows Version 2.12.4

Anyone have any ideas why or where I can start looking? multiple re-installs and even trying newer Agent versions doesn’t seem to make a difference.

stevie-sy · August 11, 2021, 6:50pm

Hi

is the log output from the agent or from the master/satellite? If the first point, are there any hints/errors/… in the log on the master/satellite servers?

Figment · August 12, 2021, 9:04am

Hi Stevie,

This log is from the Windows Agent.

Here’s the errors from log on the master server.

[2021-08-11 17:59:29 +0100] information/ApiListener: New client connection from [::ffff:hostip]:62925 (no client certificate)
[2021-08-11 17:59:29 +0100] information/ApiListener: No data received on new API connection from [::ffff:hostip]:62925. Ensure that the remote endpoints are properly configured in a cluster setup.
[2021-08-11 17:59:29 +0100] information/ApiListener: New client connection for identity 'HOSTNAME' from [::ffff:hostip]:62926 (certificate validation failed: code 18: self signed certificate)
[2021-08-11 17:59:29 +0100] information/JsonRpcConnection: Received certificate request for CN 'HOSTNAME' not signed by our CA: self signed certificate (code 18)
[2021-08-11 17:59:29 +0100] warning/JsonRpcConnection: Ticket 'ticket' for CN 'HOSTNAME' is invalid.
[2021-08-11 17:59:29 +0100] warning/JsonRpcConnection: API client disconnected for identity 'HOSTNAME'

Thanks
Ash

log1c · August 12, 2021, 9:15am

check icinga2 ca list on the master. I`m pretty sure there are pending cert requests for the agents.
To have them automatically signed the agent would need the CA cert present at rollout, IIRC.

Figment · August 12, 2021, 9:18am

Odd that I didn’t have to sign any certs before (a lot of my other servers are working just fine)

I’ve since signed this cert for my host but its still not working, do I need to restart the service on my agent?

log1c · August 12, 2021, 9:21am

Normally not, the master should then push the config to the agent host after some time.

Figment · August 12, 2021, 9:22am

thanks Log1c! I’ll check back in like 30 mins to see if there’s any difference.

In terms of the CA cert, I’m still fairly new to Icinga, is the CA cert located on the master server and if so where would I find it?

stevie-sy · August 12, 2021, 10:28am

The cert requets are saved in /var/lib/icinga2/certificate-requests. But you should use the commands as @log1c mentioned. Everything about ca with icinga2 you’ll find here: Distributed Monitoring - Icinga 2

rsx · August 12, 2021, 11:00am

You have at your master:

but no corresponding entry at your agent? This cannot be true.

And you should have a look at:

Figment · August 13, 2021, 10:37am

thanks everyone for the help so far, I’ve tried including the CA in the deployment script as below;

Use-Icinga;
Start-IcingaAgentInstallWizard `
    -UseDirectorSelfService 1 `
    -DirectorUrl 'https://icinga.domain.com/icingaweb2/director/' `
    -SelfServiceAPIKey '9dfe6463e78e1ff9abd73a31569e405418be6ab6' `
    -OverrideDirectorVars 0 `
    -ConvertEndpointIPConfig 1 `
    -AcceptConnections 1 `
    -EmptyCA 0 `
    -CAFile 'C:\IT\Icinga\ca.crt' `
    -InstallFrameworkPlugins 0 `
    -InstallFrameworkService 0 `
    -RunInstaller

however I’m still getting a few errors that seem to be related to failing to sign the certificate, “Invalid Ticket” appearing to be the issue. I generated a new API Key in director but still not getting anywhere at this step.

[Error]: The remote host for address "https://icinga.domain.com/icingaweb2/di
rector/self-service/powershell-parameters?key=001b8848fdb555bee47e9124a2558a63dcf8c4ab" co
uld not be resolved
[Error]: Your local stored host key is no longer valid. Using provided template key
[Warning]: Your Windows system does not support "Get-NetRoute". A fallback solution is used to fet
ch the IP of the first Network Interface routing through 0.0.0.0
[Notice]: Host was successfully registered within Icinga Director
[Notice]: Using hostname "HOSTNAME" for the Icinga Agent configuration
[Notice]: The wizard is complete. These are the configured settings:
========
-UseDirectorSelfService 1
-DirectorUrl 'https://icinga.domain.com/icingaweb2/director/'
-SelfServiceAPIKey '9dfe6463e78e1ff9abd73a31569e405418be6ab6'
-AcceptConnections 1
-OverrideDirectorVars 0
-InstallFrameworkService 0
-InstallFrameworkPlugins 0
-ConvertEndpointIPConfig 1
-CAFile 'C:\IT\Icinga\ca.crt'
-EmptyCA 0
-InstallFrameworkService 0
-RunInstaller

============
Start-IcingaAgentInstallWizard -UseDirectorSelfService 1 -DirectorUrl 'https://icinga.domain.com/icingaweb2/director/' -SelfServiceAPIKey '9dfe6463e78e1ff9abd73a
31569e405418be6ab6' -AcceptConnections 1 -OverrideDirectorVars 0 -InstallFrameworkService 
0 -InstallFrameworkPlugins 0 -ConvertEndpointIPConfig 1 -CAFile 'C:\IT\Icinga\ca.crt
' -EmptyCA 0 -InstallFrameworkService 0 -RunInstaller
============

[Notice]: Found installed .NET Framework version 4.8.03761
[Notice]: Downloading Icinga 2 Agent installer "Icinga2-v2.12.4-x86_64.msi" into temp directory "
C:\Users\$Admin\AppData\Local\Temp\Icinga2-v2.12.4-x86_64.msi"
[Notice]: Installing new Icinga Agent version into "C:\Program Files\ICINGA2"
[Notice]: Icinga Agent was successfully installed
[Notice]: Successfully backed up Icinga 2 Agent default config
[Notice]: Your hostname was successfully changed to "HOSTNAME"
[Notice]: The Icinga Service User already has permission to run as service
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writeable by the Icinga 
Service User "domain\$admin"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writeable by the Icinga 
Service User "domain\$admin"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework
\cache" is accessible and writeable by the Icinga Service User "domain\$admin"
[Notice]: Service User successfully updated
[Notice]: Background daemon Cmdlet "Start-IcingaServiceCheckDaemon" has been configured
[Notice]: Feature "checker" was successfully disabled
[Notice]: Feature "notification" was successfully disabled
[Notice]: Feature "api" was successfully enabled
[Notice]: Generating host certificates for host "HOSTNAME"
[Notice]: information/base: Writing private key to 'C:\ProgramData\icinga2\var\lib\icinga2\certs\
HOSTNAME.key'.
information/base: Writing X509 certificate to 'C:\ProgramData\icinga2\var\lib\icinga2\cert
s\HOSTNAME.crt'.
[Notice]: Fetching trusted master certificate from "icinga.domain.com"
[Notice]: information/cli: Retrieving TLS certificate for 'icinga.domain.com:5
665'.

 Version:             3
 Subject:             CN = icinga.domain.com
 Issuer:              CN = Icinga CA
 Valid From:          Mar 18 15:52:24 2021 GMT
 Valid Until:         Mar 14 15:52:24 2036 GMT
 Serial:              b2:ff:db:b6:ef:14:d9:44:79:b9:03:c0:5d:07:20:05:4d:6d:ed:14

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   icinga.domain.com
 Fingerprint:         77 0C E7 55 0C 1E A1 76 EC 94 F4 11 8D C8 33 E4 03 CE 64 22 54 81 1C
 3E E2 3F 0B 02 2C D8 A7 E8 

***
*** You have to ensure that this certificate actually matches the parent
*** instance's certificate in order to avoid man-in-the-middle attacks.
***

information/pki: Writing certificate to file 'C:\ProgramData\icinga2\var\lib\icinga2\certs
\trusted-parent.crt'.
[Error]: Failed to create certificate.
Arguments: pki request --host icinga.domain.com --port 5665 --ticket 7b
7b80813812f75bda7177b15cde7151122c98fa --key C:\ProgramData\icinga2\var\lib\icinga2\certs\
HOSTNAME.key --cert C:\ProgramData\icinga2\var\lib\icinga2\certs\HOSTNAME.cr
t --trustedcert C:\ProgramData\icinga2\var\lib\icinga2\certs\trusted-parent.crt --ca C:\Pr
ogramData\icinga2\var\lib\icinga2\certs\ca.crt
Error:information/cli: Writing CA certificate to file 'C:\ProgramData\icinga2\var\lib\icin
ga2\certs\ca.crt'.
critical/cli: !!! Invalid ticket for CN 'HOSTNAME'. 
[Error]: Failed to sign Icinga certificate
[Notice]: Api configuration has been written successfully
[Notice]: Icinga Agent zones.conf has been written successfully
[Notice]: Successfully removed Icinga firewall rule
[Notice]: Successfully enabled firewall for port "5665"
[Passed]: Icinga Agent service is installed
[Passed]: The specified user "domain\$admin" is allowed to run as service
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writeable by the Icinga 
Service User "domain\$admin"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writeable by the Icinga 
Service User "domain\$admin"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework
\cache" is accessible and writeable by the Icinga Service User "domain\$admin"
[Passed]: Icinga Agent configuration is valid
[Passed]: Icinga Agent debug log is disabled
[Notice]: Restarting service "icinga2"

Any Ideas as to where I am going wrong? Reading through the docs I can’t see what I’m missing but honestly I could just be going brain blind after being at this for days!