The Icinga for Windows REST-Api was not able to fetch the Icinga Agent or icingaforwindows.pfx certificate file

Aleksey-Maksimov · June 14, 2024, 7:10am

Hello.

The Icinga agent and service were successfully installed on Windows Server 2022 via the IMC console.
After installing the agent, we configured it using the icinga2-agent-kickstart.ps1 script from Icinga Director.
Standard TCP port 5665 is working and the agent successfully connected to Icinga Director (Invoke-IcingaCheckCPU check is working)
The “Icinga 2” and “Icinga PowerShell Service” services are running.

Due to the high processor load during active checks, we would like to use the REST-Api functionality

But we can’t get REST-Api running in Icinga for Windows Server 2022.
TCP listener does not appear on port 5668.

A warning appears in the event log “Icinga for Windows” once a minute:

Log Name:      Icinga for Windows
Source:        IfW::RESTApi
Date:          13.06.2024 17:04:44
Event ID:      2002
Task Category: (1)
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      WinHost002.holding.com
Description:
Icinga for Windows certificate not ready

The Icinga for Windows REST-Api was not able to fetch the Icinga Agent or icingaforwindows.pfx certificate file. You can manually enforce the certificate creation of the icingaforwindows.pfx by using the command "Start-IcingaWindowsScheduledTaskRenewCertificate". Once successful, this message should disappear and the REST-Api start in case you are running inside a JEA-Context. If you are not using JEA, the Icinga Agent certificate has to be present and signed by the Icinga CA. You can test if a certificate is present by using "Get-IcingaSSLCertForSocket". This should return a certificate object with the subject "CN=<hostname>", while "<hostname>" should match your hostname or object name in Icinga. This check is queued every 5 minutes and should vanish once everything works fine.

Checking with the Test-IcingaForWindows cmdlet says there is no certificate

Test-IcingaForWindows

[Notice]: Collecting Icinga for Windows environment information
[Passed]: The Icinga Agent service and the Icinga Agent are installed on the system
[Passed]: The Icinga for Windows service is installed on the system
[Passed]: The Icinga for Windows service binary does exist: "C:\Program Files\icinga-framework-service\icinga-service.exe"
[Passed]: Your service installation is not affected by IWKB000009
[Passed]: Your service installation is properly referring to "icinga-powershell-framework.psd1" for module imports.
[Passed]: The Icinga Agent service user "NT AUTHORITY\NetworkService" is matching the Icinga for Windows service user "NT Authority\NetworkService"
[Passed]: The specified user "NT AUTHORITY\NetworkService" is allowed to run as service
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\config" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\certificate" is accessible and writable by the Icinga Service User "NT AUTHORITY\NetworkService"
[Passed]: The Icinga Agent state file is healthy
[Passed]: Icinga Agent configuration is valid
[Passed]: Icinga Agent debug log is disabled
[Passed]: The Icinga for Windows REST-Api is configured to start with the daemon
[Passed]: The Icinga for Windows REST-Api is configured to allow API checks
[Failed]: The Icinga for Windows certificate is not installed on the system
[Warning]: Icinga for Windows is configured without a JEA-Profile. It is highly recommended to use JEA for advanced security and easier permission handling
[Passed]: The Icinga for Windows service is running
[Failed]: The Icinga for Windows REST-Api responded with an error on "https://localhost:5668/v1", which is expected when using the default NetworkService account [IWKB000018]: "Unable to connect to the remote server"

The Get-IcingaSSLCertForSocket cmdlet does not return anything

The Start-IcingaWindowsScheduledTaskRenewCertificate cmdlet starts the Windows scheduler task, it runs, but nothing changes

If we manually run the script “C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\jobs\RenewCertificate.ps1” then an error appears

[Error]: Unable to install Icinga for Windows certificate, as with specified arguments and auto-lookup for Icinga Agent certificate, no certificate could be created

Environment configuration:

PowerShell Root                 => C:\Program Files\WindowsPowerShell\Modules\
Icinga for Windows Service Path => C:\Program Files\icinga-framework-service\
Icinga for Windows Service User => NT Authority\NetworkService
Icinga for Windows Service Pid  => 1124
Icinga for Windows JEA Pid      =>
Icinga Agent Path               => C:\Program Files\ICINGA2\
Icinga Agent User               => NT AUTHORITY\NetworkService
Defined Default User            => NT Authority\NetworkService
Icinga Managed User             => False
PowerShell Version              => 5.1.20348.2110
Operating System                => Microsoft Windows Server 2022 Standard
Operating System Version        => 10.0.20348
JEA Context                     =>
JEA Session File                =>
Api Check Forwarder             => True
Debug Mode                      => False

Icinga for Windows Certificate:

Not installed

List of configured background daemons on this system:

Start-IcingaWindowsRESTApi
-----------
No arguments defined

List of configured background service checks on this system:
=> https://icinga.com/docs/icinga-for-windows/latest/doc/110-Installation/06-Collect-Metrics-over-Time/

No background service checks configured

List of configured repositories on this system. The list order matches the apply order:

Icinga Stable
-----------
CloneSource  =>
Enabled      => True
LocalPath    =>
Order        => 0
RemotePath   => https://packages.icinga.com/IcingaForWindows/stable/ifw.repo.json
UseSCP       => False

Installed components on this system:

Component    Version   Available
---          ---       ---
agent        2.14.2    2.14.2
apichecks    1.2.0     1.2.0
cluster      1.3.0     1.3.0
framework    1.12.3    1.12.3
hyperv       1.3.0     1.3.0
inventory    1.2.0     1.2.0
kickstart    1.4.0
mssql        1.5.0     1.5.0
plugins      1.12.0    1.12.0
restapi      1.2.0     1.2.0
service      1.2.0     1.2.0

Please tell me how we can solve this problem

Aleksey-Maksimov · June 14, 2024, 8:34am

As I said earlier, the Start-IcingaWindowsScheduledTaskRenewCertificate cmdlet executes, but the icingaforwindows.pfx file does not appear in the C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\certificate\ directory

Investigation of the situation showed that the root of the problem is in the Get-IcingaAgentHostCertificate script from the C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\lib\core\icingaagent\getters\ directory

It contains the line:

$Hostname = Get-IcingaHostname -ReadConstants;

A test run of Get-IcingaHostname –ReadConstants shows that the value “localhost” is returned, which makes it impossible to find the Icinga agent certificate file with that name in the C:\ProgramData\icinga2\var\lib\icinga2\certs\ directory. That is, the localhost.crt file is searched, although, for example, in our case the file has a name in the format winhost002.holding.com.crt.

The experiment showed that the problem can be solved if in the Get-IcingaAgentHostCertificate script we replace the line with:

$Hostname = Get-IcingaHostname -LowerCase 1 -AutoUseFQDN 1;

If I understand correctly, the Get-IcingaAgentHostCertificate script needs to be fixed.

rivad · June 14, 2024, 10:32am

Maybe the value Get-IcingaHostname -ReadConstants returns needs to be changed.
Not sure if it’s defined in c:\appdata\icinga\etc\icinga\constants.conf or a similar path.

Aleksey-Maksimov · June 14, 2024, 7:47pm

Hello Dominik

Thanks for the answer and yes, you are right.

As I understand it, the Icinga Director configuration script does not change the C:\ProgramData\icinga2\etc\icinga2\constants.conf file.
I manually changed the line in the file
const NodeName = “localhost”
per line
const NodeName = “winhost002.holding.com”

After this, the Start-IcingaWindowsScheduledTaskRenewCertificate command began to work normally.

theFeu · June 17, 2024, 10:21am

Hey there! I would like to ask you to mark an answer as the solution, if it resolves your topic.