Setup agent node without connecting to the master

Hi Icinga Community,

I’m extending an existing monitoring enviroment with some Windows Servers 2019.
For security reasons one of them is not allowed to connect from the agent to the master.
So using the Icinga Wizard is not possible as it can’t fetch a certificate from the master.

I’ve found a similar article from 2yrs ago but it doesn’t work for me the same way:

##############################

  • OS master: Debian GNU/Linux, platform version 10
  • Version master: The Icinga 2 network monitoring daemon (version: r2.12.3-1)
  • OS agent: Windows Servers 2019
  • Version agent: Icinga2-v2.12.4
  • Enabled features on agent: api mainlog notification
  • Disabled features on agent: checker debuglog elasticsearch gelf graphite influxdb opentsdb perfdata

I run only one master with several agents connected. Using the Icinga Wizard works absolutely fine for all other agents. But not using the Wizard fails.

The agent was integrated in the monitoring and performs successfully a ping check. But the master can’t fetch any remote check results from the agent

Icinga2-Service is running on the agent
But a telnet from the master to the agent on port 5665 fails, connection refused.

Using netstat -a on the agent shows, that the port 5665 is not active (not listed)
##############################

What I’ve done:

  1. I’ve installed Icinga on the Windows Server, then created and signed the certificate on the master with the following two commands:
cinga2 pki new-cert --cn server.foo.bar --key server.foo.bar.key --csr server.foo.bar.csr
icinga2 pki sign-csr --csr server.foo.bar.csr --cert server.foo.bar.crt
  1. Then copied the certificate and the key into /var/lib/icinga2/certs on the agent node
  2. Then copied the ca.crt from /var/lib/icinga2/ca (on the master) in the same folder on the agent
  3. Enabled the api-feature with C:\Program Files\ICINGA2\sbin> .\icinga2.exe feature enable api

Checking the log-files on icinga master and agent, I don’t get much information.
Log entry from master:

[2021-07-09 12:57:04 +0200] information/ApiListener: Reconnecting to endpoint 'ms2019p-epdhcp.domain.com' via host 'ms2019p-epdhcp.domain.com' and port '5665'
[2021-07-09 12:57:04 +0200] critical/ApiListener: Cannot connect to host 'ms2019p-epdhcp.domain.com' on port '5665': Connection refused
[2021-07-09 12:57:14 +0200] information/ApiListener: Reconnecting to endpoint 'ms2019p-epdhcp.domain.com' via host 'ms2019p-epdhcp.domain.com' and port '5665'
[2021-07-09 12:57:14 +0200] critical/ApiListener: Cannot connect to host 'ms2019p-epdhcp.domain.com' on port '5665': Connection refused
[2021-07-09 12:57:24 +0200] information/ApiListener: Reconnecting to endpoint 'ms2019p-epdhcp.domain.com' via host 'ms2019p-epdhcp.domain.com' and port '5665'
[2021-07-09 12:57:24 +0200] critical/ApiListener: Cannot connect to host 'ms2019p-epdhcp.domain.com' on port '5665': Connection refused
[2021-07-09 12:57:34 +0200] information/ApiListener: Reconnecting to endpoint 'ms2019p-epdhcp.domain.com' via host 'ms2019p-epdhcp.domain.com' and port '5665'
[2021-07-09 12:57:34 +0200] critical/ApiListener: Cannot connect to host 'ms2019p-epdhcp.domain.com' on port '5665': Connection refused
[2021-07-09 12:57:44 +0200] information/ApiListener: Reconnecting to endpoint 'ms2019p-epdhcp.domain.com' via host 'ms2019p-epdhcp.domain.com' and port '5665'
[2021-07-09 12:57:44 +0200] critical/ApiListener: Cannot connect to host 'ms2019p-epdhcp.domain.com' on port '5665': Connection refused


Log entry from agent: (comes daily)

[2021-07-08 02:00:03 +0200] information/Checkable: Checkable 'MS2019P-EPDHCP!load' has 1 notification(s). Checking filters for type 'DowntimeStart', sends will be logged.
[2021-07-08 02:00:03 +0200] information/Notification: Sending 'DowntimeStart' notification 'MS2019P-EPDHCP!load!mail-icingaadmin' for user 'icingaadmin'
[2021-07-08 02:00:03 +0200] information/Downtime: Triggering downtime 'MS2019P-EPDHCP!load!6b853396-81b9-4c95-b64c-d5d1576a3996' for checkable 'MS2019P-EPDHCP!load'.
[2021-07-08 02:00:03 +0200] warning/PluginUtility: Error: Non-optional macro 'service.output' used in argument '-o' is missing.


[2021-07-08 02:00:03 +0200] warning/PluginNotificationTask: Notification command for object 'MS2019P-EPDHCP!load' (PID: 4294967295, arguments: '') terminated with exit code 3, output: Error: Non-optional macro 'service.output' used in argument '-o' is missing.


[2021-07-08 02:00:03 +0200] information/Notification: Completed sending 'DowntimeStart' notification 'MS2019P-EPDHCP!load!mail-icingaadmin' for checkable 'MS2019P-EPDHCP!load' and user 'icingaadmin' using command 'mail-service-notification'.
[2021-07-08 02:00:23 +0200] information/ConfigObjectUtility: Created and activated object 'MS2019P-EPDHCP!load!551240ba-0bad-4460-933e-4f72440fae0f' of type 'Downtime'.
[2021-07-08 02:00:23 +0200] information/Downtime: Added downtime 'MS2019P-EPDHCP!load!551240ba-0bad-4460-933e-4f72440fae0f' between '2021-07-09 02:00:00' and '2021-07-09 03:00:00', author: 'icingaadmin', fixed
[2021-07-08 02:04:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:09:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:14:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:19:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:24:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:29:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:34:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:39:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:44:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:49:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:54:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 02:59:18 +0200] information/ConfigObject: Dumping program state to file 'C:\ProgramData\icinga2\var\lib\icinga2/icinga2.state'
[2021-07-08 03:00:24 +0200] information/Checkable: Checkable 'MS2019P-EPDHCP!load' has 1 notification(s). Checking filters for type 'DowntimeEnd', sends will be logged.
[2021-07-08 03:00:24 +0200] information/Notification: Sending 'DowntimeEnd' notification 'MS2019P-EPDHCP!load!mail-icingaadmin' for user 'icingaadmin'
[2021-07-08 03:00:24 +0200] warning/PluginUtility: Error: Non-optional macro 'service.output' used in argument '-o' is missing.


[2021-07-08 03:00:24 +0200] information/ConfigObjectUtility: Deleted object 'MS2019P-EPDHCP!load!6b853396-81b9-4c95-b64c-d5d1576a3996' of type 'Downtime'.
[2021-07-08 03:00:24 +0200] warning/PluginNotificationTask: Notification command for object 'MS2019P-EPDHCP!load' (PID: 4294967295, arguments: '') terminated with exit code 3, output: Error: Non-optional macro 'service.output' used in argument '-o' is missing.


[2021-07-08 03:00:24 +0200] information/Downtime: Removed downtime 'MS2019P-EPDHCP!load!6b853396-81b9-4c95-b64c-d5d1576a3996' from checkable 'MS2019P-EPDHCP!load' (Reason: expired at 2021-07-08 03:00:00 +0200).
[2021-07-08 03:00:24 +0200] information/Notification: Completed sending 'DowntimeEnd' notification 'MS2019P-EPDHCP!load!mail-icingaadmin' for checkable 'MS2019P-EPDHCP!load' and user 'icingaadmin' using command 'mail-service-notification'.

I’m fairly new to Icinga and don’t understand all the magic behind it entirely. It seems to me, that the Icinga service is running but not listening to the port 5665?

It would be very appreciated if someone can push me in the right direction.

Thank you very much and kind regards
Kevin

Hi @visablehamburg, may I ask you to run the following command on the agent and share the output here.

C:\Program Files\ICINGA2\sbin> .\icinga2.exe daemon -C

So you can also check which port the service is listening to.

Hi @visablehamburg ,
as @yhabteab wrote, have a look at the the output of

C:\Program Files\ICINGA2\sbin> .\icinga2.exe daemon -C

Are the certificates reachable and on the right place?
Did you try to configure the agent manually?
The Wizard won’t be work because he try to connect to the master!

Check all these things and it should work!

I successfully connected the master to the host.

That’s great, I think I haven’t really understood the way the agent works.
C:\Program Files\ICINGA2\sbin> .\icinga2.exe daemon -C gave me a valuable hint.

The name of the certificate was written is lower case letters, but Icinga was looking for a certificate name written in capitals. So I changed that and came a step further.

Next error message then was that the zones.conf on the agent wasn’t set, so I wrote a proper zones.conf.


object Endpoint "master.domain.de" {
        host = "master.domain.de"
        port = "5665"
}

object Zone "master" {
  endpoints = [ "master.domain.de" ]
}

object Endpoint "agent.domain.de" {
}

object Zone "agent.domain.de" {
   endpoints = [ "agent.domain.de" ]
   parent = "master"
}

Last thing was that the agent didn’t accept commands from the master, which can be configured in the api.conf file.

object ApiListener "api" {
  accept_config = true
  accept_commands = true

Thanks again for the great help guys!