Remote Icinga Instance (Satellite) Not Connected to Masters

Icinga 2
, ,
1

Hello Icinga community,

At my job our Icinga setup has two masters and two satellites. Lately we have been unable to receive service checks from Satellite 1 because the endpoint is not connected to Master 1 and Master 2, as shown in the screenshot below.

Sat1 status
Sat1 status828×564 26.3 KB

I’ve tried 2 different configurations: 1) Masters initiate connection to Satellite 1 2) Satellite 1 initiates connection to Masters. With both configurations, Satellite 1 has a connection with the Masters for a day or two before it fails again. All certificates for these servers are valid and inbound connections on port 5665 are allowed on all servers. I’ve consulted the troubleshooting documentation, but nothing in there has resolved the issue so far.

Below are the relevant configuration details from the zones.conf files on all three servers (with domains and IPs obfuscated for security).

zones.conf file for Master 1 and Master 2 are the same

object Endpoint "icinga2master1.domain.org" {
  host = "xx.xx.xx.xx"
  port = "5665"
}

object Endpoint "icinga2master2.domain.org" {
  host = "xx.xx.xx.xx"
  port = "5665"
}

object Endpoint "icinga2sat1.domain.org" {
 host = "xx.xx.xx.xx"
 port = "5665"
}

zones.conf file on Satellite 1:


object Endpoint "icinga2master1.domain.org" {
}

object Endpoint "icinga2master2.healthnet.phidc.org" {
}

object Endpoint "icinga2sat1.uhtasi.org" {
}

Below is Satellite 1’s host definition (hosts.conf) from Master 1.

object Host "icinga2sat1.domain.org" {
  import "generic-host"
  zone = "master"
  address = "xx.xx.xx.xx"
  vars.linux = true
  vars.icinga = true
  vars.client_endpoint = "icinga2sat1.domain.org"
  groups = [ "ApplicationServers"]
}

Logs (/var/log/icinga2/icinga2.log) on both Masters show the following:

[2022-07-19 09:52:57 -1000] information/ApiListener: Reconnecting to endpoint 'icinga2sat1.domain.org' via host 'xx.xx.xx.xx' and port '5665'
[2022-07-19 09:52:57 -1000] critical/ApiListener: Cannot connect to host 'xx.xx.xx.xx' on port '5665': Connection refused

Features enabled on Master 1:

api checker command debuglog ido-mysql influxdb mainlog notification

Features enabled on Master 2

api checker command ido-mysql mainlog notification

Features enabled on Satellite 1:

api checker debuglog gelf ido-mysql influxdb mainlog
  • Version used: 2.13.2
  • Operating System and version: CentOS 7
  • Icinga Web 2 version: 2.9.5
  • Config validation: running “icinga2 daemon -C” on all servers shows configurations are valid

Any insight is appreciated, thanks!