Satellites zone loosing connection

we have an HA master → satellite-zones set up which has been fine for a long time. But recently one satellite zone reports that the satellites are not connected to each other - based on the default “load” and “icinga” service checks going into an UNKNOWN state every minute. they were fine for several months until adding some additional host objects config then doing a “reload” on the primary master.

output from load check:
Remote Icinga instance 'icinga-s6.mon.domain' is not connected to 'icinga-s7.mon.domain'

i’ve stopped/restarted each of them as well as both masters which did not resolve it. i then reran the node wizard on both satellites but still did not resolve the issue. the zones.conf file is the same as in all the other satellite zones which do not exhibit this behavior.

on satellites zones.conf

object Endpoint “icinga-master1.mon.domain” {
// host = “icinga-master1.mon.domain”
//do not actively connect to the satellite by leaving out the ‘host’ attribute

object Endpoint “icinga-master2.mon.domain” {
// host = “icinga-master2.mon.domain”
//do not actively connect to the satellite by leaving out the ‘host’ attribute

//Jinja2 syntax to add the corresponding satellite zone’s corresponding endpoint
object Endpoint “icinga-s6.mon.domain” {
host = “icinga-s6.mon.domain”
object Endpoint “icinga-s7.mon.domain” {
host = “icinga-s7.mon.domain”

object Zone “master” {
endpoints = [ “icinga-master1.mon.domain”, “icinga-master2.mon.domain” ]

//Jinja2 syntax to add the corresponding satellite zone and then the zone’s corresponding endpoint
object Zone “satellite-zone5” {
endpoints = [ “icinga-s6.mon.domain”, “icinga-s7.mon.domain” ]

parent = “master”

object Zone “global-templates” {
global = true

object Zone “director-global” {
global = true

yes i know the satellites can have one-way connection by commenting the other satellite but all the other satellite zones have the same set-up and have been fine for years.

the satellite host objects are defined in the zones.d/master/satellite-zone5.conf so get synched to second master without having to update the zones.conf on both whenever a new satellite zone is added. again this has been working completely fine for years.

how do i get these two satellites to stay connected to each other?

icinga2 r2.11.2-1
icingaweb 2.7.3

icinga2 r2.10.5-1

yes i know we should run the same version but this is a production environment and can’t afford downtime for upgrade testing - and it’s been operating just fine for a couple of years so please don’t recommend updating.

Hello @petew,
I hope you are well. I know you did not want this recommendation about upgrading but that is what you should probably do. Version 2.11.2 has a lot of bugs in it. Please view the change log here about the bug you are experiencing. I experienced this same problem when I upgraded from 2.10.5 to 2.11.2. I waited over an year before upgrading to 2.11.8. Version 2.11.8 is working great.

Maybe you could downgrade your Master to version 2.10.5 so all Master and Satellites are on the same version.

I hope this information helps. Good luck.

i’ll keep 2.11.8 as reference for our future plan on upgrading - thanks for the reference to the bug list too.