Zone connected, but not -- only pulling global templates

I’ve been at this for over a day and I think my eyeballs are starting to get burn-in from the terminal. I’m unsure of what I’m doing wrong, as the result seems to be inconsistent.

The above is the basic layout I have, but the root problem seems to be two things:

  • Cluster check shows “Icinga2 cluster OK - 0 endpoints connected” and
  • The satellite node at “SITE02-icinga2.localdomain” is receiving the “global-templates” as a satellite, but is not receiving the checks and config for it’s actual zone (SITE02)

Details:

  • Icinga2 Master
    • Version: icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)
    • OS: Debian 11
    • Enabled Features: Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb livestatus opentsdb perfdata statusdata syslog Enabled features: api checker ido-mysql mainlog notification
  • SITE02-icinga2.localdomain
    • Version: icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.5-1)
    • OS: CentOS Linux release 7.9.2009 (Core)
    • Enabled Features: Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus mainlog notification opentsdb perfdata statusdata syslog Enabled features: api checker

Icinga2 Master (icinga2.azure.localdomain) zones.conf

/*
 * Generated by Icinga 2 node setup commands
 * on 2022-09-29 13:23:34 +0000
 */

object Endpoint "icinga2.azure.localdomain" {
}

object Zone "master" {
        endpoints = [ "icinga2.azure.localdomain" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

//
// Zones added for other places
//

object Endpoint "site02-icinga.localdomain" {
        host = "1.2.3.4"
        port = "5665"
}

object Zone "SITE02" {
        endpoints = [ "site02-icinga.localdomain" ]
}

Icinga2 Satellite in SITE02 (Zone SITE02) (site02-icinga.localdomain) zones.conf

/*
 * Generated by Icinga 2 node setup commands
 * on 2022-09-29 14:01:18 +0000
 */

object Endpoint "icinga2.azure.localdomain" {
        host = "5.6.7.8" // Public IP of the master in Azure
        port = "5665"
}

object Zone "master" {
        endpoints = [ "icinga2.azure.localdomain" ]
}

object Endpoint "site02-icinga.localdomain" {
}

object Zone "SITE02" {
        endpoints = [ "site02-icinga.localdomain" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Additionally, here is some output of the directory structure within the satellite (site02-icinga2.localdomain) /var/lib/icinga2/api directory:

/var/lib/icinga2/api/
├── log
├── packages
│   └── _api
│       └── f6d35ade-ece1-4019-8799-e51f52a93cc7
│           ├── conf.d
│           └── zones.d
├── zones
│   └── global-templates
│       └── _etc
└── zones-stage
    └── global-templates
        └── _etc

You can also see the cluster check itself shows 0 endpoints connected:

image

Let me know if there’s additional information required.

I’ve resolved this on my own after hammering away at it.

The documentation states that the master/satellite/agent all has to know each other, and they do. The problem is that the master seemingly didn’t know it was the master for some of the zones and you have it instruct it as such.

Old lines in the master config:

object Zone "SITE02" {
        endpoints = [ "site02-icinga.localdomain" ]
}

…has been modified to:

object Zone "SITE02" {
        endpoints = [ "site02-icinga.localdomain" ]
        parent = "master"
}

This resolved the issue and the checks went from “Icinga 2 Cluster OK: 0 endpoints are connected” to having 2 endpoints connected as expected.