Satellite checks 'greyed out' and Client stuck at pending

Hi Guys,

I’m hoping you can help me. I am in a muddle after too much fiddling without proper understanding. I have two issues, 1) the checks for the satellite are ‘greyed out’ (which I think means they’re not running automatically?) and 2) Any client I attempt to connect (to the satellite zone) just seems to be stuck at ‘pending’. I have the following setup:

Master (“centos81”) >>> Satellite (“centos82”) >>> Client (“client1”)

icinga2 version (same on all 3 servers, default settings)

[root@centos81 ~]# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.4-1)

icinga2 feature list

[root@centos81 ~]# icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

[root@centos82 ~]# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog

[root@client1 icinga2]# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog

icinga2 daemon -C (there are some errors here, but I don’t think they’re critical at this point)

[root@centos81 ~]# icinga2 daemon -C
[2020-07-24 10:48:57 +0100] information/cli: Icinga application loader (version: 2.11.4-1)
[2020-07-24 10:48:57 +0100] information/cli: Loading configuration file(s).
[2020-07-24 10:48:57 +0100] information/ConfigItem: Committing config item(s).
[2020-07-24 10:48:57 +0100] information/ApiListener: My API identity: centos81
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/zones.d/global-templates/satellite.conf: 25:1-25:41) for type 'Dependency' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/email-notifications.conf: 1:0-1:44) for type 'Notification' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/email-notifications.conf: 11:1-11:48) for type 'Notification' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/zones.d/global-templates/downtimes.conf: 4:1-4:52) for type 'ScheduledDowntime' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/zones.d/global-templates/checks/all/ping.conf: 13:1-13:21) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'ntpd' (in /etc/icinga2/zones.d/global-templates/checks/linux/ntpd.conf: 3:1-3:20) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'automysqlbackup' (in /etc/icinga2/zones.d/global-templates/checks/services/automysqlbackup.conf: 3:1-3:31) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'mysql' (in /etc/icinga2/zones.d/global-templates/checks/services/mysql.conf: 3:1-3:21) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'solr_ping' (in /etc/icinga2/zones.d/global-templates/checks/services/solr.conf: 3:1-3:25) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'http-' (in /etc/icinga2/zones.d/global-templates/checks/services/vhosts.conf: 3:1-3:73) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'https-' (in /etc/icinga2/zones.d/global-templates/checks/services/vhosts.conf: 17:1-17:76) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'cert-' (in /etc/icinga2/zones.d/global-templates/checks/services/vhosts.conf: 33:1-33:75) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'vsftpd' (in /etc/icinga2/zones.d/global-templates/checks/services/vsftpd.conf: 3:1-3:22) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] warning/ApplyRule: Apply rule 'vsftpd-procs' (in /etc/icinga2/zones.d/global-templates/checks/services/vsftpd.conf: 22:1-22:28) for type 'Service' does not match anywhere!
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 2 HostGroups.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 EventCommand.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 3 Hosts.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 5 Zones.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 3 Endpoints.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 236 CheckCommands.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 5 TimePeriods.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 1 User.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 35 Services.
[2020-07-24 10:48:58 +0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2020-07-24 10:48:58 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2020-07-24 10:48:58 +0100] information/cli: Finished validating the configuration file(s).

[root@centos82 ~]# icinga2 daemon -C
[2020-07-24 10:49:55 +0100] information/cli: Icinga application loader (version: 2.11.4-1)
[2020-07-24 10:49:55 +0100] information/cli: Loading configuration file(s).
[2020-07-24 10:49:55 +0100] information/ConfigItem: Committing config item(s).
[2020-07-24 10:49:55 +0100] information/ApiListener: My API identity: centos82
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'satellite-host' (in /var/lib/icinga2/api/zones/global-templates/_etc/satellite.conf: 25:1-25:41) for type 'Dependency' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /var/lib/icinga2/api/zones/global-templates/_etc/email-notifications.conf: 1:0-1:44) for type 'Notification' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /var/lib/icinga2/api/zones/global-templates/_etc/email-notifications.conf: 11:1-11:48) for type 'Notification' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'backup-downtime' (in /var/lib/icinga2/api/zones/global-templates/_etc/downtimes.conf: 4:1-4:52) for type 'ScheduledDowntime' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'cluster-zone' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/cluster-health.conf: 2:1-2:28) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'master-cluster-connections' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/cluster-health.conf: 15:1-15:42) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'icinga' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/icinga.conf: 3:1-3:22) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'ping4' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/ping.conf: 3:1-3:21) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'ping6' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/ping.conf: 13:1-13:21) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'procs' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/procs.conf: 3:1-3:21) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'users' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/all/users.conf: 3:1-3:21) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'chronyd' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/chronyd.conf: 3:1-3:23) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule '' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/disk.conf: 3:1-3:53) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'load' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/load.conf: 3:1-3:20) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'mail queue' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/mailq.conf: 3:1-3:26) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'ntpd' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/ntpd.conf: 3:1-3:20) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'ssh' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/ssh.conf: 3:1-3:19) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'swap' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/swap.conf: 3:1-3:20) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'time' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/linux/time.conf: 3:1-3:20) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'automysqlbackup' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/automysqlbackup.conf: 3:1-3:31) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'mysql' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/mysql.conf: 3:1-3:21) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'solr_ping' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/solr.conf: 3:1-3:25) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'http-' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/vhosts.conf: 3:1-3:73) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'https-' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/vhosts.conf: 17:1-17:76) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'cert-' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/vhosts.conf: 33:1-33:75) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'vsftpd' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/vsftpd.conf: 3:1-3:22) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] warning/ApplyRule: Apply rule 'vsftpd-procs' (in /var/lib/icinga2/api/zones/global-templates/_etc/checks/services/vsftpd.conf: 22:1-22:28) for type 'Service' does not match anywhere!
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 2 HostGroups.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 EventCommand.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 4 Zones.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 236 CheckCommands.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 5 TimePeriods.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 1 User.
[2020-07-24 10:49:55 +0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2020-07-24 10:49:55 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2020-07-24 10:49:55 +0100] information/cli: Finished validating the configuration file(s).

[root@client1 icinga2]# icinga2 daemon -C
[2020-07-24 10:50:18 +0100] information/cli: Icinga application loader (version: 2.11.4-1)
[2020-07-24 10:50:18 +0100] information/cli: Loading configuration file(s).
[2020-07-24 10:50:18 +0100] information/ConfigItem: Committing config item(s).
[2020-07-24 10:50:18 +0100] information/ApiListener: My API identity: client1
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 4 Zones.
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2020-07-24 10:50:18 +0100] information/ConfigItem: Instantiated 235 CheckCommands.
[2020-07-24 10:50:18 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2020-07-24 10:50:18 +0100] information/cli: Finished validating the configuration file(s).

zones.conf files

[root@centos81 icinga2]# cat /etc/icinga2/zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-22 09:55:11 +0100
 */

object Endpoint "centos81" {
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

[root@centos82 ~]# cat /etc/icinga2/zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-24 09:46:54 +0100
 */

object Endpoint "centos81" {
        host = "192.168.1.203"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Endpoint "centos82" {
}

object Zone "centos82" {
        endpoints = [ "centos82" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

[root@client1 icinga2]# cat /etc/icinga2/zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-23 21:19:56 +0100
 */

object Endpoint "centos82" {
}

object Zone "satellite" {
        endpoints = [ "centos82" ]
}

object Endpoint "client1" {
}

object Zone "client1" {
        endpoints = [ "client1" ]
        parent = "satellite"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Endpoint lists

[root@centos81 icinga2]# icinga2 object list --type endpoint
Object 'client1' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.d/satellite/client1.conf', lines 33:1-33:25
  * __name = "client1"
  * host = ""
  * log_duration = 86400
  * name = "client1"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 33
    * last_column = 25
    * last_line = 33
    * path = "/etc/icinga2/zones.d/satellite/client1.conf"
  * templates = [ "client1" ]
    % = modified in '/etc/icinga2/zones.d/satellite/client1.conf', lines 33:1-33:25
  * type = "Endpoint"
  * zone = "satellite"

Object 'centos81' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * __name = "centos81"
  * host = ""
  * log_duration = 86400
  * name = "centos81"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 26
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos81" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * type = "Endpoint"
  * zone = ""

Object 'centos82' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.d/satellite/centos82.conf', lines 33:1-33:26
  * __name = "centos82"
  * host = ""
  * log_duration = 86400
  * name = "centos82"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 33
    * last_column = 26
    * last_line = 33
    * path = "/etc/icinga2/zones.d/satellite/centos82.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.d/satellite/centos82.conf', lines 33:1-33:26
  * type = "Endpoint"
  * zone = "satellite"

[root@centos82 ~]# icinga2 object list --type endpoint
Object 'centos82' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 15:1-15:26
  * __name = "centos82"
  * host = ""
  * log_duration = 86400
  * name = "centos82"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 26
    * last_line = 15
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 15:1-15:26
  * type = "Endpoint"
  * zone = ""

Object 'centos81' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * __name = "centos81"
  * host = "192.168.1.203"
    % = modified in '/etc/icinga2/zones.conf', lines 7:2-7:23
  * log_duration = 86400
  * name = "centos81"
  * package = "_etc"
  * port = "5665"
    % = modified in '/etc/icinga2/zones.conf', lines 8:2-8:14
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 26
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos81" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * type = "Endpoint"
  * zone = ""

[root@client1 icinga2]# icinga2 object list --type endpoint
Object 'client1' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 13:1-13:25
  * __name = "client1"
  * host = ""
  * log_duration = 86400
  * name = "client1"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 13
    * last_column = 25
    * last_line = 13
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "client1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 13:1-13:25
  * type = "Endpoint"
  * zone = ""

Object 'centos82' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * __name = "centos82"
  * host = ""
  * log_duration = 86400
  * name = "centos82"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 26
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * type = "Endpoint"
  * zone = ""

Zones list

[root@centos81 icinga2]# icinga2 object list --type Zone
Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 13:1-13:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 14:2-14:14
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 13
    * last_column = 30
    * last_line = 13
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 13:1-13:30
  * type = "Zone"
  * zone = ""

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 17:1-17:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 18:2-18:14
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 17
    * last_column = 29
    * last_line = 17
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 17:1-17:29
  * type = "Zone"
  * zone = ""

Object 'master' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 9:1-9:20
  * __name = "master"
  * endpoints = [ "centos81" ]
    % = modified in '/etc/icinga2/zones.conf', lines 10:2-10:27
  * global = false
  * name = "master"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 20
    * last_line = 9
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "master" ]
    % = modified in '/etc/icinga2/zones.conf', lines 9:1-9:20
  * type = "Zone"
  * zone = ""

Object 'satellite' of type 'Zone':
  % declared in '/etc/icinga2/satellite.d/satellite.conf', lines 1:0-1:22
  * __name = "satellite"
  * endpoints = [ "centos82" ]
    % = modified in '/etc/icinga2/satellite.d/satellite.conf', lines 4:3-4:28
  * global = false
  * name = "satellite"
  * package = "_etc"
  * parent = "master"
    % = modified in '/etc/icinga2/satellite.d/satellite.conf', lines 3:3-3:19
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 22
    * last_line = 1
    * path = "/etc/icinga2/satellite.d/satellite.conf"
  * templates = [ "satellite" ]
    % = modified in '/etc/icinga2/satellite.d/satellite.conf', lines 1:0-1:22
  * type = "Zone"
  * zone = ""

Object 'client1' of type 'Zone':
  % declared in '/etc/icinga2/zones.d/satellite/client1.conf', lines 36:1-36:21
  * __name = "client1"
  * endpoints = [ "client1" ]
    % = modified in '/etc/icinga2/zones.d/satellite/client1.conf', lines 37:9-37:33
  * global = false
  * name = "client1"
  * package = "_etc"
  * parent = "satellite"
    % = modified in '/etc/icinga2/zones.d/satellite/client1.conf', lines 38:9-38:28
  * source_location
    * first_column = 1
    * first_line = 36
    * last_column = 21
    * last_line = 36
    * path = "/etc/icinga2/zones.d/satellite/client1.conf"
  * templates = [ "client1" ]
    % = modified in '/etc/icinga2/zones.d/satellite/client1.conf', lines 36:1-36:21
  * type = "Zone"
  * zone = "satellite"

[root@centos82 ~]#  icinga2 object list --type Zone
Object 'centos82' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 18:1-18:22
  * __name = "centos82"
  * endpoints = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 19:2-19:27
  * global = false
  * name = "centos82"
  * package = "_etc"
  * parent = "master"
    % = modified in '/etc/icinga2/zones.conf', lines 20:2-20:18
  * source_location
    * first_column = 1
    * first_line = 18
    * last_column = 22
    * last_line = 18
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 18:1-18:22
  * type = "Zone"
  * zone = ""

Object 'master' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 11:1-11:20
  * __name = "master"
  * endpoints = [ "centos81" ]
    % = modified in '/etc/icinga2/zones.conf', lines 12:2-12:27
  * global = false
  * name = "master"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 20
    * last_line = 11
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "master" ]
    % = modified in '/etc/icinga2/zones.conf', lines 11:1-11:20
  * type = "Zone"
  * zone = ""

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 27:1-27:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 28:2-28:14
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 27
    * last_column = 29
    * last_line = 27
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 27:1-27:29
  * type = "Zone"
  * zone = ""

Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 23:1-23:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 24:2-24:14
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 23
    * last_column = 30
    * last_line = 23
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 23:1-23:30
  * type = "Zone"
  * zone = ""

[root@client1 icinga2]#  icinga2 object list --type Zone
Object 'satellite' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 9:1-9:23
  * __name = "satellite"
  * endpoints = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 10:2-10:27
  * global = false
  * name = "satellite"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 23
    * last_line = 9
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "satellite" ]
    % = modified in '/etc/icinga2/zones.conf', lines 9:1-9:23
  * type = "Zone"
  * zone = ""

Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 21:1-21:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 22:2-22:14
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 21
    * last_column = 30
    * last_line = 21
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 21:1-21:30
  * type = "Zone"
  * zone = ""

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 25:1-25:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 26:2-26:14
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 25
    * last_column = 29
    * last_line = 25
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 25:1-25:29
  * type = "Zone"
  * zone = ""

Object 'client1' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 16:1-16:21
  * __name = "client1"
  * endpoints = [ "client1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 17:2-17:26
  * global = false
  * name = "client1"
  * package = "_etc"
  * parent = "satellite"
    % = modified in '/etc/icinga2/zones.conf', lines 18:2-18:21
  * source_location
    * first_column = 1
    * first_line = 16
    * last_column = 21
    * last_line = 16
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "client1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 16:1-16:21
  * type = "Zone"
  * zone = ""

I’d dearly love it if someone can nudge me in the right direction. Thanks for your time.

With V2.11 zone and endpoint objects needs to be defined in zones.conf only.

1 Like

Thanks @rsx for taking the time to reply. I have actioned the following on your advice:

  1. Deleted satellite.conf from /etc/icinga2/satellite.d

  2. Added satellite Zone definition to /etc/icinga2/zones.conf

  3. Removed Zone/Endpoint definitions in /etc/icinga2/zones.d/satellite/client1.conf

  4. Added Zones/Endpoints to /etc/icinga2/zones.conf

  5. Check configuration with icinga2 daemon -C

  6. Restarted icinga2 on Master/Satellite/Agent.

My Master zones.conf now looks like this:

[root@centos81 icinga2]# cat /etc/icinga2/zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-22 09:55:11 +0100
 */

object Endpoint "centos81" {
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

object Zone "satellite" {
  parent = "master"
  endpoints = [ "centos82", "client1" ]
}

object Endpoint "centos82" {
}

object Endpoint "client1" {
}

Master Endpoint object list :

[root@centos81 icinga2]# icinga2 object list --type endpoint
Object 'client1' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 29:1-29:25
  * __name = "client1"
  * host = ""
  * log_duration = 86400
  * name = "client1"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 29
    * last_column = 25
    * last_line = 29
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "client1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 29:1-29:25
  * type = "Endpoint"
  * zone = ""

Object 'centos82' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 26:1-26:26
  * __name = "centos82"
  * host = ""
  * log_duration = 86400
  * name = "centos82"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 26
    * last_column = 26
    * last_line = 26
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 26:1-26:26
  * type = "Endpoint"
  * zone = ""

Object 'centos81' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * __name = "centos81"
  * host = ""
  * log_duration = 86400
  * name = "centos81"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 26
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos81" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * type = "Endpoint"
  * zone = ""

Satellite Endpoint object list (should there be a “client1” definition in here?)

[root@centos82 ~]#  icinga2 object list --type Endpoint
Object 'centos82' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 15:1-15:26
  * __name = "centos82"
  * host = ""
  * log_duration = 86400
  * name = "centos82"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 26
    * last_line = 15
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 15:1-15:26
  * type = "Endpoint"
  * zone = ""

Object 'centos81' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * __name = "centos81"
  * host = "192.168.1.203"
    % = modified in '/etc/icinga2/zones.conf', lines 7:2-7:23
  * log_duration = 86400
  * name = "centos81"
  * package = "_etc"
  * port = "5665"
    % = modified in '/etc/icinga2/zones.conf', lines 8:2-8:14
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 26
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos81" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * type = "Endpoint"
  * zone = ""

Client Endpoint object list (looks ok?):

[root@client1 icinga2]# icinga2 object list --type Endpoint
Object 'client1' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 13:1-13:25
  * __name = "client1"
  * host = ""
  * log_duration = 86400
  * name = "client1"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 13
    * last_column = 25
    * last_line = 13
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "client1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 13:1-13:25
  * type = "Endpoint"
  * zone = ""

Object 'centos82' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * __name = "centos82"
  * host = ""
  * log_duration = 86400
  * name = "centos82"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 26
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "centos82" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:26
  * type = "Endpoint"
  * zone = ""

Although this has certainly simplified the configuration (and my understanding) the two initial problems still stand, but also now the cluster connection check is failing.

Any further advice you can offer would be most welcome. :slightly_smiling_face:

This is wrong. client1 needs to have its own zone and endpoint object (having centos82 as its parent).

Every zone and endpoint objects needs to be in master’s zones.conf. Every zone and endpoint objects for agents for waich a satellite is parent needs to be in satellite’s zones.conf incl. satellites’s and parent’s zone and endpoint objects.

1 Like

Hi.

A short answer due to a lack of time.

here is an example of the zones.conf of master/satellite and agent

Additionally here (partially) an example zones.conf of an agent in a satellite zone:

...
object Endpoint NodeName {
}

object Zone ZoneName {
        endpoints = [ NodeName ]
        parent = "<your-satellite-zone>"
}
...

In your posted output, the agents seem to not be in a zone.

Hope this helps.

Greetings.

1 Like

Thanks @rsx @homerjay. I’m feeling more confident with this, but, if I may, there appears to be some contradictory information being offered:

  • Every zone and endpoint objects needs to be in master’s zones.conf.

But in homerjay’s example Master zones.conf, there is no client Zone/Endpoint definition.

  • Every zone and endpoint object for agents for which a satellite is parent needs to be in satellite’s zones.conf incl. satellites’s and parent’s zone and endpoint objects.

But in homerjay’s example Satellite zones.conf, there is no client Zone/Endpoint definition.

If I may, I’d like to start ‘bottom-up’. Here is my Clients zones.conf, which I believe is correct?

[root@client1 icinga2]# cat /etc/icinga2/zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-23 21:19:56 +0100
 */

object Endpoint "centos82" {
        host = "centos82"
        port = "5665"
}

object Zone "satellite" {
        endpoints = [ "centos82" ]
}

object Endpoint "client1" {
}

object Zone "client1" {
        endpoints = [ "client1" ]
        parent = "satellite"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Here is my Satellite’s zones.conf:

/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-24 09:46:54 +0100
 */

object Endpoint "centos81" {
        host = "192.168.1.203"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Endpoint "centos82" {
}

object Zone "centos82" {
        endpoints = [ "centos82" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Here is my Master zones.conf:

[root@centos81 icinga2]# clear ; cat /etc/icinga2/zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-07-22 09:55:11 +0100
 */

object Endpoint "centos81" {
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Endpoint "centos82" {
        host = "centos82"
}

object Zone "satellite" {
  endpoints = [ "centos82" ]
  parent = "master"
}

object Endpoint "client1" {
}

object Zone "client1" {
  endpoints = [ "client1" ]
  parent = "satellite"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

All of these ‘pass’ the configuration checks. But I still have the same errors in icinga2web (namely, greyed-out checks on the satellite, and pending status of the client.)

Hi again.

I am really glad you asked. You are absolutely right, my linked post was in another context and doesn’t contain all information.

So here, some further information.

The master:

  • “knows them all”.
  • all endpoints and zones are known to the master.
Side note

(Side note: I have to admit that I use a way not placing the agent-nodes
in the zones.conf of the master, but in files under the zone-directory.
I don’t want to spread confusion, but this is also possible.
Files under

/etc/icinga2/zones.d/<zonename>/

containing “Endpoint” and “Zone”)

The satellite:

  • knows the master
  • know itself and the zone it is the satellite for
  • has no endpoints of the agents (!) in the zones.conf

The agent in the satellite zone:

  • knows the satellite
  • knows itself and his own, unique zone

Here real examples of the zones.conf with dummy-values.
For better readability, without the global zones:

In this example:

master : zone of the master
satellite-zone: zone of the satellite
hostin.satellite.zone: an example host in the satellite-zone

Master
object Endpoint "master.your.domain"  {
}

object Endpoint "satellite.your.domain" {
        host = "satellite.your.domain"
}

object Zone "master"  {
         endpoints = [ "master.your.domain", ]
}

object Zone "satellite-zone" {
        endpoints = [ "satellite.your.domain" ]
        parent = "master"
}

object Endpoint "hostin.satellite.zone"  {
}

object Zone "hostin.satellite.zone"  {
  endpoints = [ "hostin.satellite.zone", ]
  parent = "satellite-zone"
}
Satellite
// Satellite (complete zones.conf - just without global zones).
// No informations about the agents of the connected zone

object Endpoint "master.your.domain" {
        host = "master.your.domain"
        port = "5665"
}

object Endpoint "satellite.your.domain" {
}

object Zone "master" {
        endpoints = [ "master.your.domain" ]
}

object Zone "satellite-zone" {
        endpoints = [ "satellite.your.domain" ]
        parent = "master"
}
Agent in satellite zone
object Endpoint "satellite.your.domain" {
        host = "satellite.your.domain";
        port = "5665";
}

object Zone "satellite-zone" {
        endpoints = [ "satellite.your.domain" ]
}

object Endpoint "hostin.satellite.zone" {
}

object Zone "hostin.satellite.zone" {
        endpoints = [ "hostin.satellite.zone" ]
        parent = "satellite-zone"
}

Maybe this clears it up a little bit.

Greetings.

@homerjay Thanks again. I’d like to say it clears it up, but it doesn’t. I can’t see any difference in the examples you’ve kindly provided versus the zones.conf files for each of my [Master|Satellite|Client] servers.

I actually did go ahead and add the Zone and Endpoint definitions for “client1” into the Satellites zones.conf file and “client1” started working. It was no longer pending and was running checks. The Check Source is even showing as “centos82” on relevant checks? I should also add that the ‘greyed out’ checks on the Satellite are no longer grey(!).

Unfortunately, that has broken the cluster checks and now I am more confused than ever. :cry:

icinga2web3

Did you restart icinga2 on master, satellite and agent after the changes?

If this doesn’t help:
Would you please show us the zones.conf of master, satellite and the agent in the satellite-zone?

Greetings.

1 Like

@homerjay I’d be delighted to, sir :slightly_smiling_face:

Master (“centos81”) /etc/icinga2/zones.conf (globals removed):

[root@centos81 icinga2]# cat /etc/icinga2/zones.conf

object Endpoint "centos81" {
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Endpoint "centos82" {
        host = "centos82"
}

object Zone "satellite" {
  endpoints = [ "centos82" ]
  parent = "master"
}

object Endpoint "client1" {
}

object Zone "client1" {
  endpoints = [ "client1" ]
  parent = "satellite"
}

Satellite (“centos82”) /etc/icinga2/zones.conf (globals removed):

object Endpoint "centos81" {
        host = "centos81"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "centos81" ]
}

object Endpoint "centos82" {
}

object Zone "satellite" {
        endpoints = [ "centos82" ]
        parent = "master"
}

object Endpoint "client1" {
}

object Zone "client1" {
  endpoints = [ "client1" ]
  parent = "satellite"
}

Client (“client1”) /etc/icinga2/zones.conf (globals removed):

object Endpoint "centos82" {
        host = "centos82"
        port = "5665"
}

object Zone "satellite" {
        endpoints = [ "centos82" ]
}

object Endpoint "client1" {
}

object Zone "client1" {
        endpoints = [ "client1" ]
        parent = "satellite"
}

Yes sir.

Thank you.

I can see no errors in your posted zones.conf.
Except that the client1 part you added to the satellite’s zones.conf should not be there.

Maybe anyone else has an idea what is going wrong?

There are some ideas which might help:

  • You’re addressing by name (in your examples). Is the resolution to the right IPs working?
  • Are the endpoints equal to the setup from the “icinga2 node wizard”?
    E.g. the Host with “client1” has a corresponding configuration on the master
  • Did you notice problems while running the “icinga2 node wizard”?
  • What do the logs on the corresponding hosts show?
  • On the satellite: Are there hosts visible at
/var/lib/icinga2/api/zones/satellite/

?

Sorry that my posts have not been helpful.
Unfortunately i don’t have time for more answers today.

Greetings.

1 Like

Hi @homerjay. Many thanks for the assistance offered. I have largely managed to get things working now by diligently re-reading both your helpful posts and the documentation.

I now concur that the satellite zones.conf should have no client configuration in it.

I added the client Endpoint and Zone definitions to client1.conf in /etc/icinga2/zones.d/satellite on the master and restarted everything.

I now have just a single issue, namely a failing “cluster-zone-client1” check.

Can I ask how relevant this ‘failing’ check is for an agent reporting to a satellite? If the agent is in the satellite zone I am not sure how this check would pass?

1 Like

Scratch that. I have fixed that issue too. Incorrect vars.icinga_zone configured in /etc/icinga2/zones.d/satellite/client1.conf! :grinning:

Hi again.

Great! I’m very glad this worked for you.
This kept me busy for some hours, trying to find the error with your config in a spinned up testing-environment.

I was about to post very much content, but that you solved it yourself is much better.

Thanks for your informations.

Greetings.

1 Like