Distributed monitoring || Host checks are "pending"/"overdue"

Hello!

I’m starting my first post here, since I don’t know where else to look for my error.

I have an existing master that already monitors 5 hosts, which are connected directly to the master via agents. The setup now needs to be expanded to include a satellite and about 20 underlying hosts/agents. The satellite and the hosts are running on a separate network, and the master is connected to it via VPN. To test it out, I first connected 3 hosts directly to the master and set up a few test checks… everything worked without any problems. Afterward, I ventured into distributed monitoring and implemented it according to the instructions at Distributed Monitoring - Icinga 2 . I then loaded the configuration into the Director using the “Kickstart Wizard.”

The thing is, I can monitor the satellite as a host without any problems—but only as long as I don’t enable the “Cluster Zone” option in Director.
The even bigger problem is that I can’t get the actual hosts to be monitored. Here, the “hostalive” command already fails with an “OVERDUE” error.

So far, I can rule out a firewall issue. All ICMP and TCP (5665) packets are being allowed through the firewall.
I’ve also completely reinstalled the satellite and the test host… but unfortunately, that didn’t help either.

It seems to me as if the satellite is simply ignoring the checks it’s supposed to run for the test host.
Can someone please give me a tip on where else I could look for the error?

I replaced the hostnames:

Master -> master.anywhere.com
Satellite -> satellite.cloud.com
Test-Host -> test.cloud.com

Director Screenshots:

Satellite:

Test-Host:

Quick note: I’ve also played around a lot with the settings for “Icinga2 Agent”, “Establish connection”, “Accepts Config” and “Cluster Zone” but unfortunately, that didn’t work either.

Master → zones.conf

object Endpoint "master.anywhere.com" {
}
 
object Zone "main" {
  endpoints = [ "master.anywhere.com", ]
}

object Endpoint "satellite.cloud.com" {
    host = "satellite.cloud.com"
}

 
object Zone "cloud" {
  endpoints = [ "satellite.cloud.com", ]
  parent = "main"
}

object Zone "global-templates" {
    global = true
}

object Zone "director-global" {
  global = true
}

Master → /etc/icinga2/zones.d/cloud/test.cloud.com.conf

object Endpoint "test.cloud.com" {
  host = "test.cloud.com" // The satellite actively tries to connect to the agent
  log_duration = 0 // Disable the replay log for command endpoint agents
}

object Zone "test.cloud.com" {
  endpoints = [ "test.cloud.com" ]

  parent = "cloud"
}

Satellite → zones.conf

object Endpoint "master.anywhere.com" {
}

object Zone "main" {
	endpoints = [ "master.anywhere.com" ]
}

object Endpoint "satellite.cloud.com" {
}

object Zone "cloud" {
	endpoints = [ "satellite.cloud.com" ]
	parent = "main"
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

Test-Host → zones.conf


object Endpoint "satellite.cloud.com" {
}

object Zone "cloud" {
	endpoints = [ "satellite.cloud.com" ]
}

object Endpoint "test.cloud.com" {
}

object Zone "test.cloud.com" {
	endpoints = [ "test.cloud.com" ]
	parent = "cloud"
}

object Zone "global-templates" {
	global = true
}

object Zone "director-global" {
	global = true
}

Diretory trees of zones and zones-stage in /var/lib/icinga2/api/

Master:

/var/lib/icinga2/api/zones
/var/lib/icinga2/api/zones/director-global
/var/lib/icinga2/api/zones/director-global/.checksums
/var/lib/icinga2/api/zones/director-global/director
/var/lib/icinga2/api/zones/director-global/director/command_templates.conf
/var/lib/icinga2/api/zones/director-global/director/commands.conf
/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf
/var/lib/icinga2/api/zones/director-global/director/servicesets.conf
/var/lib/icinga2/api/zones/director-global/director/user_templates.conf
/var/lib/icinga2/api/zones/director-global/director/timeperiods.conf
/var/lib/icinga2/api/zones/director-global/director/hostgroups.conf
/var/lib/icinga2/api/zones/director-global/director/service_templates.conf
/var/lib/icinga2/api/zones/director-global/.authoritative
/var/lib/icinga2/api/zones/director-global/.timestamp
/var/lib/icinga2/api/zones/test.cloud.com
/var/lib/icinga2/api/zones/test.cloud.com/.checksums
/var/lib/icinga2/api/zones/test.cloud.com/director
/var/lib/icinga2/api/zones/test.cloud.com/director/host_templates.conf
/var/lib/icinga2/api/zones/test.cloud.com/.authoritative
/var/lib/icinga2/api/zones/test.cloud.com/.timestamp
/var/lib/icinga2/api/zones/cloud
/var/lib/icinga2/api/zones/cloud/.checksums
/var/lib/icinga2/api/zones/cloud/_etc
/var/lib/icinga2/api/zones/cloud/_etc/test.cloud.com.conf
/var/lib/icinga2/api/zones/cloud/director
/var/lib/icinga2/api/zones/cloud/director/services.conf
/var/lib/icinga2/api/zones/cloud/director/hosts.conf
/var/lib/icinga2/api/zones/cloud/director/host_templates.conf
/var/lib/icinga2/api/zones/cloud/.authoritative
/var/lib/icinga2/api/zones/cloud/.timestamp
/var/lib/icinga2/api/zones/satellite.cloud.com
/var/lib/icinga2/api/zones/satellite.cloud.com/.checksums
/var/lib/icinga2/api/zones/satellite.cloud.com/director
/var/lib/icinga2/api/zones/satellite.cloud.com/director/host_templates.conf
/var/lib/icinga2/api/zones/satellite.cloud.com/.authoritative
/var/lib/icinga2/api/zones/satellite.cloud.com/.timestamp
/var/lib/icinga2/api/zones/main
/var/lib/icinga2/api/zones/main/.checksums
/var/lib/icinga2/api/zones/main/_etc
/var/lib/icinga2/api/zones/main/_etc/commands.conf
/var/lib/icinga2/api/zones/main/_etc/apiusers.conf
/var/lib/icinga2/api/zones/main/_etc/gotify.conf
/var/lib/icinga2/api/zones/main/director
/var/lib/icinga2/api/zones/main/director/notification_apply.conf
/var/lib/icinga2/api/zones/main/director/services.conf
/var/lib/icinga2/api/zones/main/director/agent_zones.conf
/var/lib/icinga2/api/zones/main/director/users.conf
/var/lib/icinga2/api/zones/main/director/hosts.conf
/var/lib/icinga2/api/zones/main/director/agent_endpoints.conf
/var/lib/icinga2/api/zones/main/director/notification_templates.conf
/var/lib/icinga2/api/zones/main/director/host_templates.conf
/var/lib/icinga2/api/zones/main/director/usergroups.conf
/var/lib/icinga2/api/zones/main/.authoritative
/var/lib/icinga2/api/zones/main/.timestamp

Satellite:

/var/lib/icinga2/api/zones-stage-startup.log
/var/lib/icinga2/api/zones
/var/lib/icinga2/api/zones/director-global
/var/lib/icinga2/api/zones/director-global/.checksums
/var/lib/icinga2/api/zones/director-global/director
/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf
/var/lib/icinga2/api/zones/director-global/director/command_templates.conf
/var/lib/icinga2/api/zones/director-global/director/servicesets.conf
/var/lib/icinga2/api/zones/director-global/director/user_templates.conf
/var/lib/icinga2/api/zones/director-global/director/commands.conf
/var/lib/icinga2/api/zones/director-global/director/hostgroups.conf
/var/lib/icinga2/api/zones/director-global/director/timeperiods.conf
/var/lib/icinga2/api/zones/director-global/director/service_templates.conf
/var/lib/icinga2/api/zones/director-global/.timestamp
/var/lib/icinga2/api/zones/test.cloud.com
/var/lib/icinga2/api/zones/test.cloud.com/.checksums
/var/lib/icinga2/api/zones/test.cloud.com/director
/var/lib/icinga2/api/zones/test.cloud.com/director/host_templates.conf
/var/lib/icinga2/api/zones/test.cloud.com/.timestamp
/var/lib/icinga2/api/zones/cloud
/var/lib/icinga2/api/zones/cloud/.checksums
/var/lib/icinga2/api/zones/cloud/_etc
/var/lib/icinga2/api/zones/cloud/_etc/test.cloud.com.conf
/var/lib/icinga2/api/zones/cloud/director
/var/lib/icinga2/api/zones/cloud/director/hosts.conf
/var/lib/icinga2/api/zones/cloud/director/services.conf
/var/lib/icinga2/api/zones/cloud/director/host_templates.conf
/var/lib/icinga2/api/zones/cloud/.timestamp
/var/lib/icinga2/api/zones-stage
/var/lib/icinga2/api/zones-stage/director-global
/var/lib/icinga2/api/zones-stage/director-global/.checksums
/var/lib/icinga2/api/zones-stage/director-global/director
/var/lib/icinga2/api/zones-stage/director-global/director/001-director-basics.conf
/var/lib/icinga2/api/zones-stage/director-global/director/command_templates.conf
/var/lib/icinga2/api/zones-stage/director-global/director/servicesets.conf
/var/lib/icinga2/api/zones-stage/director-global/director/user_templates.conf
/var/lib/icinga2/api/zones-stage/director-global/director/commands.conf
/var/lib/icinga2/api/zones-stage/director-global/director/hostgroups.conf
/var/lib/icinga2/api/zones-stage/director-global/director/timeperiods.conf
/var/lib/icinga2/api/zones-stage/director-global/director/service_templates.conf
/var/lib/icinga2/api/zones-stage/director-global/.timestamp
/var/lib/icinga2/api/zones-stage/test.cloud.com
/var/lib/icinga2/api/zones-stage/test.cloud.com/.checksums
/var/lib/icinga2/api/zones-stage/test.cloud.com/director
/var/lib/icinga2/api/zones-stage/test.cloud.com/director/host_templates.conf
/var/lib/icinga2/api/zones-stage/test.cloud.com/.timestamp
/var/lib/icinga2/api/zones-stage/cloud
/var/lib/icinga2/api/zones-stage/cloud/.checksums
/var/lib/icinga2/api/zones-stage/cloud/_etc
/var/lib/icinga2/api/zones-stage/cloud/_etc/test.cloud.com.conf
/var/lib/icinga2/api/zones-stage/cloud/director
/var/lib/icinga2/api/zones-stage/cloud/director/hosts.conf
/var/lib/icinga2/api/zones-stage/cloud/director/services.conf
/var/lib/icinga2/api/zones-stage/cloud/director/host_templates.conf
/var/lib/icinga2/api/zones-stage/cloud/.timestamp
/var/lib/icinga2/api/zones-stage-status

Test-Host:

/var/lib/icinga2/api/zones
/var/lib/icinga2/api/zones/director-global
/var/lib/icinga2/api/zones/director-global/.timestamp
/var/lib/icinga2/api/zones/director-global/director
/var/lib/icinga2/api/zones/director-global/director/timeperiods.conf
/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf
/var/lib/icinga2/api/zones/director-global/director/user_templates.conf
/var/lib/icinga2/api/zones/director-global/director/commands.conf
/var/lib/icinga2/api/zones/director-global/director/service_templates.conf
/var/lib/icinga2/api/zones/director-global/director/servicesets.conf
/var/lib/icinga2/api/zones/director-global/director/command_templates.conf
/var/lib/icinga2/api/zones/director-global/director/hostgroups.conf
/var/lib/icinga2/api/zones/director-global/.checksums
/var/lib/icinga2/api/zones/test.cloud.com
/var/lib/icinga2/api/zones/test.cloud.com/.timestamp
/var/lib/icinga2/api/zones/test.cloud.com/director
/var/lib/icinga2/api/zones/test.cloud.com/director/host_templates.conf
/var/lib/icinga2/api/zones/test.cloud.com/.checksums
/var/lib/icinga2/api/zones-stage
/var/lib/icinga2/api/zones-stage/director-global
/var/lib/icinga2/api/zones-stage/director-global/.timestamp
/var/lib/icinga2/api/zones-stage/director-global/director
/var/lib/icinga2/api/zones-stage/director-global/director/timeperiods.conf
/var/lib/icinga2/api/zones-stage/director-global/director/001-director-basics.conf
/var/lib/icinga2/api/zones-stage/director-global/director/user_templates.conf
/var/lib/icinga2/api/zones-stage/director-global/director/commands.conf
/var/lib/icinga2/api/zones-stage/director-global/director/service_templates.conf
/var/lib/icinga2/api/zones-stage/director-global/director/servicesets.conf
/var/lib/icinga2/api/zones-stage/director-global/director/command_templates.conf
/var/lib/icinga2/api/zones-stage/director-global/director/hostgroups.conf
/var/lib/icinga2/api/zones-stage/director-global/.checksums
/var/lib/icinga2/api/zones-stage/test.cloud.com
/var/lib/icinga2/api/zones-stage/test.cloud.com/.timestamp
/var/lib/icinga2/api/zones-stage/test.cloud.com/director
/var/lib/icinga2/api/zones-stage/test.cloud.com/director/host_templates.conf
/var/lib/icinga2/api/zones-stage/test.cloud.com/.checksums
/var/lib/icinga2/api/zones-stage-startup.log
/var/lib/icinga2/api/zones-stage-status

Versions:

Master = r2.16.5-1
Satellite = r2.16.5-1
Test-Host = r2.16.5-1

icinga daemon -C

Master:

[2026-10-05 17:01:22 +0200] information/cli: Icinga application loader (version: r2.16.5-1)
[2026-10-05 17:01:22 +0200] information/cli: Loading configuration file(s).
[2026-10-05 17:01:22 +0200] information/ConfigItem: Committing config item(s).
[2026-10-05 17:01:22 +0200] information/ApiListener: My API identity: master.anywhere.com
[2026-10-05 17:01:22 +0200] warning/ApiListener: Security notice:
    Currently, all ApiUsers are allowed to use Icinga 2 DSL filter expressions
    in API queries because enforce_filter_expression_permission is set to false.
    This can pose a security risk as filters are evaluated within the Icinga 2
    process and their complexity can be used for denial of service attacks. The
    new 'filter-expression' permission can be used to allow this for individual
    ApiUsers, which should only be granted to trusted users. It is recommended
    to set enforce_filter_expression_permission to true to enforce the
    permission. This will become the default in v2.17.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 4 NotificationCommands.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 146 Notifications.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 HostGroup.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 8 Hosts.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 10 Zones.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 8 Endpoints.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 250 CheckCommands.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 User.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2026-10-05 17:01:22 +0200] information/ConfigItem: Instantiated 69 Services.
[2026-10-05 17:01:22 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2026-10-05 17:01:22 +0200] information/cli: Finished validating the configuration file(s).

Satellite:

[2026-10-05 16:58:41 +0200] information/cli: Icinga application loader (version: r2.16.5-1)
[2026-10-05 16:58:41 +0200] information/cli: Loading configuration file(s).
[2026-10-05 16:58:41 +0200] information/ConfigItem: Committing config item(s).
[2026-10-05 16:58:41 +0200] information/ApiListener: My API identity: satellite.cloud.com
[2026-10-05 16:58:41 +0200] warning/ApiListener: Security notice:
    Currently, all ApiUsers are allowed to use Icinga 2 DSL filter expressions
    in API queries because enforce_filter_expression_permission is set to false.
    This can pose a security risk as filters are evaluated within the Icinga 2
    process and their complexity can be used for denial of service attacks. The
    new 'filter-expression' permission can be used to allow this for individual
    ApiUsers, which should only be granted to trusted users. It is recommended
    to set enforce_filter_expression_permission to true to enforce the
    permission. This will become the default in v2.17.
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Webserver - Port' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 67:1-67:32) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Webserver - Cert' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 75:1-75:32) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Backup' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 93:1-93:22) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Docker Status' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 106:1-106:29) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Docker Health' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 117:1-117:29) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Proxmox GC Job' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 137:1-137:30) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Proxmox Prune Job' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 151:1-151:33) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Proxmox Verify Job' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 165:1-165:34) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Backup' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 187:1-187:22) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] warning/ApplyRule: Apply rule 'Docker Status' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 200:1-200:29) for type 'Service' does not match anywhere!
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 1 HostGroup.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 1 Host.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 5 Zones.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 3 Endpoints.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 250 CheckCommands.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2026-10-05 16:58:41 +0200] information/ConfigItem: Instantiated 5 Services.
[2026-10-05 16:58:41 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2026-10-05 16:58:41 +0200] information/cli: Finished validating the configuration file(s).

Test-Host:

[2026-10-05 16:55:04 +0200] information/cli: Icinga application loader (version: r2.16.5-1)
[2026-10-05 16:55:04 +0200] information/cli: Loading configuration file(s).
[2026-10-05 16:55:04 +0200] information/ConfigItem: Committing config item(s).
[2026-10-05 16:55:04 +0200] information/ApiListener: My API identity: test.cloud.com
[2026-10-05 16:55:04 +0200] warning/ApiListener: Security notice:
    Currently, all ApiUsers are allowed to use Icinga 2 DSL filter expressions
    in API queries because enforce_filter_expression_permission is set to false.
    This can pose a security risk as filters are evaluated within the Icinga 2
    process and their complexity can be used for denial of service attacks. The
    new 'filter-expression' permission can be used to allow this for individual
    ApiUsers, which should only be granted to trusted users. It is recommended
    to set enforce_filter_expression_permission to true to enforce the
    permission. This will become the default in v2.17.
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Linux - APT' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 9:1-9:27) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Linux - Disk' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 21:1-21:28) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Linux - Load' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 31:1-31:28) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Linux - SSH' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 40:1-40:27) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Linux - Processes' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 49:1-49:33) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Webserver - Port' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 67:1-67:32) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Webserver - Cert' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 75:1-75:32) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Backup' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 93:1-93:22) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Docker Status' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 106:1-106:29) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Docker Health' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 117:1-117:29) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Proxmox GC Job' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 137:1-137:30) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Proxmox Prune Job' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 151:1-151:33) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Proxmox Verify Job' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 165:1-165:34) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Backup' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 187:1-187:22) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] warning/ApplyRule: Apply rule 'Docker Status' (in /var/lib/icinga2/api/zones/director-global/director/servicesets.conf: 200:1-200:29) for type 'Service' does not match anywhere!
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 1 HostGroup.
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 4 Zones.
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 2 Endpoints.
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 250 CheckCommands.
[2026-10-05 16:55:04 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2026-10-05 16:55:04 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2026-10-05 16:55:04 +0200] information/cli: Finished validating the configuration file(s).

I hope I didn’t forget anything.

Thanks in advance!!!

Hello, and welcome in the Community!

Your Test-Host zones.conf is missing a reference to reach its satellite. Below an amended file.

Best regards,

Jean

Hi Jean!

Thank’s for your quick response!

Unfortunately, that didn’t work. The host is still showing as “PENDING.” If I understood the documentation correctly, that shouldn’t even be necessary, right? Since the satellite connects to the host, doesn’t it? Or did I misunderstand something?

Documentation says:

[root@icinga2-agent1.localdomain /]# vim /etc/icinga2/zones.conf

object Endpoint "icinga2-satellite1.localdomain" {
  // Do not actively connect to the satellite by leaving out the 'host' attribute
}

object Endpoint "icinga2-satellite2.localdomain" {
  // Do not actively connect to the satellite by leaving out the 'host' attribute
}

object Endpoint "icinga2-agent1.localdomain" {
  // That's us
}

object Zone "satellite" {
  endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain" ]
}

object Zone "icinga2-agent1.localdomain" {
  endpoints = [ "icinga2-agent1.localdomain" ]

  parent = "satellite"
}

/* sync global commands */
object Zone "global-templates" {
  global = true
}

object Zone "director-global" {
  global = true
}

After the change i shut down the satellite and test-host. Execute rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/* and started icinga2 again.

Log for this:

Satellite:

satellite.log (42.8 KB)

Test-Host:

testHost.log (36.4 KB)

Do you have any suggestions on where and how I can continue my search?

Thanks for your help!

Regards Tommy

Hello and welcom :slight_smile:

Am I correct in assuming that you currently have the test.cloud.com host defined in the Director as well as in the config file on the master under

If yes: this should be the source problem.

General rule of thumb: Either define “stuff” in the Director or the local config.

As the Directors option for Icinga2 Agent also creates the Endpoint and Zone object, removing the local config file should be enough. The Cluster Zone is correctly set to the Satellite zone.

Apart from that the correct path for confugrations that belong to a satellite zone would be /etc/icinga2/zones.d/<zone name>, so /etc/icinga2/zones.d/cloud in your case.

The following log line from the satellite hint to that:

[2026-10-05 21:50:09 +0200] warning/ApiListener: Ignoring config update from endpoint 'master.anywhere.com' for unknown zone 'test.cloud.com'.

Hi!

Sorry…that was my mistake. The test.cloud.com.conf was correct under /etc/icinga2/zones.d/cloud.

I’ve now deleted the file and ran the Kickstart Wizard again, so the Director recognized the zone and endpoint as deleted.

I also noticed that the Endpoint, Zone, and Host config files for test.cloud.com are located on the satellite at /var/lib/icinga2/api/zones/cloud/director.

Unfortunately, it still isn’t working. The hostalive check still stucks in PENDING.

Do you perhaps have any other tips?

Just to be sure: You deployed the Director config?

If yes and the deployment was successful: Nothing jumps out, that looks wrong. Neither in the logs nor the config.

Try enabling the debug log on both the agent and the satellite.

Maybe that gives some more insight.

Hi again,

After Kickstart Wizard and Deploy, could you go in Director pages “Endpoints” and “Zones” (both next to Kickstart), and send a screenshot? This to check if Director hasn’t been configured twice, both from the GUI and from the Kickstart.

Thank you,

Jean

Sure!

It’s just that the actual hostnames are listed here, but blurred out. Sorry for that…

Zones:

Cloud-Zone:

Endpoints:

Yes, I deployed everything properly afterward, and both the satellite and the test host were updated with the “new” configuration.

Logs attached via icinga2 daemon -x notice. Before starting Icinga2 again i executed rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*

satellite.debug.log (55.4 KB)

testHost.debug.log (46.3 KB)

That looks very good. The only difference with me is that in the Endpoints screen, I do have the Host name listed for the masters (but in my case both masters need to know about each other).

NB: I haven’t looked in the log files yet, will do that tomorrow as this PM is full - I hope Log1c will be able to pinpoint the issue

[2026-10-06 13:49:38 +0200] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'test.cloud.com': Error: End of file [asio.misc:2]

Stacktrace:
 0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 1# 0x000056486FB88B19 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 2# icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor>, long) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor>) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 4# 0x000056486FF50785 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
 5# make_fcontext in /lib/x86_64-linux-gnu/libboost_context.so.1.83.0
[2026-10-06 13:49:38 +0200] notice/JsonRpcConnection: Disconnecting API client for identity 'test.cloud.com'

this looks strange. happens twice in the satellite log, but seems to correspond with “reload finished” and “program stopped” on the agent side.

I still don’t really have an idea. Maybe try re-running the icinga2 node wizard on the agent

Hi Tommy,

I was in Icinga for another reason, and I just noticed that even though my config files on the satellites look similar to what you have, in Director, the satellite Host objects are placed in the Master (= your main) cluster zone.

So:

  • in their config files, the satellites are listed as Endpoints within the satellite (= your cloud) Zone definition
  • in Director, the satellite Host objects have attribute “Cluster Zone” set to master (= your main)

I am no longer sure why we have it this way, or if it required, but it works for us.

You might want to give it a try…

Best regards,

Jean

Hi Jean,

also done this before…without expected result.

Thank’s for your hint!

That really helped!

First, I deleted the TestHost in Director and deployed the configuration.

Then, on the satellite and the TestHost, I uninstalled everything related to icinga2 (apt purge cinga2*), reinstalled it, and ran the node wizard.

I added the host again. After that: hostalive worked right away, and all service checks are okay.

The crazy thing is that I’d done this at least twice before. Apparently, I somehow always managed to introduce an mistake…

I’m going to keep trying to install the remaining hosts using the Ansible roles. https://github.com/NETWAYS/ansible-collection-icinga/

Thank you so much for your quick help!

I’m really grateful!

Best regards,

Tommy

Happy you got it working! Might be something we all missed with the configs or the certificates.

This (master doing the host checks against the satellites) is best practice imo, so that you notice a satellite going down. The satellite only does it’s service checks itself (command_endpoint = satellite)