Icinga2 HA setup - High log lag in master zone

Hi together :slight_smile:

since switching to an icinga2 ha setup, we have unfortunately had a large log lag, but only in the master_zone. This value is increasing by minute. The current log in /var/lib/icinga2/api/log/current is on config master 280 kb and on secondary master 0 kb.

there is no network problem between the two masters. We have a 3 Level Setup with 2 Masters. X Satellites and Agents. Icinga director is in use

many thanks for your support in advance! :slight_smile:

Give as much information as you can, e.g.

  • Version used (icinga2 --version)

All icinga2 master and satellites: version: r2.14.2-1

  • Operating System and version

Icinga2 masters on: Debian 11

Icinga2 satellites on Ubuntu 22.04 or Debian 11

  • Enabled features (icinga2 feature list)

Icinga2 config master: api checker command compatlog graphite ido-mysql livestatus mainlog notification perfdata

icinga2 second master: api checker command graphite ido-mysql livestatus mainlog notification perfdata

  • Icinga Web 2 version and modules (System - About)

2.12.1

businessprocess 2.5.0

company 1.0.0

director 1.11.0

fileshipper 1.2.0

graphite 1.2.4

idoreports 0.10.1

incubator 0.20.0

monitoring 2.12.1

pdfexport 0.10.2

reporting 0.10.0

  • Config validation (icinga2 daemon -C)
icinga2 config master:

2024-02-08 09:04:16 +0100] information/cli: Icinga application loader (version: r2.14.2-1)

[2024-02-08 09:04:16 +0100] information/cli: Loading configuration file(s).

[2024-02-08 09:04:16 +0100] information/ConfigItem: Committing config item(s).

[2024-02-08 09:04:16 +0100] information/ApiListener: My API identity: icinga_configmaster

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Apache2 http TCP Port 80’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 17:1-17:46) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Graphite Carbon Relay TCP Port 20013’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 81:1-81:58) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Graphite Carbon Relay TCP Port 20014’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 94:1-94:58) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check MSSQL AVG Latches-Wait Time’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 324:1-324:49) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check MySQL Keycache Hitrate’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 754:1-754:44) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Nginx http TCP Port 80’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 766:1-766:44) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check MySQL Tablecache Hitrate’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 832:1-832:46) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check MySQL Index Usage’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 856:1-856:39) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Debian Syslog Daemon’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1444:1-1444:42) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check haproxy Main Daemon’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1498:1-1498:41) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Nginx Main Daemon’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1542:1-1542:39) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Tomcat8 Main Daemon’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1552:1-1552:41) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Tomcat9 Main Daemon’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1561:1-1561:41) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check disk-smart’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1612:1-1612:32) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘Check Nginx Status’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/director-global/service_apply.conf: 1712:1-1712:34) for type ‘Service’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘ilert-service-notification’ (in /etc/icinga2/zones.d/global_notifications/notifications_ilert_icinga2.conf: 33:1-33:58) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘mail-service’ (in /etc/icinga2/zones.d/global_notifications/notifications_mail-services.conf: 6:1-6:44) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘mail-service2’ (in /etc/icinga2/zones.d/global_notifications/notifications_mail-services.conf: 18:1-18:45) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘notifications_slack-services’ (in /etc/icinga2/zones.d/global_notifications/notifications_slack-services.conf: 6:1-6:60) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘notifications_slack-services2’ (in /etc/icinga2/zones.d/global_notifications/notifications_slack-services.conf: 19:1-19:61) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘teams-service’ (in /etc/icinga2/zones.d/global_notifications/notifications_teams-services.conf: 6:1-6:45) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘teams-service2’ (in /etc/icinga2/zones.d/global_notifications/notifications_teams-services.conf: 18:1-18:46) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘view-service’ (in /etc/icinga2/zones.d/global_notifications/notifications_view-services.conf: 6:1-6:44) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘view-service2’ (in /etc/icinga2/zones.d/global_notifications/notifications_view-services.conf: 20:1-20:45) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘ilert-host-notification’ (in /etc/icinga2/zones.d/global_notifications/notifications_ilert_icinga2.conf: 23:1-23:52) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘mail-hots’ (in /etc/icinga2/zones.d/global_notifications/notifications_mail-hosts.conf: 6:1-6:38) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘notifications_slack-hosts’ (in /etc/icinga2/zones.d/global_notifications/notifications_slack-hosts.conf: 6:1-6:54) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘teams-hosts’ (in /etc/icinga2/zones.d/global_notifications/notifications_teams-hosts.conf: 6:1-6:40) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘view-host’ (in /etc/icinga2/zones.d/global_notifications/notifications_view-hosts.conf: 6:1-6:38) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] warning/ApplyRule: Apply rule ‘ilert-host-notification’ (in /var/lib/icinga2/api/packages/director/2028b7c0-21c2-409f-919a-025ed1c76472/zones.d/ZONE_MASTER/notification_apply.conf: 1:0-1:51) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 NotificationCommand.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 636 Notifications.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 10 HostGroups.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 206 Hosts.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 Comment.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 FileLogger.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 161 Zones.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 4 Users.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 149 Endpoints.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 CompatLogger.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 3 ApiUsers.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 2 LivestatusListeners.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 ApiListener.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 317 CheckCommands.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 PerfdataWriter.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 2 UserGroups.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 31 ServiceGroups.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 1 TimePeriod.

[2024-02-08 09:04:16 +0100] information/ConfigItem: Instantiated 3780 Services.

[2024-02-08 09:04:16 +0100] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’

[2024-02-08 09:04:16 +0100] information/cli: Finished validating the configuration file(s).

icinga2 secondmaster:

[2024-02-08 09:07:18 +0100] information/cli: Icinga application loader (version: r2.14.2-1)

[2024-02-08 09:07:18 +0100] information/cli: Loading configuration file(s).

[2024-02-08 09:07:18 +0100] information/ConfigItem: Committing config item(s).

[2024-02-08 09:07:18 +0100] information/ApiListener: My API identity: icinga2_secondmaster

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Apache2 http TCP Port 80’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 17:1-17:46) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Graphite Carbon Relay TCP Port 20013’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 81:1-81:58) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Graphite Carbon Relay TCP Port 20014’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 94:1-94:58) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check MSSQL AVG Latches-Wait Time’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 324:1-324:49) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check MySQL Keycache Hitrate’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 754:1-754:44) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Nginx http TCP Port 80’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 766:1-766:44) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check MySQL Tablecache Hitrate’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 832:1-832:46) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check MySQL Index Usage’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 856:1-856:39) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Debian Syslog Daemon’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1444:1-1444:42) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check haproxy Main Daemon’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1498:1-1498:41) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Nginx Main Daemon’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1542:1-1542:39) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Tomcat8 Main Daemon’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1552:1-1552:41) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Tomcat9 Main Daemon’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1561:1-1561:41) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check disk-smart’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1612:1-1612:32) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘Check Nginx Status’ (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 1712:1-1712:34) for type ‘Service’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘ilert-service-notification’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_ilert_icinga2.conf: 33:1-33:58) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘mail-service’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_mail-services.conf: 6:1-6:44) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘mail-service2’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_mail-services.conf: 18:1-18:45) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘notifications_slack-services’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_slack-services.conf: 6:1-6:60) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘notifications_slack-services2’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_slack-services.conf: 19:1-19:61) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘teams-service’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_teams-services.conf: 6:1-6:45) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘teams-service2’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_teams-services.conf: 18:1-18:46) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘view-service’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_view-services.conf: 6:1-6:44) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘view-service2’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_view-services.conf: 20:1-20:45) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘ilert-host-notification’ (in /var/lib/icinga2/api/zones/ZONE_MASTER/director/notification_apply.conf: 1:0-1:51) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘ilert-host-notification’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_ilert_icinga2.conf: 23:1-23:52) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘mail-hots’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_mail-hosts.conf: 6:1-6:38) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘notifications_slack-hosts’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_slack-hosts.conf: 6:1-6:54) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘teams-hosts’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_teams-hosts.conf: 6:1-6:40) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] warning/ApplyRule: Apply rule ‘view-host’ (in /var/lib/icinga2/api/zones/global_notifications/_etc/notifications_view-hosts.conf: 6:1-6:38) for type ‘Notification’ does not match anywhere!

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 NotificationCommand.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 636 Notifications.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 10 HostGroups.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 206 Hosts.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 86 Downtimes.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 Comment.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 FileLogger.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 161 Zones.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 4 Users.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 149 Endpoints.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 2 ApiUsers.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 2 LivestatusListeners.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 ApiListener.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 317 CheckCommands.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 PerfdataWriter.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 2 UserGroups.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 31 ServiceGroups.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 1 TimePeriod.

[2024-02-08 09:07:18 +0100] information/ConfigItem: Instantiated 3780 Services.

[2024-02-08 09:07:18 +0100] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’

[2024-02-08 09:07:19 +0100] information/cli: Finished validating the configuration file(s).

  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
icinga2_secondmaster zones.conf:

// =========== ENDPOINT CONFIGURATION ===============

object Endpoint “icinga2_secondmaster” {

}

// Define the local Endpoint ot the local server

object Endpoint “icinga_configmaster” {

host = “icinga_configmaster”

port = “5665”

}

// Define the ENDPOINT configuration for Host: satellite1

object Endpoint “satellite1.xx” {

host = “satellite1.xx”

}

// Define the ENDPOINT configuration for Host: satellite2

object Endpoint “satellite2.xx” {

host = “satellite2.xx”

}

// Define the ENDPOINT configuration for Host: satellite3

object Endpoint “satellite3.xx” {

host = “satellite3.xx”

}

// Define the ENDPOINT configuration for Host: satellite4

object Endpoint “satellite4.xx” {

host = “satellite4.xx”

}

// Define the ENDPOINT configuration for Host: satellite5

object Endpoint “satellite5.xx” {

host = “satellite5.xx”

}

// Define the ENDPOINT configuration for Host: satellite6.xx

object Endpoint “satellite6.xx” {

host = “satellite6.xx”

}

// Define the ENDPOINT configuration for Host: satellite7.xx

object Endpoint “satellite7.xx” {

host = “satellite7.xx”

}

// Define the ENDPOINT configuration for Host: satellite8.xx

object Endpoint “satellite8.xx” {

host = “satellite8.xx”

}

// Define the ENDPOINT configuration for Host: satellite9.xx

object Endpoint “satellite9.xx” {

host = “satellite9.xx”

}

// =========== SYNC ZONE CONFIGURATION ===============

// Zone/Master Defintion: ZONE_MASTER

object Zone “ZONE_MASTER” {

endpoints = [ “icinga2_secondmaster”, “icinga_configmaster” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE1

object Zone “ZONE_SATELLITE1” {

parent = “ZONE_MASTER”

endpoints = [ “satellite1.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE2

object Zone “ZONE_SATELLITE2” {

parent = “ZONE_MASTER”

endpoints = [ “satellite2.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE3

object Zone “ZONE_SATELLITE3” {

parent = “ZONE_MASTER”

endpoints = [ “satellite3.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE4

object Zone “ZONE_SATELLITE4” {

parent = “ZONE_MASTER”

endpoints = [ “satellite4.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE5

object Zone “ZONE_SATELLITE5” {

parent = “ZONE_MASTER”

endpoints = [ “satellite5.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE6

object Zone “ZONE_SATELLITE6” {

parent = “ZONE_MASTER”

endpoints = [ “satellite6.xx” ]

}

object Zone “ZONE_SATELLITE7” {

parent = “ZONE_MASTER”

endpoints = [ “satellite7.xx” ]

}

object Zone “ZONE_SATELLITE8” {

parent = “ZONE_MASTER”

endpoints = [ “satellite8.xx” ]

}

object Zone “ZONE_SATELLITE9” {

parent = “ZONE_MASTER”

endpoints = [ “satellite8.xx” ]

}

// =========== GLOBAL ZONE CONFIGURATION ===============

// Global Zones for Templating

object Zone “global_checkcommands” {

global = true

}

object Zone “global_commands” {

global = true

}

object Zone “global_contacts” {

global = true

}

object Zone “global_contactgroups” {

global = true

}

object Zone “global_downtimes” {

global = true

}

object Zone “global_hostgroups” {

global = true

}

object Zone “global_notifications” {

global = true

}

object Zone “global_passwords” {

global = true

}

object Zone “global_servicegroups” {

global = true

}

object Zone “global_templates” {

global = true

}

object Zone “global_timeperiods” {

global = true

}

object Zone “local_dependencies” {

global = true

}

object Zone “director-global” {

global = true

}

/*

object Zone “director-global” {

global = true

}*/

icinga_configmaster zones.conf:

// =========== ENDPOINT CONFIGURATION ===============

object Endpoint “icinga2_secondmaster” {

host = “icinga2_secondmaster”

port = “5665”

}

// Define the local Endpoint ot the local server

object Endpoint “icinga_configmaster” {

}

// Define the ENDPOINT configuration for Host: satellite1.xx

object Endpoint “satellite1.xx” {

host = “satellite1.xx”

}

// Define the ENDPOINT configuration for Host: satellite2.xx

object Endpoint “satellite2.xx” {

host = “satellite2.xx”

}

// Define the ENDPOINT configuration for Host: satellite3.xx

object Endpoint “satellite3.xx” {

host = “satellite3.xx”

}

// Define the ENDPOINT configuration for Host: satellite4.xx

object Endpoint “satellite4.xx” {

host = “satellite4.xx”

}

// Define the ENDPOINT configuration for Host: satellite5.xx

object Endpoint “satellite5.xx” {

host = “satellite5.xx”

}

// Define the ENDPOINT configuration for Host: satellite6.xx

object Endpoint “satellite6.xx” {

host = “satellite6.xx”

}

// Define the ENDPOINT configuration for Host: satellite7.xx

object Endpoint “satellite7.xx” {

host = “satellite7.xx”

}

// Define the ENDPOINT configuration for Host: satellite8.xx

object Endpoint “satellite8.xx” {

host = “satellite8.xx”

}

// Define the ENDPOINT configuration for Host: satellite9.xx

object Endpoint “satellite9.xx” {

host = “satellite9.xx”

}

// =========== SYNC ZONE CONFIGURATION ===============

// Zone/Master Defintion: ZONE_MASTER

object Zone “ZONE_MASTER” {

endpoints = [ “icinga2_secondmaster”, “icinga_configmaster” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE1

object Zone “ZONE_SATELLITE1” {

parent = “ZONE_MASTER”

endpoints = [ “satellite1.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE2

object Zone “ZONE_SATELLITE2” {

parent = “ZONE_MASTER”

endpoints = [ “satellite2.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE3

object Zone “ZONE_SATELLITE3” {

parent = “ZONE_MASTER”

endpoints = [ “satellite3.xx” ]

}

// Zone/Satellite Defintion: ZONE_SATELLITE4

object Zone “ZONE_SATELLITE4” {

parent = “ZONE_MASTER”

endpoints = [ “satellite4.xx” ]

}

// Zone/Satellite Defintion: ZONE_satellite5.xx

object Zone “ZONE_satellite5” {

parent = “ZONE_MASTER”

endpoints = [ “satellite5.xx” ]

}

// Zone/Satellite Defintion: ZONE_satellite6.xx

object Zone “ZONE_satellite6” {

parent = “ZONE_MASTER”

endpoints = [ “satellite6.xx” ]

}

// Zone/Satellite Defintion: ZONE_satellite7.xx

object Zone “ZONE_SATELLITE7” {

parent = “ZONE_MASTER”

endpoints = [ “satellite7.xx” ]

}

// Zone/Satellite Defintion: ZONE_satellite8.xx

object Zone “ZONE_SATELLITE8” {

parent = “ZONE_MASTER”

endpoints = [ “satellite8.xx” ]

}

// Zone/Satellite Defintion: ZONE_satellite9.xx

object Zone “ZONE_SATELLITE9” {

parent = “ZONE_MASTER”

endpoints = [ “satellite9.xx” ]

}

// =========== GLOBAL ZONE CONFIGURATION ===============

// Global Zones for Templating

object Zone “global_checkcommands” {

global = true

}

object Zone “global_commands” {

global = true

}

object Zone “global_contacts” {

global = true

}

object Zone “global_contactgroups” {

global = true

}

object Zone “global_downtimes” {

global = true

}

object Zone “global_hostgroups” {

global = true

}

object Zone “global_notifications” {

global = true

}

object Zone “global_passwords” {

global = true

}

object Zone “global_servicegroups” {

global = true

}

object Zone “global_templates” {

global = true

}

object Zone “global_timeperiods” {

global = true

}

object Zone “local_dependencies” {

global = true

}

object Zone “director-global” {

global = true

}

does anyone have any ideas? :slight_smile: