Icinga2 distributed monitoring, distributed checks and command endpoints

Icinga 2
1

I have two icinga nodes in a master-master setup.
When the first master is “in charge”, everything works fine.
When the second master is leading, there are some errors:

Some config files can’t be accessed (e.g. the one with ipmi credentials, let it be /etc/icinga2/zones.d/master/hosts/ipmi/ipmi.cfg).
The second master stores its temporary config in /var/lib/icinga2/api/zones-stage/master/_etc/hosts/ipmi/ipmi.cfg but it tries to use the one from /etc/icinga2/zones.d/... as configured in the host file:

vars.ipmi_config_file = [ SysconfDir + "/icinga2/zones.d/master/hosts/ipmi/ipmi.cfg" ]

is there a possibility to get this running without abstaining from the use of the separate ipmi.cfg file?

  1. When the issuing of checks is distributed about both masters and master2 is the check source, there sometimes is the error message “master2 is not connected to host-xy”.
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.3-1)

System information:
  Platform: Debian GNU/Linux
  Platform version: 12 (bookworm)
  Kernel: Linux
  Kernel version: 6.1.0-28-amd64
  Architecture: x86_64

Enabled features: api checker debuglog graphite icingadb mainlog

Icinga Web 2 Version 	2.11.4
PHP Version 	8.2.26
Get Icinga Support
Icinga Community
Report a bug
Icinga Documentation
Loaded Libraries
icinga/icinga-php-library 	0.10.1
icinga/icinga-php-thirdparty 	0.11.0
Loaded Modules
icingadb 		1.0.2 	Configure
graphite 		1.2.2 	Configure
Copyright © 2013-2024 Icinga GmbH

[2024-12-10 12:32:47 +0100] information/cli: Icinga application loader (version: r2.14.3-1)
[2024-12-10 12:32:47 +0100] information/cli: Loading configuration file(s).
[2024-12-10 12:32:47 +0100] information/ConfigItem: Committing config item(s).
[2024-12-10 12:32:47 +0100] information/ApiListener: My API identity: ....de
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'startpatch-downtime-sunday2' (in /etc/icinga2/zones.d/master/downtimes.conf: 28:1-28:61) for type 'ScheduledDowntime' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'startpatch-downtime-sunday4' (in /etc/icinga2/zones.d/master/downtimes.conf: 39:1-39:61) for type 'ScheduledDowntime' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'startpatch-downtime-sunday2-4' (in /etc/icinga2/zones.d/master/downtimes.conf: 50:1-50:63) for type 'ScheduledDowntime' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'procs' (in /etc/icinga2/zones.d/master/services.conf: 42:1-42:21) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'apt' (in /etc/icinga2/zones.d/master/services.conf: 166:1-166:19) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/zones.d/master/services.conf: 194:1-194:21) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'iscsi' (in /etc/icinga2/zones.d/master/services.conf: 206:1-206:21) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'check_gpu' (in /etc/icinga2/zones.d/master/services/nvidia-smi.conf: 1:0-1:24) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'cluster' (in /etc/icinga2/zones.d/master/services/pve.conf: 8:1-8:23) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'services' (in /etc/icinga2/zones.d/master/services/pve.conf: 16:1-16:24) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'updates' (in /etc/icinga2/zones.d/master/services/pve.conf: 30:1-30:23) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'io_wait' (in /etc/icinga2/zones.d/master/services/pve.conf: 50:1-50:23) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'pve-vm ' (in /etc/icinga2/zones.d/master/services/pve.conf: 92:1-92:72) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] warning/ApplyRule: Apply rule 'check_nfs-common' (in /etc/icinga2/zones.d/master/services/systemd.conf: 9:1-9:32) for type 'Service' does not match anywhere!
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 555 Notifications.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 8 HostGroups.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 103 Hosts.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 5 Downtimes.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 796 Dependencies.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 2 FileLoggers.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 86 Zones.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 User.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 86 Endpoints.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 4 ApiUsers.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 259 CheckCommands.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 9 ServiceGroups.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 4 TimePeriods.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 7 ScheduledDowntimes.
[2024-12-10 12:32:48 +0100] information/ConfigItem: Instantiated 796 Services.
[2024-12-10 12:32:48 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2024-12-10 12:32:48 +0100] information/cli: Finished validating the configuration file(s).
2

can you post the zones.conf of one host that is not working?
/etc/icinga2/zones.conf

3

Hi, thank you for the hint!
that solved the one problem with the checks from the second master not working (master 2 was missing in the zones.conf of the hosts).

Any idea about the issue with the absolute paths in the icinga2 config not recognized by the second master?

Edit: my ugly solution is to put an array in ipmi.conf [ "value1", "value2" ] containing the original path from master1 and the var path to the synced conf dir on master2.

vars.ipmi_config_file = [ SysconfDir + "/icinga2/zones.d/master/hosts/ipmi/ipmi.cfg" , "/var/lib/icinga2/api/zones-stage/master/_etc/hosts/ipmi/ipmi.cfg" ]