HA setup distributed checks

Icinga 2
1

Hi!
I have a problem, after icinga2 upgrade to 2.15 (r2.15.0-1), all checks are done only by one master, something changed? I have seen changelog but nothing is jumping on me that could caused that.

  • Version used (icinga2 --version) - r2.15.0-1
  • Operating System and version - Debian 12
  • Enabled features (icinga2 feature list) - Enabled features: api checker debuglog icingadb mainlog notification
  • Icinga Web 2 version and modules (System - About)
  • Config validation (icinga2 daemon -C)
[2025-06-30 11:10:12 +0000] information/cli: Icinga application loader (version: r2.15.0-1)
[2025-06-30 11:10:12 +0000] information/cli: Loading configuration file(s).
[2025-06-30 11:10:12 +0000] information/ConfigItem: Committing config item(s).
[2025-06-30 11:10:12 +0000] information/ApiListener: My API identity: monitor2
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 Host.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 2 FileLoggers.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 Zone.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 2 Endpoints.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 2 ApiUsers.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 247 CheckCommands.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 1 ServiceGroup.
[2025-06-30 11:10:12 +0000] information/ConfigItem: Instantiated 10 Services.
[2025-06-30 11:10:12 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-06-30 11:10:12 +0000] information/cli: Finished validating the configuration file(s).
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
Object 'monitor1' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:35
  * __name = "monitor1"
  * host = ""
  * log_duration = 86400
  * name = "monitor1"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 35
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "monitor1" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:35
  * type = "Endpoint"
  * zone = "master"
    % = modified in '/etc/icinga2/zones.conf', lines 7:2-7:16

Object 'monitor2' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 10:1-10:35
  * __name = "monitor2"
  * host = ""
  * log_duration = 86400
  * name = "monitor2"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 10
    * last_column = 35
    * last_line = 10
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "monitor2" ]
    % = modified in '/etc/icinga2/zones.conf', lines 10:1-10:35
  * type = "Endpoint"
  * zone = "master"
    % = modified in '/etc/icinga2/zones.conf', lines 11:2-11:16

Object 'master' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 15:1-15:20
  * __name = "master"
  * endpoints = [ "monitor1", "monitor2" ]
    % = modified in '/etc/icinga2/zones.conf', lines 16:2-16:57
  * global = false
  * name = "master"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 20
    * last_line = 15
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "master" ]
    % = modified in '/etc/icinga2/zones.conf', lines 15:1-15:20
  * type = "Zone"
  * zone = "master"
    % = modified in '/etc/icinga2/zones.conf', lines 17:2-17:16

FYI: on monitor1 all check are paused.

https://localhost:5665/v1/objects/services | jq '.results[].attrs | {name: .name, zone: .zone, paused: .paused}

Regards,
Przemek

2

Hi, yes this a new feature in Icinga 2.15.0, host child objects are distributed across the nodes based on their host name. So, if the node A is responsible for the host “foo”, then it will also be responsible for all the child objects of that host like services, notifications, etc. See ApiListener::UpdateObjectAuthority(): distribute auth. by object's host by Al2Klimov · Pull Request #10161 · Icinga/icinga2 · GitHub

Edit: You can also find more info about how this works in Technical Concepts - Icinga 2

3 Likes
3

Yes, this helped :slight_smile: Thank you!

Regards,
Przemek