Proper procedure to remove zones

Icinga 2
, ,
1

Hi,

I have a large environment with two icinga masters running in HA and 70 satellites. During removal of Object zones from icinga cluster (add/remove zones with their endpoints, etc), I have come to situations where the slave master is unable to start, as there are references to ‘removed’ endpoints/objects on its _api folder; and where the only solution is to fully remove the /var/lib/icinga2/api folder so that node is able to start and api folder gets fully recreated on the replication from peer master node.

Unfortunately, this causes volatile objects (acks, comments, passive check results and downtimes) of the objects managed by the ‘touched’ master to fully dissapear from the cluster (peer masters deletes them when it syncs the new master), which becomes a major operational issue, as each lost ack means a notification for a false alert is sent to NOC/operations team; as the icinga master sending the notification has ‘forgotten’ that there exist acks/comments/passive_check_results which were registered prior to deleting the api folder.

Question:
A is there a procedure to properly remove the zone? (I have tried removing all related, config, and then removing the endpoints/zones - but leftovers on the api folder prevent startup of slave master).

current fail(ing) procedure:

  1. remove all checks involving the to-be-removed zones and endpoint from config master and wait for repilcation to slave master
  2. stop slave master
  3. run puppet on configuration master for removal of endpoints and restart of icinga service
  4. run puppet on slave master, for removal of endpoints and start of icinga2 service → Most of cases, the service start fails due to existance of unknown endpoints under the api folder
  5. if service errs due to previous:
  6. remove /var/lib/icinga2/api/zones* directories from slave master
  7. start icinga2 service on slave master

Question:
B is there an automated tool to replay the conf.d objects on any icinga master node?
( I know I could develop a script to parse the data in the .conf files, and then inject them into icinga2 api endpoint on either node, but I’m trying to avoid doing the development of said tool, as Icinga bin should somehow do this on startup - as restart of any node without touching ACTIVE_STAGE does not delete volatile data)

some_command recreate <path to tree containing the backed up_.conf files> ?

Relevant information:

  • Documentation:
    This does not solve the issue neither using the icinga2.state of the node where api folder is being deleted (slave master) nor using the icinga2.state from the other master node (configuration master)
    initial ha setup
  • sample size of volatile data on configuration master
root@monitoring-icinga-master-2 conf.d# ls
comments  downtimes
root@monitoring-icinga-master-2 conf.d# find comments/ -type f | wc -l
7468
root@monitoring-icinga-master-2 conf.d# find downtimes/ -type f | wc -l
833
  • sample size of volatile data on configuration slave
root@monitoring-icinga-master-1 c7b00ce2-5ffd-42fb-b090-98e2c9ca4bb6# cd conf.d/
root@monitoring-icinga-master-1 conf.d# find comments/ -type f | wc -l
7470
root@monitoring-icinga-master-1 conf.d# find downtimes/ -type f | wc -l
810
  • Version used (icinga2 --version)
    –tried two versions with same outcome:
# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.59.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

and

~$ sudo /usr/sbin/icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.6-1)
  • Operating System and version → Centos 7.9 (patched to Kernel version: 3.10.0-1160.53.1.el7.x86_64
    )
  • Enabled features (icinga2 feature list) → 
Disabled features: command compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql influxdb mainlog notification
  • Icinga Web 2 version and modules (System - About) → #N/A
  • Config validation (icinga2 daemon -C)
[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 184 ScheduledDowntimes.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 358 HostGroups.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 1 EventCommand.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 1 FileLogger.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 3 NotificationCommands.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 138740 Notifications.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 6477 Hosts.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 133173 Dependencies.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 60 Zones.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 72 Endpoints.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 1 ApiUser.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 433 CheckCommands.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 6 TimePeriods.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 2 Users.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 99024 Services.

[2022-04-08 07:28:14 +0000] information/ConfigItem: Instantiated 51 ServiceGroups.

[2022-04-08 07:28:14 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'

[2022-04-08 07:28:14 +0000] information/cli: Finished validating the configuration file(s).
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
$ cat zones.conf
# This file is managed by Puppet. DO NOT EDIT.

object Endpoint "c7bfa0b1bd2159e1314948d3a822a95c.somedomain.com" {
  host = "c7bfa0b1bd2159e1314948d3a822a95c.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "637fc6f5ffb79c0e950800a8bd3cd595.somedomain.com" {
  host = "637fc6f5ffb79c0e950800a8bd3cd595.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "49247b132b02d864258ff817984f5a1d.somedomain.com" {
  host = "49247b132b02d864258ff817984f5a1d.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "31e5d7094f5d2efbeb8f4ea802cbc4f1.somedomain.com" {
  host = "31e5d7094f5d2efbeb8f4ea802cbc4f1.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "b8d0f1084f084c1e45bfb6eb0e124b4d.somedomain.com" {
  host = "b8d0f1084f084c1e45bfb6eb0e124b4d.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "0ae52bb32ab87acf3125ffe87013f4ec.somedomain.com" {
  host = "0ae52bb32ab87acf3125ffe87013f4ec.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "cd1b4831b614c6132c8a2c857234fa9d.somedomain.com" {
  host = "cd1b4831b614c6132c8a2c857234fa9d.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "8fb251546d002f39cc41067a2ebf631a.somedomain.com" {
  host = "8fb251546d002f39cc41067a2ebf631a.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "10e51be6f0c13b568c1077fe53e22c96.somedomain.com" {
  host = "10e51be6f0c13b568c1077fe53e22c96.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "879971d9c5dc8379ae43cad29daa097a.somedomain.com" {
  host = "879971d9c5dc8379ae43cad29daa097a.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "636425a913730505ba235a5df31eab25.somedomain.com" {
  host = "636425a913730505ba235a5df31eab25.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "8c201720c4e6f26634e3b76560501057.somedomain.com" {
  host = "8c201720c4e6f26634e3b76560501057.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "0da17e6e5e3784f2c77175f5fa1fd31c.somedomain.com" {
  host = "0da17e6e5e3784f2c77175f5fa1fd31c.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "f5a601e6af9f9a799d4c5f4a1636a658.somedomain.com" {
  host = "f5a601e6af9f9a799d4c5f4a1636a658.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "a098e692ea1d3ab98bdaa13d53117c11.somedomain.com" {
  host = "a098e692ea1d3ab98bdaa13d53117c11.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "4b4d7a5e3793c24b229f30fc34d2254f.somedomain.com" {
  host = "4b4d7a5e3793c24b229f30fc34d2254f.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "a3adb183c443400cf778ef4b4a082d65.somedomain.com" {
  host = "a3adb183c443400cf778ef4b4a082d65.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "32eda11069f644e42de8f12d9e96c54f.somedomain.com" {
  host = "32eda11069f644e42de8f12d9e96c54f.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "41ec4e921eb9c72f23afe66e7c4d8370.somedomain.com" {
  host = "41ec4e921eb9c72f23afe66e7c4d8370.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "2a28426dfd04782fedeaa5dc481f9e72.somedomain.com" {
  host = "2a28426dfd04782fedeaa5dc481f9e72.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "d66f86c28e57dce1afd2a4a2e3a79fe3.somedomain.com" {
  host = "d66f86c28e57dce1afd2a4a2e3a79fe3.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "fd57e380161f7dcc2d357b95627d34a6.somedomain.com" {
  host = "fd57e380161f7dcc2d357b95627d34a6.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "d40f08a2655a2321d91481f76c39bbf5.somedomain.com" {
  host = "d40f08a2655a2321d91481f76c39bbf5.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "e5caba10582d1e560a0feb6be9719d27.somedomain.com" {
//  host = "monitoring-icinga-zonename_c963080767f45828c31f83ca5cd25d36-1.lat.internal"
//  port = 5665
}

object Endpoint "5b805ff225a41f61c7ca1b9c8dc64417.somedomain.com" {
//  host = "monitoring-icinga-zonename_c963080767f45828c31f83ca5cd25d36-2.lat.internal"
//  port = 5665
}

object Endpoint "61f8091658940ef5559a68c6ba3c9526.somedomain.com" {
  host = "61f8091658940ef5559a68c6ba3c9526.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "eed8ca285273fcdf288dd59efd61967a.somedomain.com" {
  host = "eed8ca285273fcdf288dd59efd61967a.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "768acfd17eaa63d1567e54f6df22a8ca.somedomain.com" {
  host = "768acfd17eaa63d1567e54f6df22a8ca.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "1549c345429695e123d7ed2f900c390a.somedomain.com" {
  host = "1549c345429695e123d7ed2f900c390a.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "cd75035e9c5f11d3bb84bf7207c04b8e.somedomain.com" {
  host = "cd75035e9c5f11d3bb84bf7207c04b8e.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "b2a7e3dfbe19ac0debb802768487d8a4.somedomain.com" {
  host = "b2a7e3dfbe19ac0debb802768487d8a4.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "688f1074c53292da248ecd33a36a054d.somedomain.com" {
  host = "688f1074c53292da248ecd33a36a054d.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "767c545db3611c4bde0e62d797b0f68e.somedomain.com" {
  host = "767c545db3611c4bde0e62d797b0f68e.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "469fb5798fc7f6b9349e533829e93eab.somedomain.com" {
  host = "469fb5798fc7f6b9349e533829e93eab.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "f2da1c07bae56bed0835319bb7d8e28a.somedomain.com" {
  host = "f2da1c07bae56bed0835319bb7d8e28a.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "28602792825bc7b395a0ebfb681dd2be.somedomain.com" {
  host = "28602792825bc7b395a0ebfb681dd2be.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "8304b655d3a0da921f38321c180271e3.somedomain.com" {
  host = "8304b655d3a0da921f38321c180271e3.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "049c9b534e7ef996221181df1ef6a854.somedomain.com" {
  host = "049c9b534e7ef996221181df1ef6a854.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "e8b9ca282cf0c4cc5d88458399859177.somedomain.com" {
  host = "e8b9ca282cf0c4cc5d88458399859177.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "817cc590329fda4025500651319f45f6.somedomain.com" {
  host = "817cc590329fda4025500651319f45f6.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "d82842ffc455009cc490dee2aeedba33.somedomain.com" {
  host = "d82842ffc455009cc490dee2aeedba33.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "2e63b714e52cff51d92e011ebeedbff0.somedomain.com" {
  host = "2e63b714e52cff51d92e011ebeedbff0.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "829294fa66d8cc843b7551e56200d0d7.somedomain.com" {
  host = "829294fa66d8cc843b7551e56200d0d7.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "670bf98527831d5da40b31d2b695da9b.somedomain.com" {
  host = "670bf98527831d5da40b31d2b695da9b.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "ec2655d90cc984289dc022975f094b2b.somedomain.com" {
  host = "ec2655d90cc984289dc022975f094b2b.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "cfb2807f543956636b927ea85b2a29d6.somedomain.com" {
  host = "cfb2807f543956636b927ea85b2a29d6.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "351808fcb2bb3a54224608642276a10c.somedomain.com" {
  host = "351808fcb2bb3a54224608642276a10c.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "82db38b6f4fcb4492bac4edc0480c9cd.somedomain.com" {
  host = "82db38b6f4fcb4492bac4edc0480c9cd.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "c092784efdf26c246cd7959923eaaf53.somedomain.com" {
  host = "c092784efdf26c246cd7959923eaaf53.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "84e421e2d2a504268c143bbd5ae9b951.somedomain.com" {
  host = "84e421e2d2a504268c143bbd5ae9b951.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "894bdde0bf3f184a3aeffcaf8032ad70.somedomain.com" {
  host = "894bdde0bf3f184a3aeffcaf8032ad70.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "9dc4d0d49c7a3754b7484c9567695c51.somedomain.com" {
  host = "9dc4d0d49c7a3754b7484c9567695c51.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "018a42dbb8f72b654401c81eed204c73.somedomain.com" {
  host = "018a42dbb8f72b654401c81eed204c73.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "b9346f31316b6c1e4ea484e6b6b168e0.somedomain.com" {
  host = "b9346f31316b6c1e4ea484e6b6b168e0.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "0bb8561f385b84134ecfffda066e3f6b.somedomain.com" {
  host = "0bb8561f385b84134ecfffda066e3f6b.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "a796e5c8e05283e3a6928517865ceea0.somedomain.com" {
  host = "a796e5c8e05283e3a6928517865ceea0.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "f284e3a0e36ee7d43ff7bfd92849d31b.somedomain.com" {
  host = "f284e3a0e36ee7d43ff7bfd92849d31b.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "ebd771631519f2a52a77a46aafcd7cfe.somedomain.com" {
  host = "ebd771631519f2a52a77a46aafcd7cfe.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "479e47c3d51fcf9852b0233484853d6d.somedomain.com" {
  host = "479e47c3d51fcf9852b0233484853d6d.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "6c3fc6adaa1fcbeebfcd8fe18cc73c70.somedomain.com" {
  host = "6c3fc6adaa1fcbeebfcd8fe18cc73c70.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "30f645994029bf64d116b7b6f16d395b.somedomain.com" {
  host = "30f645994029bf64d116b7b6f16d395b.somedomain.com"
  port = 5665
  log_duration = 0
}

object Endpoint "f8119b22e2268ad1249928a9ae89fb07.somedomain.com" {
  host = "f8119b22e2268ad1249928a9ae89fb07.somedomain.com"
  port = 5665
  log_duration = 0
}

object Zone "zonename_6775462c36c27befd443a7047728131e" {
  endpoints = [ "61f8091658940ef5559a68c6ba3c9526.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_a04f4349bbe62f77d875c33e49e02bf0" {
  endpoints = [ "eed8ca285273fcdf288dd59efd61967a.somedomain.com", "2e63b714e52cff51d92e011ebeedbff0.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_e9d791ed5b2bd1d26b5ae793497ae140" {
  endpoints = [ "84e421e2d2a504268c143bbd5ae9b951.somedomain.com", "018a42dbb8f72b654401c81eed204c73.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_b1e5e2c407b559c9638e37e16ff009b5" {
  endpoints = [ "768acfd17eaa63d1567e54f6df22a8ca.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_795a5ddf74e43056392c3ea542ad0272" {
  endpoints = [ "1549c345429695e123d7ed2f900c390a.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_26cbb186accca3b8cf8b824e486b983e" {
  endpoints = [ "cd75035e9c5f11d3bb84bf7207c04b8e.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_1008b749ec12b8d0433cad843213e89c" {
  endpoints = [ "b2a7e3dfbe19ac0debb802768487d8a4.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_fa93cc5eeb2f332eb0ef6df46a46c69c" {
  endpoints = [ "c7bfa0b1bd2159e1314948d3a822a95c.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_8887f989bb918064af100fdd7e8736cc" {
  endpoints = [ "637fc6f5ffb79c0e950800a8bd3cd595.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_8887f989bb918064af100fdd7e8736cc-2" {
  endpoints = [ "49247b132b02d864258ff817984f5a1d.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_681e0b6cfcf653d7e0d800c290455bb2" {
  endpoints = [ "31e5d7094f5d2efbeb8f4ea802cbc4f1.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_32ebaa50888c5f718aeef276a96da1ad" {
  endpoints = [ "b8d0f1084f084c1e45bfb6eb0e124b4d.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_d90b36930b002b2d407091260aa342a1" {
  endpoints = [ "0ae52bb32ab87acf3125ffe87013f4ec.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_d90b36930b002b2d407091260aa342a1-2" {
  endpoints = [ "cd1b4831b614c6132c8a2c857234fa9d.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_c15c384c514e1b3e0460a7f138ad6cf1" {
  endpoints = [ "8fb251546d002f39cc41067a2ebf631a.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_c15c384c514e1b3e0460a7f138ad6cf1-2" {
  endpoints = [ "10e51be6f0c13b568c1077fe53e22c96.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_c15c384c514e1b3e0460a7f138ad6cf1-3" {
  endpoints = [ "879971d9c5dc8379ae43cad29daa097a.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_c15c384c514e1b3e0460a7f138ad6cf1-4" {
  endpoints = [ "636425a913730505ba235a5df31eab25.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_70672b212e4f7375178a43cc5ed3ba95" {
  endpoints = [ "8c201720c4e6f26634e3b76560501057.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_4cc7e50cb3ab1143a35321e17e2fbb26" {
  endpoints = [ "0da17e6e5e3784f2c77175f5fa1fd31c.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_9e3f80030f73436f859839ccc1bf784e" {
  endpoints = [ "f5a601e6af9f9a799d4c5f4a1636a658.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_9180380c41462775b1fe68301af7815c" {
  endpoints = [ "a098e692ea1d3ab98bdaa13d53117c11.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_ac7ce4b765af998e21044e3351a2dcb9" {
  endpoints = [ "4b4d7a5e3793c24b229f30fc34d2254f.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_cbb8b3ee86d3b1c9d9d55e53bf9f4bee" {
  endpoints = [ "a3adb183c443400cf778ef4b4a082d65.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_0476f6f161a195b0bf8dc16158f2814f" {
  endpoints = [ "32eda11069f644e42de8f12d9e96c54f.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_ea9dde99bee0f9bae0dcf4009081f033" {
  endpoints = [ "41ec4e921eb9c72f23afe66e7c4d8370.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_b3d8840e5d980163ea1412ef72c36995" {
  endpoints = [ "2a28426dfd04782fedeaa5dc481f9e72.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_eea23f6def5249dabf70e540c9c17bd8" {
  endpoints = [ "d66f86c28e57dce1afd2a4a2e3a79fe3.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_7a728b0b42a7b5f2e144474001724f7b" {
  endpoints = [ "fd57e380161f7dcc2d357b95627d34a6.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_f785b2867d1a21ab29bb2a81756cc3b0" {
  endpoints = [ "d40f08a2655a2321d91481f76c39bbf5.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_df08ba46cef74ae8dc79b234eea8438f" {
  endpoints = [ "688f1074c53292da248ecd33a36a054d.somedomain.com", "829294fa66d8cc843b7551e56200d0d7.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_b6b3f51996c7da2d8adcbb44baa39750" {
  zonename_b6b3f51996c7da2d8adcbb44baa39750 = true
}

object Zone "zonename_b6b3f51996c7da2d8adcbb44baa39750-external" {
  zonename_b6b3f51996c7da2d8adcbb44baa39750 = true
}

object Zone "zonename_b6b3f51996c7da2d8adcbb44baa39750-internal" {
  zonename_b6b3f51996c7da2d8adcbb44baa39750 = true
}

object Zone "zonename_87d6f48d599a6c5c699bfaa86a5ffb1b" {
  endpoints = [ "767c545db3611c4bde0e62d797b0f68e.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_80f4b26de6671fe5dc598a5dc88763bd" {
  endpoints = [ "469fb5798fc7f6b9349e533829e93eab.somedomain.com", "670bf98527831d5da40b31d2b695da9b.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_311602f99d3c564a3042ba498b152fba" {
  endpoints = [ "894bdde0bf3f184a3aeffcaf8032ad70.somedomain.com", "b9346f31316b6c1e4ea484e6b6b168e0.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_717c79b316521b4b2ace7b69705ac0a3" {
  endpoints = [ "f2da1c07bae56bed0835319bb7d8e28a.somedomain.com", "ec2655d90cc984289dc022975f094b2b.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_338c7d8f2d0d77353db15ce26bacfde0" {
  endpoints = [ "9dc4d0d49c7a3754b7484c9567695c51.somedomain.com", "0bb8561f385b84134ecfffda066e3f6b.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_f8bf1e7f0a14db782dc1c1f550d8cce4" {
  endpoints = [ "28602792825bc7b395a0ebfb681dd2be.somedomain.com", "cfb2807f543956636b927ea85b2a29d6.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_c963080767f45828c31f83ca5cd25d36" {
  endpoints = [ "e5caba10582d1e560a0feb6be9719d27.somedomain.com", "5b805ff225a41f61c7ca1b9c8dc64417.somedomain.com", ]
}

object Zone "zonename_f342d7cace82fa0fc2d7bc8f3bf0f650" {
  endpoints = [ "8304b655d3a0da921f38321c180271e3.somedomain.com", "351808fcb2bb3a54224608642276a10c.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_cfe84fbb81790752cbe9066b238dd0c8" {
  endpoints = [ "049c9b534e7ef996221181df1ef6a854.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_f74d6d7375ad147d89934787b63ebbac" {
  endpoints = [ "e8b9ca282cf0c4cc5d88458399859177.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_f902e00c5df1ccff6a1f410ee70ce752" {
  endpoints = [ "a796e5c8e05283e3a6928517865ceea0.somedomain.com", "6c3fc6adaa1fcbeebfcd8fe18cc73c70.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_6b54dc2ec168dcafc692d87ccaa09588" {
  endpoints = [ "f284e3a0e36ee7d43ff7bfd92849d31b.somedomain.com", "30f645994029bf64d116b7b6f16d395b.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_acb5b1a7d595c4c0a5aa220fa8a2d015" {
  endpoints = [ "ebd771631519f2a52a77a46aafcd7cfe.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_b401444e2ada1c5ebf21458196f4d22d" {
  endpoints = [ "479e47c3d51fcf9852b0233484853d6d.somedomain.com", "f8119b22e2268ad1249928a9ae89fb07.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_0c52548ff711c3044f69a982c387fadf" {
  endpoints = [ "817cc590329fda4025500651319f45f6.somedomain.com", "82db38b6f4fcb4492bac4edc0480c9cd.somedomain.com", ]
  parent = ZoneName
}

object Zone "zonename_8c57c05a44c7ea5bdb8f164c140c309c" {
  endpoints = [ "d82842ffc455009cc490dee2aeedba33.somedomain.com", "c092784efdf26c246cd7959923eaaf53.somedomain.com", ]
  parent = ZoneName
}

Replication steps (shall you be curious enough)

  1. Identify a service that is being notified by slave master
root@monitoring-icinga-master-1.ltrudev.internal ~# grep Notification icinga2.log |  grep Completed | grep snmp-check-postgres-disabled_triggers |tail -n2
[2022-04-07 10:21:26 -0400] information/Notification: Completed sending 'Problem' notification 'postgresql-crash-game-2.ltrudev.internal!snmp-check-postgres-disabled_triggers!alertaio-event-service-nonprod' for checkable 'postgresql-crash-game-2.ltrudev.internal!snmp-check-postgres-disabled_triggers' and user 'alertaio-events' using command 'alertaio-event-service-command'.

2.Create an ACK on an alerted service (i.e. icinga api or icinga web) that gets notified by icinga master node slave

[2022-04-07 10:22:57 -0400] information/Notification: Completed sending 'Acknowledgement' notification 'postgresql-crash-game-2.ltrudev.internal!snmp-check-postgres-disabled_triggers!alertaio-event-service-nonprod' for checkable 'postgresql-crash-game-2.ltrudev.internal!snmp-check-postgres-disabled_triggers' and user 'alertaio-events' using command 'alertaio-event-service-command'.
  1. confirm the ack has been created under /var/lib/icinga2/api/packages/_api//conf.d/hostname!servicename.conf on both icinga2 masters
root@monitoring-icinga-master-1 ~# ls /var/lib/icinga2/api/packages/_api/fa84caeb-9fa4-4714-afb1-97d7f9a8c9b0/conf.d/comments/
postgresql-crash-game-2.ltrudev.internal!snmp-check-postgres-disabled_triggers!c5afbc6a-9a4b-4de2-91d0-6068c19b5acc.conf
  1. stop the slave master (the one not holding anything under /etc/icinga2/conf.d but receiving it directly from its peer)
  2. remove the /var/lib/icinga2/api/zones and /var/lib/icinga2/api/zones-stage
  3. start icinga2 service
  4. give the node a few minutes to startup / sync / start test/notification cycle. (3 to10 depending on the size)
  5. confirm the conf.d/comment/ has been removed from the master nodes, but still exists on icingaweb (database).
  6. watch how the notification is sent again.
2

Hi,

please give an example failure. (icinga2 daemon -C)

Best,
A/K

3

Hi Al, I’m afraid I don’t have that information available at the moment. The journal has already been rotated, and I cleared the api without registering the error. I’ll get hold of it next time it occurs, but I’m afraid that has no ‘estimated’ date, as zone removal happens a few times during a year. Best,