Icinga2 HA - enabling notification on second host crashes icinga2

Icinga 2
1

I have a Icinga2 HA setup with to masters.
On one of my masters (node2) the feature notification is currently disabled. I discovered that this was causing the other master (node1) to not send sms notifications since it would :

[2021-07-12 14:51:32 -0400] notice/Notification: Notification 'ca04vlgraylogweb01!check_events!sms-escalation': HA cluster active, this endpoint does not have the authority (paused=true). Skipping.

In my attempt to fix this after much debugging (including enabling debuglog temporarily) I enabled notification feature on node2. at this point the icinga2 service started crashing repeatedly.
It seemed to crash just as it finished sync based to node1 and started attempting to send notifications.

Does any one have any insight into why the crash is occuring?

Crash log below

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2021-07-12 15:07:49 -0400

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2021-07-12 15:12:15 -0400

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2021-07-12 15:19:37 -0400

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2021-07-12 15:25:09 -0400

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2021-07-12 15:29:31 -0400

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:199: T* boost::intrusive_ptr<T>::operator->() const [with T = icinga::Dictionary]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2021-07-12 15:34:18 -0400

node2:

  • Version used (version: r2.12.4-1)
  • Operating System and version 20.04.2 LTS (Focal Fossa)
  • Enabled features (Enabled features: api checker ido-mysql influxdb mainlog)
  • Icinga Web 2 version and modules (System - About)
|setup|2.9.0|
|---|---|
|grafana|1.4.2|
|monitoring|2.9.0|
  • Config validation (valid)
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
    node1
/* THIS FILE IS MANAGED BY PUPPET.*/
object Endpoint "ca02vlnms01.agilitypr.internal" {
}
object Endpoint "ca04vlnms01.agilitypr.internal" {

        host = "10.4.10.41"

}
object Zone "master" {
    endpoints = [ "ca02vlnms01.agilitypr.internal","ca04vlnms01.agilitypr.internal" ]
}
object Zone "global-templates" {
        global = true
}

node2

object Endpoint "ca02vlnms01.agilitypr.internal" {
}
object Endpoint "ca04vlnms01.agilitypr.internal" {

}
object Zone "master" {
    endpoints = [ "ca02vlnms01.agilitypr.internal","ca04vlnms01.agilitypr.internal" ]
}
object Zone "global-templates" {
        global = true
}
2

I have added additional details in a related ticket I found:

Does anyone know how to clear notification as the other user idicated they did in the ticket?
this would at least let me enable notifications and not have the service crash.

3

Both masters will stay up and running in the following state, but as soon as I enable notifications the secondary (not config master) icinga2 service crashes. No core dump, just a crash log.

image
image1312×426 19.6 KB