Backend icinga is not running Error

Hello,

We have had an Icinga2 cluster setup for some time and this morning for some reason we are seeing ‘Backend icinga is not running’ message in the webUI seemingly with no changes. We’ve done some troubleshooting and narrowed it down to our 2nd icinga2 host. When we stop the icinga2 service on PRDICINGA02, the error goes away in the IcingaWeb UI and hosts and service being reporting correctly again. The icinga2 service starts as expected.

Details

  • Icinga Version
    Icinga 2.13.1-1

  • Operating System and version
    Debian9

  • Enabled features (icinga2 feature list)
    PRDICINGA01
    root@PRDICINGA01:/etc/systemd/system# icinga2 feature list
    Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 livestatus opentsdb perfdata statusdata syslog
    Enabled features: api checker command ido-mysql influxdb mainlog notification
    PRDICINGA02
    root@PRDICINGA02:/etc/systemd/system# icinga2 feature list
    Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 livestatus opentsdb perfdata statusdata syslog
    Enabled features: api checker command ido-mysql influxdb mainlog notification

  • Icinga Web 2 version and modules (System - About)
    IcingaWeb2 2.9.5

  • Config validation (icinga2 daemon -C)
    PRDICINGA01
    root@PRDICINGA01:/etc/systemd/system# icinga2 daemon -C
    [2022-01-19 11:54:27 -0800] information/cli: Icinga application loader (version: r2.13.1-1)
    [2022-01-19 11:54:27 -0800] information/cli: Loading configuration file(s).
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Committing config item(s).
    [2022-01-19 11:54:27 -0800] information/ApiListener: My API identity: PRDICINGA01.domain.net
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘backup-downtime’ (in /etc/icinga2/zones.d/master/downtimes.conf: 5:1-5:52) for type ‘ScheduledDowntime’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘apt’ (in /etc/icinga2/zones.d/master/apt.conf: 1:0-1:18) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘Services Task’ (in /etc/icinga2/zones.d/master/azure-services.conf: 45:1-45:33) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘Firewall Auth Gateway’ (in /etc/icinga2/zones.d/master/sso-agent.conf: 1:0-1:37) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘Firewall Event Log’ (in /etc/icinga2/zones.d/master/firewall-sso-agent.conf: 12:1-12:35) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website0.com’ (in /etc/icinga2/zones.d/master/certs/website0_com.conf: 1:0-1:55) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website1.com’ (in /etc/icinga2/zones.d/master/certs/website1_com.conf: 1:0-1:47) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website2.com’ (in /etc/icinga2/zones.d/master/certs/website2_com.conf: 1:0-1:53) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website3.com’ (in /etc/icinga2/zones.d/master/certs/website3.conf: 1:0-1:53) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website4.com’ (in /etc/icinga2/zones.d/master/certs/website4.conf: 1:0-1:48) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website5.com’ (in /etc/icinga2/zones.d/master/website5.conf: 1:0-1:49) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website6.com’ (in /etc/icinga2/zones.d/master/certs/website6_com.conf: 1:0-1:47) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website7.com’ (in /etc/icinga2/zones.d/master/certs/website7_com.conf: 1:0-1:55) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website8.com’ (in /etc/icinga2/zones.d/master/certs/website8_com.conf: 1:0-1:49) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website9.com’ (in /etc/icinga2/zones.d/master/certs/website9_com.conf: 1:0-1:48) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website10.com’ (in /etc/icinga2/zones.d/master/certs/website10_com.conf: 1:0-1:48) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 5 NotificationCommands.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1691 Notifications.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 3 HostGroups.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 348 Hosts.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 14 Downtimes.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 2 Comments.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 FileLogger.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 ApiListener.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 109 Zones.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 ExternalCommandListener.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 108 Endpoints.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 3 ApiUsers.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 InfluxdbWriter.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 259 CheckCommands.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 1 NotificationComponent.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 9 UserGroups.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 13 Users.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 3 TimePeriods.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 3 ServiceGroups.
    [2022-01-19 11:54:27 -0800] information/ConfigItem: Instantiated 521 Services.
    [2022-01-19 11:54:27 -0800] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
    [2022-01-19 11:54:27 -0800] information/cli: Finished validating the configuration file(s).
    PRDICINGA02
    root@PRDICINGA02:/etc/systemd/system# icinga2 daemon -C
    [2022-01-19 11:53:40 -0800] information/cli: Icinga application loader (version: r2.13.1-1)
    [2022-01-19 11:53:40 -0800] information/cli: Loading configuration file(s).
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Committing config item(s).
    [2022-01-19 11:53:41 -0800] information/ApiListener: My API identity: PRDICINGA02.domain.net
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘backup-downtime’ (in /etc/icinga2/zones.d/master/downtimes.conf: 5:1-5:52) for type ‘ScheduledDowntime’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘apt’ (in /etc/icinga2/zones.d/master/apt.conf: 1:0-1:18) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘Services Task’ (in /etc/icinga2/zones.d/master/azure-services.conf: 45:1-45:33) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘Firewall Auth Gateway’ (in /etc/icinga2/zones.d/master/sso-agent.conf: 1:0-1:37) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘Firewall Event Log’ (in /etc/icinga2/zones.d/master/firewall-sso-agent.conf: 12:1-12:35) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website0.com’ (in /etc/icinga2/zones.d/master/certs/website0_com.conf: 1:0-1:55) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website1.com’ (in /etc/icinga2/zones.d/master/certs/website1_com.conf: 1:0-1:47) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website2.com’ (in /etc/icinga2/zones.d/master/certs/website2_com.conf: 1:0-1:53) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website3.com’ (in /etc/icinga2/zones.d/master/certs/website3.conf: 1:0-1:53) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website4.com’ (in /etc/icinga2/zones.d/master/certs/website4.conf: 1:0-1:48) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website5.com’ (in /etc/icinga2/zones.d/master/website5.conf: 1:0-1:49) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website6.com’ (in /etc/icinga2/zones.d/master/certs/website6_com.conf: 1:0-1:47) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website7.com’ (in /etc/icinga2/zones.d/master/certs/website7_com.conf: 1:0-1:55) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website8.com’ (in /etc/icinga2/zones.d/master/certs/website8_com.conf: 1:0-1:49) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website9.com’ (in /etc/icinga2/zones.d/master/certs/website9_com.conf: 1:0-1:48) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:54:27 -0800] warning/ApplyRule: Apply rule ‘CERTIFICATE - website10.com’ (in /etc/icinga2/zones.d/master/certs/website10_com.conf: 1:0-1:48) for type ‘Service’ does not match anywhere!
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 5 NotificationCommands.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1691 Notifications.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 IcingaApplication.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 3 HostGroups.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 348 Hosts.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 14 Downtimes.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 2 Comments.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 FileLogger.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 CheckerComponent.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 ApiListener.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 109 Zones.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 ExternalCommandListener.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 108 Endpoints.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 3 ApiUsers.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 InfluxdbWriter.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 259 CheckCommands.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 1 NotificationComponent.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 9 UserGroups.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 13 Users.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 3 TimePeriods.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 3 ServiceGroups.
    [2022-01-19 11:53:41 -0800] information/ConfigItem: Instantiated 521 Services.
    [2022-01-19 11:53:41 -0800] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
    [2022-01-19 11:53:41 -0800] information/cli: Finished validating the configuration file(s).

Database Info
1 HAProxy server - Debian 9
2 MariaDB servers - Debian 9
10.1.48-MariaDB-0+deb9u2 Debian 9.13

MariaDB01 status_update_time
image

MariaDB02 status_update_time
image

NTP is configured on each host and the time is correct

First time poster, please let me know if you are interested in more information. Thanks for taking the time.

Hello @junkratt!

(How) Do /etc/icinga2/features-available/ido-mysql.conf on the hosts differ?

Best,
A/K