Monitoring backend 'icinga' is not running

  • Icinga Web 2 version
    2.8.2
    Git commit
    8a89839af94a247ee2149b2336c73b8251b477c0
    PHP Version
    7.4.3-4ubuntu2.18
    Git commit date
    2020-08-17

  • Used modules and their versions (System - About)
    aws 1.1.0
    businessprocess 2.3.0
    cube 1.1.0
    director 1.8.0
    elasticsearch 1.0.0
    graphite 1.1.0
    idoreports 0.9.1
    incubator 0.6.0
    ipl v0.5.0
    monitoring 2.8.2
    pdfexport 0.9.1
    reactbundle 0.8.0

  • Web browser used
    Chrome

  • Icinga 2 version used (icinga2 --version)
    icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.1-1)

  • PHP version used (php --version)
    PHP 7.4.3-4ubuntu2.18 (cli) (built: Feb 23 2023 12:43:23) ( NTS )
    Copyright (c) The PHP Group
    Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3-4ubuntu2.18, Copyright (c), by Zend Technologies

  • Server operating system and version
    System information:
    Platform: Ubuntu
    Platform version: 20.04.1 LTS (Focal Fossa)
    Kernel: Linux
    Kernel version: 5.15.0-1037-aws
    Architecture: x86_64

  • Currently, I see the following message: 'Backend icinga is not running ’
    Everything looks good.
    All systemctl status are green

  • icinga2 daemon -C

[2023-06-14 11:49:24 +0000] information/cli: Icinga application loader (version: r2.12.1-1)
[2023-06-14 11:49:24 +0000] information/cli: Loading configuration file(s).
[2023-06-14 11:49:24 +0000] information/ConfigItem: Committing config item(s).
[2023-06-14 11:49:24 +0000] information/ApiListener: My API identity: ************
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ElasticsearchWriter.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 150 Hosts.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 4 NotificationCommands.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 Comment.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 246 Notifications.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 9 HostGroups.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 EventCommand.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 917 Dependencies.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 152 Zones.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 150 Endpoints.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 267 CheckCommands.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 TimePeriod.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 2 Users.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1029 Services.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 9 ServiceGroups.
[2023-06-14 11:49:24 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-06-14 11:49:24 +0000] information/cli: Finished validating the configuration file(s).
  • cat /etc/icinga2/features-enabled/ido-mysql.conf
    /**

  • The db_ido_mysql library implements IDO functionality

  • for MySQL.
    /
    library “db_ido_mysql”
    object IdoMysqlConnection “ido-mysql” {
    user = “admin”,
    password = “********”,
    host = "
    *********",
    database = “icinga”,
    categories = [ “DbCatConfig”, “DbCatState”, “DbCatAcknowledgement”, “DbCatComment”, “DbCatDowntime”, “DbCatEventHandler”, “DbCatFlapping”, “DbCatCheck”, “DbCatNotification”, “DbCatProgramStatus”, “DbCatRetention”, “DbCatStateHistory” ]
    }

  • grep Ido /var/log/icinga2/icinga2.log | tail -n 200

[2023-06-14 11:43:35 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2023-06-14 11:43:36 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2023-06-14 11:43:45 +0000] information/IdoMysqlConnection: Pending queries: 608 (Input: 63/s; Output: 2/s)
[2023-06-14 11:43:55 +0000] information/IdoMysqlConnection: Pending queries: 1133 (Input: 52/s; Output: 0/s)
[2023-06-14 11:44:05 +0000] information/IdoMysqlConnection: Pending queries: 1588 (Input: 45/s; Output: 0/s)
[2023-06-14 11:44:15 +0000] information/IdoMysqlConnection: Pending queries: 2119 (Input: 53/s; Output: 1/s)
[2023-06-14 11:44:25 +0000] information/IdoMysqlConnection: Pending queries: 2736 (Input: 62/s; Output: 1/s)
[2023-06-14 11:44:35 +0000] information/IdoMysqlConnection: Pending queries: 3306 (Input: 57/s; Output: 1/s)
[2023-06-14 11:44:45 +0000] information/IdoMysqlConnection: Pending queries: 3725 (Input: 40/s; Output: 1/s)
[2023-06-14 11:44:55 +0000] information/IdoMysqlConnection: Pending queries: 4131 (Input: 41/s; Output: 0/s)
[2023-06-14 11:45:05 +0000] information/IdoMysqlConnection: Pending queries: 4521 (Input: 39/s; Output: 0/s)
[2023-06-14 11:45:15 +0000] information/IdoMysqlConnection: Pending queries: 5115 (Input: 59/s; Output: 0/s)
[2023-06-14 11:45:25 +0000] information/IdoMysqlConnection: Pending queries: 5498 (Input: 38/s; Output: 0/s)
[2023-06-14 11:45:35 +0000] information/IdoMysqlConnection: Pending queries: 6021 (Input: 52/s; Output: 0/s)
[2023-06-14 11:45:45 +0000] information/IdoMysqlConnection: Pending queries: 6416 (Input: 39/s; Output: 0/s)
[2023-06-14 11:45:55 +0000] information/IdoMysqlConnection: Pending queries: 6996 (Input: 58/s; Output: 0/s)
[2023-06-14 11:46:05 +0000] information/IdoMysqlConnection: Pending queries: 7401 (Input: 40/s; Output: 0/s)
[2023-06-14 11:46:12 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga' in 156.585 second(s).
[2023-06-14 11:48:55 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 58/s; Output: 59/s)

Once every ±2 minutes, icigna issues the message “Backend icinga is not running”, and in the logs, I can see that it is trying to reconnect to MySQL (RDS db.r5.xlarge). It takes ±2 minutes to reconnect and then everything becomes normal for 2 minutes. Then the error starts again.

It looks like icinga and MySQL are running on different hosts. Hence, you might have a network issue or the db machine is too busy.

I use RDS db.r5.xlarge (from AWS), it has a ±15% load.

“network issue” can it be checked manually?
because I have the main problem with icinga, it opens an alert for pagerduty, I see in the logs a request to create an alert, but when it is fixed, it sometimes does not close alert in pagerduty, I think it’s happening because I get “Backend icinga is not running”