-
Icinga Web 2 version
2.8.2
Git commit
8a89839af94a247ee2149b2336c73b8251b477c0
PHP Version
7.4.3-4ubuntu2.18
Git commit date
2020-08-17 -
Used modules and their versions (System - About)
aws 1.1.0
businessprocess 2.3.0
cube 1.1.0
director 1.8.0
elasticsearch 1.0.0
graphite 1.1.0
idoreports 0.9.1
incubator 0.6.0
ipl v0.5.0
monitoring 2.8.2
pdfexport 0.9.1
reactbundle 0.8.0 -
Web browser used
Chrome -
Icinga 2 version used (icinga2 --version)
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.1-1) -
PHP version used (php --version)
PHP 7.4.3-4ubuntu2.18 (cli) (built: Feb 23 2023 12:43:23) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.3-4ubuntu2.18, Copyright (c), by Zend Technologies -
Server operating system and version
System information:
Platform: Ubuntu
Platform version: 20.04.1 LTS (Focal Fossa)
Kernel: Linux
Kernel version: 5.15.0-1037-aws
Architecture: x86_64 -
Currently, I see the following message: 'Backend icinga is not running ’
Everything looks good.
All systemctl status are green -
icinga2 daemon -C
[2023-06-14 11:49:24 +0000] information/cli: Icinga application loader (version: r2.12.1-1)
[2023-06-14 11:49:24 +0000] information/cli: Loading configuration file(s).
[2023-06-14 11:49:24 +0000] information/ConfigItem: Committing config item(s).
[2023-06-14 11:49:24 +0000] information/ApiListener: My API identity: ************
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ElasticsearchWriter.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 150 Hosts.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 4 NotificationCommands.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 Comment.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 246 Notifications.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 9 HostGroups.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 EventCommand.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 917 Dependencies.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 152 Zones.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 150 Endpoints.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 267 CheckCommands.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 TimePeriod.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 2 Users.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 1029 Services.
[2023-06-14 11:49:24 +0000] information/ConfigItem: Instantiated 9 ServiceGroups.
[2023-06-14 11:49:24 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-06-14 11:49:24 +0000] information/cli: Finished validating the configuration file(s).
-
cat /etc/icinga2/features-enabled/ido-mysql.conf
/** -
The db_ido_mysql library implements IDO functionality
-
for MySQL.
/
library “db_ido_mysql”
object IdoMysqlConnection “ido-mysql” {
user = “admin”,
password = “********”,
host = "*********",
database = “icinga”,
categories = [ “DbCatConfig”, “DbCatState”, “DbCatAcknowledgement”, “DbCatComment”, “DbCatDowntime”, “DbCatEventHandler”, “DbCatFlapping”, “DbCatCheck”, “DbCatNotification”, “DbCatProgramStatus”, “DbCatRetention”, “DbCatStateHistory” ]
} -
grep Ido /var/log/icinga2/icinga2.log | tail -n 200
[2023-06-14 11:43:35 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2023-06-14 11:43:36 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2023-06-14 11:43:45 +0000] information/IdoMysqlConnection: Pending queries: 608 (Input: 63/s; Output: 2/s)
[2023-06-14 11:43:55 +0000] information/IdoMysqlConnection: Pending queries: 1133 (Input: 52/s; Output: 0/s)
[2023-06-14 11:44:05 +0000] information/IdoMysqlConnection: Pending queries: 1588 (Input: 45/s; Output: 0/s)
[2023-06-14 11:44:15 +0000] information/IdoMysqlConnection: Pending queries: 2119 (Input: 53/s; Output: 1/s)
[2023-06-14 11:44:25 +0000] information/IdoMysqlConnection: Pending queries: 2736 (Input: 62/s; Output: 1/s)
[2023-06-14 11:44:35 +0000] information/IdoMysqlConnection: Pending queries: 3306 (Input: 57/s; Output: 1/s)
[2023-06-14 11:44:45 +0000] information/IdoMysqlConnection: Pending queries: 3725 (Input: 40/s; Output: 1/s)
[2023-06-14 11:44:55 +0000] information/IdoMysqlConnection: Pending queries: 4131 (Input: 41/s; Output: 0/s)
[2023-06-14 11:45:05 +0000] information/IdoMysqlConnection: Pending queries: 4521 (Input: 39/s; Output: 0/s)
[2023-06-14 11:45:15 +0000] information/IdoMysqlConnection: Pending queries: 5115 (Input: 59/s; Output: 0/s)
[2023-06-14 11:45:25 +0000] information/IdoMysqlConnection: Pending queries: 5498 (Input: 38/s; Output: 0/s)
[2023-06-14 11:45:35 +0000] information/IdoMysqlConnection: Pending queries: 6021 (Input: 52/s; Output: 0/s)
[2023-06-14 11:45:45 +0000] information/IdoMysqlConnection: Pending queries: 6416 (Input: 39/s; Output: 0/s)
[2023-06-14 11:45:55 +0000] information/IdoMysqlConnection: Pending queries: 6996 (Input: 58/s; Output: 0/s)
[2023-06-14 11:46:05 +0000] information/IdoMysqlConnection: Pending queries: 7401 (Input: 40/s; Output: 0/s)
[2023-06-14 11:46:12 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga' in 156.585 second(s).
[2023-06-14 11:48:55 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 58/s; Output: 59/s)
Once every ±2 minutes, icigna issues the message “Backend icinga is not running”, and in the logs, I can see that it is trying to reconnect to MySQL (RDS db.r5.xlarge). It takes ±2 minutes to reconnect and then everything becomes normal for 2 minutes. Then the error starts again.