The monitoring backend "icinga" is not running

BrotherA · October 17, 2019, 4:03pm

Hello,

my Icinga2 is running, but the Web GUI shows the message

The monitoring backend “icinga” is not running.
I tried already a reload and a restart of the icinga2 service

# systemctl restart icinga2
# systemctl status icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           └─limits.conf
   Active: active (running) since Thu 2019-10-17 18:01:00 CEST; 3s ago
  Process: 4139 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 4146 (icinga2)
    Tasks: 395
   Memory: 122.7M
   CGroup: /system.slice/icinga2.service
           ├─4146 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
           ├─4559 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
           └─4560 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 1 ApiListener.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 235 CheckCommands.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 1 User.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 16 Services.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
Okt 17 18:01:00 neckar icinga2[4146]: [2019-10-17 18:01:00 +0200] information/cli: Closing console log.
Okt 17 18:01:00 neckar systemd[1]: Started Icinga host/service/network monitoring system.

How I can restart the monitoring backend?

Best regards
BrotherA

Pooh · October 17, 2019, 4:15pm

Any clues in /var/log/icinga2/startup.log ?

Antony.

BrotherA · October 17, 2019, 5:27pm

No, that file doesn’t exist. But if you remember for the log files, at /var/log/icinga2/icinga2.log I found

[2019-10-17 14:14:09 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 51, rate:  0/s (0/min 0/5min 0/15min); empty in 17 seconds
[2019-10-17 14:14:19 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 86, rate:  0/s (0/min 0/5min 0/15min); empty in 24 seconds
[2019-10-17 14:14:29 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 121, rate:  0/s (0/min 0/5min 0/15min); empty in 34 seconds
[2019-10-17 14:14:39 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 156, rate:  0/s (0/min 0/5min 0/15min); empty in 44 seconds
[2019-10-17 14:14:49 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 195, rate:  0/s (0/min 0/5min 0/15min); empty in 49 seconds
[2019-10-17 14:14:59 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 228, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 9 seconds
[2019-10-17 14:15:09 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 257, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 28 seconds
[2019-10-17 14:15:19 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 292, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 23 seconds
[2019-10-17 14:15:29 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 325, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 38 seconds
[2019-10-17 14:15:39 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 360, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 42 seconds
[2019-10-17 14:15:49 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 397, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 47 seconds
[2019-10-17 14:15:59 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 432, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 3 seconds
[2019-10-17 14:15:59 +0200] critical/IdoMysqlConnection: Connection to database 'icinga2' with user 'icinga2' on '192.166.1.66:3306' failed: "Can't connect to MySQL server on '192.166.1.66' (115)"
Context:
        (0) Reconnecting to MySQL IDO database 'ido-mysql'

[2019-10-17 14:15:59 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2019-10-17 14:16:09 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 460, rate: 0.0166667/s (1/min 1/5min 1/15min); empty in 2 minutes and 44 seconds
[2019-10-17 14:16:19 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 497, rate: 0.0166667/s (1/min 1/5min 1/15min); empty in 2 minutes and 14 seconds
[2019-10-17 14:16:29 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 528, rate: 0.0166667/s (1/min 1/5min 1/15min); empty in 2 minutes and 50 seconds
[2019-10-17 14:16:39 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 565, rate: 0.0166667/s (1/min 1/5min 1/15min); empty in 2 minutes and 32 seconds
[2019-10-17 14:16:49 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 600, rate: 0.0166667/s (1/min 1/5min 1/15min); empty in 2 minutes and 51 seconds
[2019-10-17 14:16:59 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 635, rate:  0/s (0/min 1/5min 1/15min); empty in 3 minutes and 1 second
[2019-10-17 14:17:09 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 664, rate:  0/s (0/min 1/5min 1/15min); empty in 3 minutes and 48 seconds
[2019-10-17 14:17:19 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 701, rate:  0/s (0/min 1/5min 1/15min); empty in 3 minutes and 9 seconds
[2019-10-17 14:17:29 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 734, rate:  0/s (0/min 1/5min 1/15min); empty in 3 minutes and 42 seconds
[2019-10-17 14:17:39 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 771, rate:  0/s (0/min 1/5min 1/15min); empty in 3 minutes and 28 seconds
[2019-10-17 14:17:49 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 804, rate:  0/s (0/min 1/5min 1/15min); empty in 4 minutes and 3 seconds
[2019-10-17 14:17:59 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 841, rate:  0/s (0/min 1/5min 1/15min); empty in 3 minutes and 47 seconds
[2019-10-17 14:18:09 +0200] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 870, rate:  0/s (0/min 1/5min 1/15min); empty in 4 minutes and 59 seconds
[2019-10-17 14:18:10 +0200] critical/IdoMysqlConnection: Connection to database 'icinga2' with user 'icinga2' on '192.166.1.66:3306' failed: "Can't connect to MySQL server on '192.166.1.66' (115)"
Context:
        (0) Reconnecting to MySQL IDO database 'ido-mysql'

And that part is repeating the whole time.

Thanks,

BrotherA

dnsmichi · October 17, 2019, 6:38pm

That means that the IDO festure cannot write anything to the MySQL database thus rendering sour web view stale.

In a secondary topic you are referring to firewall rules - likely affecting this as well. Port 3306 seems to be blocked.

Cheers,
Michael

BrotherA · October 18, 2019, 6:16am

Good morning,
and thanks for your information - I’m new in using Icinga. At to the issue of the firewall rules: MySQL connections were never a problem, the Icinga was running well before the ido-mysql feature stopped suddenly in the afternoon yesterday.

netstat -tlpn | grep 3306
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      73208/mysqld

Additionally the Icinga and MySQL run on the same host. So I took a view at /etc/icinga2/features-available/ido-mysql.conf, where I found the IP address 192.168.1.1 instead of localhost. I changed it and restarted Icinga again. Now this problem is solved.

Best regards

BrotherA