"Backend icinga is not running" but everything seams fine

gaboo · October 31, 2019, 11:05am

Hello icinga users,
I’ve been using icinga for a few years now. I recently upgraded from icinga-web to icingaweb2.

I also upgraded icinga2 from v2.7 to latest 2.11. I did the schema upgrades.
I cleaned the config from deprecated stuff.

The IDO plugin works, I can see all my hosts and services from the web2 ui, and I can send commands using the API.

After a few settings changes and upgrades, everything seams to work nicely.

Except a small annoying thing :

Screenshot_20191031_114648

I have read many threads regarding this issue on github or other forum and unfortunately, I don’t know how to fix it.

The 2014-07-23 date is probably the date the instance was first setup.

I have this in my icinga2.log :

[2019-10-31 11:55:28 +0100] information/FileLogger: 'main-log' started.
[2019-10-31 11:55:28 +0100] information/ApiListener: 'api' started.
[2019-10-31 11:55:28 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2019-10-31 11:55:28 +0100] information/DbConnection: 'ido-mysql' started.
[2019-10-31 11:55:28 +0100] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2019-10-31 11:55:28 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2019-10-31 11:55:28 +0100] information/NotificationComponent: 'notification' started.
[2019-10-31 11:55:28 +0100] information/CheckerComponent: 'checker' started.
[2019-10-31 11:55:28 +0100] information/ConfigItem: Activated all objects.
[2019-10-31 11:55:28 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 8 (schema version: '1.15.0')
[2019-10-31 11:55:28 +0100] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga' in 0.0245249 second(s).
[2019-10-31 11:55:38 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 3.86667/s (232/min 232/5min 232/15min);
[2019-10-31 11:55:38 +0100] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 0, rate: 1.15/s (69/min 69/5min 69/15min);
[2019-10-31 11:55:38 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);

And this in syslog :

systemd[1]: Starting Icinga host/service/network monitoring system...
icinga2[]: [ ] information/cli: Icinga application loader (version: r2.11.2-1)
icinga2[]: [ ] information/cli: Loading configuration file(s).
icinga2[]: [ ] information/ConfigItem: Committing config item(s).
icinga2[]: [ ] information/ApiListener: My API identity: localhost
icinga2[]: [ ] information/ConfigItem: Instantiated 1 FileLogger.
icinga2[]: [ ] information/ConfigItem: Instantiated 187 Dependencies.
icinga2[]: [ ] information/ConfigItem: Instantiated 3 NotificationCommands.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 NotificationComponent.
icinga2[]: [ ] information/ConfigItem: Instantiated 174 Notifications.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 IcingaApplication.
icinga2[]: [ ] information/ConfigItem: Instantiated 4 HostGroups.
icinga2[]: [ ] information/ConfigItem: Instantiated 7 Hosts.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 CheckerComponent.
icinga2[]: [ ] information/ConfigItem: Instantiated 3 Zones.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 Endpoint.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 ApiUser.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 UserGroup.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 ApiListener.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
icinga2[]: [ ] information/ConfigItem: Instantiated 237 CheckCommands.
icinga2[]: [ ] information/ConfigItem: Instantiated 3 TimePeriods.
icinga2[]: [ ] information/ConfigItem: Instantiated 1 User.
icinga2[]: [ ] information/ConfigItem: Instantiated 158 Services.
icinga2[]: [ ] information/ConfigItem: Instantiated 3 ServiceGroups.
icinga2[]: [ ] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
icinga2[]: [ ] information/cli: Closing console log.
systemd[1]: Started Icinga host/service/network monitoring system.

So, any idea ?

rsx · October 31, 2019, 11:27am

Logon to your db and run:

use icinga2;
select status_update_time from icinga_programstatus;

If this is not up to date you’ll get this error. Hence, you’ll need to find out why it is not updated by icinga2.

gaboo · October 31, 2019, 2:26pm

Dang, you got it on first try, thank you very much!

There was two records in this table, which I assume was wrong :

mysql> select status_update_time from icinga_programstatus;
+---------------------+
| status_update_time  |
+---------------------+
| 2014-07-23 17:56:38 |
| 2019-10-31 15:22:14 |
+---------------------+

I deleted them both, restarted the icinga2 daemon and it’s now fixed

maltesandre · November 20, 2019, 6:59am

A query, how did you delete the update_time tables?

Thank you

Regards

rsx · November 20, 2019, 7:01am

Try

delete from icinga_programstatus;

Solkren · January 13, 2020, 10:34am

This issue also could be caused by changing instance_name in IDO feature settings, icingaweb reads only first field from mysql table, if instance_name changed somehow and you writing to another instance_id.

to check instances try this SQL command from DB console:

select * from icinga_instances;

It will show you a list of instances known for the DB
Following SQL will show you additional details about active instance:

select instance_id,program_version,is_currently_running,endpoint_name,status_update_time from icinga_programstatus;

To set correct instance_id for Icinga2 IDO feature, you need set instance_name option in ido-mysq.conf identic to select * from icinga_instances limit 1; and restart Icinga2 master, WARNING if you have a big installation it can cause a delay during restart, because DB will overwrite all services/hosts with new instance_id (see tables like select * from icinga_services limit 1\G;, they all contain this instance_id option)