Trouble installing "There is currently no icinga instance writing to the IDO."

samwarez · July 29, 2019, 9:11pm

I have made it to the “Monitoring IDO Resource” part of the setup. But I receive this error whenever I click “Validate Configuration”

There is currently no icinga instance writing to the IDO. Make sure that a icinga instance is configured and able to write to the IDO.
Validation Log

Connection to icingaido as idouser on localhost: successful
have_ssl: YES
protocol_version: 10
version: 8.0.17
version_compile_os: Linux
Ssl_cipher:

From what I can tell this is a common issue and is caused by improper permissions on the features-available directory, however I have since fixed those permissions and still have the same issue.

[root@icinga features-available]# ls -lah
total 72K
drwxr-x—. 2 icinga icinga 4.0K Jul 25 15:45 .
drwxr-x—. 7 root icinga 160 Jul 25 15:35 …
-rw-rw----. 1 icinga icinga 183 May 23 06:22 api.conf
-rw-rw----. 1 icinga icinga 112 May 23 06:22 checker.conf
-rw-rw----. 1 icinga icinga 140 May 23 06:22 command.conf
-rw-rw----. 1 icinga icinga 159 May 23 06:22 compatlog.conf
-rw-rw----. 1 icinga icinga 235 May 23 06:22 debuglog.conf
-rw-rw----. 1 icinga icinga 193 May 23 06:22 elasticsearch.conf
-rw-rw----. 1 icinga icinga 206 May 23 06:22 gelf.conf
-rw-rw----. 1 icinga icinga 190 May 23 06:22 graphite.conf
-rw-rw----. 1 icinga icinga 302 Jul 25 15:44 ido-mysql.conf
-rw-rw----. 1 icinga icinga 560 May 23 06:22 influxdb.conf
-rw-rw----. 1 icinga icinga 127 May 23 06:22 livestatus.conf
-rw-rw----. 1 icinga icinga 159 May 23 06:22 mainlog.conf
-rw-rw----. 1 icinga icinga 129 May 23 06:22 notification.conf
-rw-rw----. 1 icinga icinga 190 May 23 06:22 opentsdb.conf
-rw-rw----. 1 icinga icinga 146 May 23 06:22 perfdata.conf
-rw-rw----. 1 icinga icinga 225 May 23 06:22 statusdata.conf
-rw-rw----. 1 icinga icinga 127 May 23 06:22 syslog.conf

My ido-mysql.conf:

[root@icinga features-available]# cat /etc/icinga2/features-enabled/ido-mysql.conf
/**
* The IdoMysqlConnection type implements MySQL support
* for DB IDO.
*/
object IdoMysqlConnection "ido-mysql" {
        host = "localhost"
        //port = 3306
        user = "idouser"
        password = "***"
        database = "icingaido"

        cleanup = {
                downtimehistory_age = 48h
                contactnotifications_age = 31d
        }
}

And icinga2 is running:
> systemctl status icinga2 -l

● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2019-07-25 15:44:36 MDT; 3 days ago
  Process: 26443 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/sysconfig/icinga2 (code=exited, status=0/SUCCESS)
 Main PID: 26451 (icinga2)

My OS version:

centos-release-7-6.1810.2.el7.centos.x86_64

Any help would be greatly appreciated.

dnsmichi · July 30, 2019, 9:07am

Look into the warning, it says something about that Icinga 2 isn’t actively writing to the IDO database backend. Since you’ve posted the features-enabled configuration already, your next step should be to verify that the connection worked. You can do so within the icinga2.log in /var/log/icinga2.

Cheers,
Michael

samwarez · July 31, 2019, 3:52pm

Thank you, looking at the logs shows its was not so much an issue with IDO - Database communication (as the error indicated) but missing nagios plugins. Some how nagios-plugins-all never got installed. installed them and it validated. Might I suggest file validation checks just to make sure all needing files are in the right places?

dnsmichi · August 1, 2019, 8:02am

Wondering why missing plugins would cause the core not to write anything to the IDO database. Within the icinga2.log file you did see entries for successful connects and writes, right?

Problem is more or less, that the plugins are not necessarily run on the same host where Icinga Web 2 is running on. You might just setup web on a standalone server, not related to the core itself. That being said, the installer for web doesn’t know much about the core - except for the resources you’re defining as backend (IDO, REST API).

Cheers,
Michael

samwarez · August 1, 2019, 7:13pm

the Log file was filled with nothing but errors about the plugins. never saw anything regarding IDO

dnsmichi · August 1, 2019, 7:46pm

Try using grep and friends when analysing the log files. This allows to filter for specific strings and words.

MrMas · October 29, 2019, 10:17pm

I’m having the same issue. I initially was fighting with issues with authentication. At some point, the errors switched to these IDO errors (perhaps after I changed my MySQL icinga user passwd). It looks like icinga is having no issues using the IDO connection. Here is the latest IDO-related message:
[2019-10-29 16:10:51 -0600] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 2, rate: 3.68333/s (221/min 1121/5min 2108/15min);

I do see a place where icinga noticed that mysql stopped running but it seemed to have recovered just fine:

[2019-10-29 15:48:02 -0600] critical/IdoMysqlConnection: Error "MySQL server has gone away" when executing query "UPDATE icinga_programstatus SET active_host
_checks_enabled = '1',  active_service_checks_enabled = '1',  daemon_mode = '1',  endpoint_name = 'raspberrypi',  event_handlers_enabled = '1',  flap_detecti
on_enabled = '1',  instance_id = 1,  is_currently_running = '1',  last_command_check = FROM_UNIXTIME(1572385681),  notifications_enabled = '1',  passive_host
_checks_enabled = '1',  passive_service_checks_enabled = '1',  process_id = '808',  process_performance_data = '1',  program_start_time = FROM_UNIXTIME(15723
85334),  program_version = 'r2.10.3-1',  status_update_time = FROM_UNIXTIME(1572385681) WHERE instance_id = 1;COMMIT;BEGIN;DELETE FROM icinga_runtimevariable
s WHERE instance_id = 1;INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '29');INSERT INTO icinga_runtimevar
iables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '29');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VAL
UES (1, 'total_hosts', '15');INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '15');COMMIT;BEGIN"
[2019-10-29 15:48:02 -0600] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2019-10-29 15:48:11 -0600] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2019-10-29 15:48:11 -0600] information/IdoMysqlConnection: Finished reconnecting to MySQL IDO database in 0.173366 second(s).
...
[2019-10-29 15:50:11 -0600] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 2, rate: 3.68333/s (221/min 1117/5min 1763/15min);
[2019-10-29 15:50:50 -0600] information/DbConnection: 'ido-mysql' started.
[2019-10-29 15:50:50 -0600] information/DbConnection: Resuming IDO connection: ido-mysql
[2019-10-29 15:50:50 -0600] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2019-10-29 15:50:50 -0600] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2019-10-29 15:50:50 -0600] information/IdoMysqlConnection: Finished reconnecting to MySQL IDO database in 0.119915 second(s).
[2019-10-29 15:50:59 -0600] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 0, rate: 0.516667/s (31/min 31/5min 31/15min);

Thoughts?

MrMas · October 30, 2019, 3:15pm

I was using the wrong database user. I logged into my database and ran
SHOW GRANTS for 'icinga'@'localhost'
SHOW GRANTS for 'icinga2'@'localhost'
SHOW GRANTS for 'icingaweb'@'localhost'
SHOW GRANTS for 'icingaweb2'@'localhost'
to discover which databases I had created under which users.

jwaghcinga · January 12, 2020, 2:03pm

Hi Michael,

I am getting below logs, can you please help me on this…

root@localhost icinga2]# tailf /var/log/icinga2/icinga2.log
[2020-01-12 11:39:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2020-01-12 11:44:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2020-01-12 11:49:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2020-01-12 11:54:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2020-01-12 11:55:47 +0530] information/Notification: Sending reminder ‘Problem’ notification ‘localhost.localdomain!http!mail-icingaadmin’ for user ‘icingaadmin’
[2020-01-12 11:55:47 +0530] information/Notification: Completed sending ‘Problem’ notification ‘localhost.localdomain!http!mail-icingaadmin’ for checkable ‘localhost.localdomain!http’ and user ‘icingaadmin’ using command ‘mail-service-notification’.
[2020-01-12 11:55:57 +0530] information/Notification: Sending reminder ‘Problem’ notification ‘localhost.localdomain!procs!mail-icingaadmin’ for user ‘icingaadmin’
[2020-01-12 11:55:57 +0530] information/Notification: Completed sending ‘Problem’ notification ‘localhost.localdomain!procs!mail-icingaadmin’ for checkable ‘localhost.localdomain!procs’ and user ‘icingaadmin’ using command ‘mail-service-notification’.
[2020-01-12 11:59:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2020-01-12 12:04:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’

[2020-01-12 12:09:28 +0530] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’