Monitoring Backend Icinga not running

Icinga Web
1
  • Icinga Web 2 version
Icinga Web 2 Version 	2.11.1
Git commit 	eecf8d9934cbf30ec063cea4a525b26bfd2bf99a
PHP Version 	7.3.29
Git commit date 	2022-07-06

Loaded Libraries
icinga/icinga-php-thirdparty 	0.11.0
icinga/icinga-php-library 	0.9.1
  • Used modules and their versions (System - About)
Loaded Modules
director 		master 	Configure
doc 		2.11.1 	Configure
grafana 		1.4.2 	Configure
incubator 		0.17.0 	Configure
monitoring 		2.11.1 	Configure

  • Web browser used
    Firefox

  • Icinga 2 version used (icinga2 --version)
    icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.4-1)

  • PHP version used (php --version)

# /opt/rh/rh-php73/root/usr/bin/php -version
PHP 7.3.29 (cli) (built: Aug  3 2021 12:26:40) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.29, Copyright (c) 1998-2018 Zend Technologies
  • Server operating system and version
System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.71.1.el7.x86_64
  Architecture: x86_64

Currently I see the following message: 'Backend icinga is not running ’
Everything looks good.
All systemctl status are green

# icinga2 daemon -C
[2022-07-22 10:24:32 +0200] information/cli: Icinga application loader (version: r2.13.4-1)
[2022-07-22 10:24:32 +0200] information/cli: Loading configuration file(s).
[2022-07-22 10:24:32 +0200] information/ConfigItem: Committing config item(s).
[2022-07-22 10:24:33 +0200] information/ApiListener: My API identity: myservr.fqdn.com
[2022-07-22 10:24:33 +0200] warning/ApplyRule: Apply rule 'by_ssh_check_proc_dsmcad' (in /var/lib/icinga2/api/packages/director/f3f5aa7b-32bb-451d-ba3a-164a11bc743d/zones.d/director-global/service_apply.conf: 1:0-1:39) for type 'Service' does not match anywhere!
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 6 UserGroups.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 4 TimePeriods.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 9 Users.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1490 Services.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 4 ServiceGroups.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 4 NotificationCommands.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 3698 Notifications.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 97 Hosts.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 24 HostGroups.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 23 Comments.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 250 CheckCommands.
[2022-07-22 10:24:33 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2022-07-22 10:24:33 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-07-22 10:24:33 +0200] information/cli: Finished validating the configuration file(s).

# cat /etc/icinga2/features-enabled/ido-mysql.conf
/**
 * The db_ido_mysql library implements IDO functionality
 * for MySQL.
 */

/* library "db_ido_mysql" */

object IdoMysqlConnection "ido-mysql" {
  user = "myuser"
  password = "<password>"
  host = "localhost"
  database = "icinga"
}

MariaDB [icinga]> select instance_id,program_version,is_currently_running,endpoint_name,status_update_time from icinga_programstatus;
+-------------+-----------------+----------------------+------------------+---------------------+
| instance_id | program_version | is_currently_running | endpoint_name    | status_update_time  |
+-------------+-----------------+----------------------+------------------+---------------------+
|           1 | r2.13.4-1       |                    0 | myservr.fqdn.com | 2022-07-22 07:12:49 |
+-------------+-----------------+----------------------+------------------+---------------------+
1 row in set (0.000 sec)

MariaDB [icinga]> select * from icinga_instances;
+-------------+---------------+----------------------+
| instance_id | instance_name | instance_description |
+-------------+---------------+----------------------+
|           1 | default       |                      |
+-------------+---------------+----------------------+
1 row in set (0.001 sec)

My last activities over the past 2 days:
I updated Icinga2, Icingaweb2
changed from rh-php71 to rh-php73
updated the sql schemas incrementally
and today migrated from mariadb 5.5 to mariadb 10.5

Any assistance is greatly appreciated.

Kind regards
Frank

2

I found something in /var/log/icinga2/icinga2.log

Context:
        (0) Reconnecting to MySQL IDO database 'ido-mysql'

[2022-07-22 14:10:29 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2022-07-22 14:10:39 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Can't connect to local MySQL server through socket '/var/lib/mysq
l/mysql.sock' (2 "No such file or directory")"

Where is the configuration for that MySQL socket?
I am under the assumption that I changed all *mysql.default_socket to my desired directory.

# grep default_socket /etc/opt/rh/rh-php73/php.ini
default_socket_timeout = 60
pdo_mysql.default_socket=/data/db/mysql/mysql.sock
mysqli.default_socket = /data/db/mysql/mysql.sock

Kind regards
Frank

3

You can set the socket path in /etc/icinga2/features-enabled/ido-mysql.conf for the IdoMysqlConnection object.

socket_path

4

Exactly what I was searching for, thank you.
After adding the attribute socket_path I get a permission denied:

[2022-07-25 09:24:49 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2022-07-25 09:24:59 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'USER' on 'localhost:3306' failed: "Can't connect to local MySQL server through socket '/data/db/mysql/' (13 "Permission denied")"

Manually on the command line it works, and the verification form Icinga frontend succeeds as well.

Okay, nevermind.
The string has to be the socket file, not just the path.
Thanks again, this fixed the problem.
Do you by chance have more information on the deprecation of this functionality? I read the warning but cannot find any information how it is supposed to work in the future.

5

Probably because ido will be replaced by icingadb. :slight_smile: