information/IdoMysqlConnection: Pending queries

Icinga 2
,
1

Hello,

I’m observing “Pending queries” messages on the both master whenever they take over the IDO ownership. I don’t understand why.

I’ve “enable_ha = true” on both masters. I’ve “ido” check setup to be assigned to both masters but for some reason it only shows up on the config master. It should apply to secondary master which does take the IDO ownership. See the logs below. Checking on the IDO DB. it does show that only secondary is connected. WhenI stop “icinga2” service on the secondary, config master takes over and DB shows that it’s tje only one connected. However it will spit out “Pending queries” messaeges.

Have someone seen this before? Please advise, Thanks

Config master

[2025-03-07 23:46:03 +0000] information/IdoMysqlConnection: Pending queries: 15 (Input: 4/s; Output: 2/s)
[2025-03-07 23:47:33 +0000] information/DbConnection: Pausing IDO connection: ido-mysql
[2025-03-07 23:47:33 +0000] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga'.
[2025-03-07 23:47:33 +0000] information/IdoMysqlConnection: 'ido-mysql' paused.

Secondary master

[2025-03-07 23:47:36 +0000] information/DbConnection: Pausing IDO connection: ido-mysql
[2025-03-07 23:47:36 +0000] information/IdoMysqlConnection: 'ido-mysql' paused.
[2025-03-07 23:47:36 +0000] information/DbConnection: 'ido-mysql' stopped.
[2025-03-07 23:47:36 +0000] information/DbConnection: 'ido-mysql' started.
[2025-03-07 23:47:36 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2025-03-07 23:47:36 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
[2025-03-07 23:47:36 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 2.98446s ago (< failover timeout of 30s). Retrying.
[2025-03-07 23:47:46 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 12.9825s ago (< failover timeout of 30s). Retrying.
[2025-03-07 23:47:56 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 22.9813s ago (< failover timeout of 30s). Retrying.
[2025-03-07 23:48:06 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 32.9827s ago. Taking over 'ido-mysql' in HA zone 'master'.
[2025-03-07 23:48:06 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.15.1')
[2025-03-07 23:48:07 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga' in 0.096422 second(s).
[2025-03-07 23:48:16 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 8/s; Output: 8/s)
[2025-03-07 23:53:16 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 4/s; Output: 3/s)
  • Version used (icinga2 --version)
# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.5-1)

Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 24.04.2 LTS (Noble Numbat)
  Kernel: Linux
  Kernel version: 6.8.0-54-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 13.3.0
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 3.0.13 30 Jan 2024

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
  • Enabled features (icinga2 feature list)
# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 journald livestatus notification opentsdb perfdata syslog
Enabled features: api checker ido-mysql mainlog
  • Icinga Web 2 version and modules (System - About)
IcingaWeb2 2.12-2
  • Config validation (icinga2 daemon -C)
# icinga2 daemon -C
[2025-03-08 00:41:15 +0000] information/cli: Icinga application loader (version: r2.14.5-1)
[2025-03-08 00:41:15 +0000] information/cli: Loading configuration file(s).
[2025-03-08 00:41:15 +0000] information/ConfigItem: Committing config item(s).
[2025-03-08 00:41:15 +0000] information/ApiListener: My API identity: m
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/notifications.conf: 23:1-23:48) for type 'Notification' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/notifications.conf: 11:1-11:45) for type 'Notification' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/zones.d/global-templates/services.conf: 35:1-35:21) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'ssh' (in /etc/icinga2/zones.d/global-templates/services.conf: 49:1-49:19) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-templates/services.conf: 59:1-59:65) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-templates/services.conf: 67:1-67:53) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 6 Hosts.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 4 Zones.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 6 Endpoints.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 User.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 246 CheckCommands.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 18 Services.
[2025-03-08 00:41:15 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-03-08 00:41:15 +0000] information/cli: Finished validating the configuration file(s).


apply Service "icinga2-ido-mysql" {
  import "generic-service"
  check_command = "ido"

  vars.ido_type = "IdoMysqlConnection"
  vars.ido_name = "ido-mysql"
  assign where ( host.name == "m" || host.name == "01")
}
2

I’ve another cluster running - r2.14.3-1. There is no “enable_ha = true” and '“ido” check is applied to both masters. The master check shows the result as “paused” and the secondary shows it as “Connected”

3

So the issue appears not just related to “ido” check but other check such as “icinga” only applied to the config master and not to the secondary master.

Any idea what could be wrong here?

Regarding “Pending queries” messages , can someone confirm if they see that in icinga2.log? If these queries are currently on the track of being executed, then showing them as “Pending” could be mistaken and add confusion.

4

Ok the problem with the “ido” and “icinga” check was a typo in the secondary master FQDN hostname. It is fixed now.

So issue now I’ve with the “ido” check result reported in Icingaweb2 ( 2.12.2) which doesn’t seem match with actually reported in Icinga2.

Config master (icingam-002)

[2025-03-14 20:38:34 +0000] information/DbConnection: Pausing IDO connection: ido-mysql
[2025-03-14 20:38:34 +0000] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga'.
[2025-03-14 20:38:34 +0000] information/IdoMysqlConnection: 'ido-mysql' paused.

Secondary master (icingam-003)

[2025-03-14 21:49:04 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
[2025-03-14 21:49:04 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.15.1')
[2025-03-14 21:49:04 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga' in 0.217954 second(s).
[2025-03-14 21:49:14 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 6/s; Output: 13/s)
[2025-03-14 21:54:14 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 4/s; Output: 4/s)
[2025-03-14 21:59:14 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 4/s; Output: 5/s)


icingacli monitoring list services --host=icingam-002.x.com --service=icinga2-ido-mysql --columns "service_output" --format='$service_output$'
Connected to the database server (Schema version: '1.15.1'). Queries per second: 3.667 Pending queries: 0.000. Last failover: 2025-03-14 20:39:04 +0000.

 icingacli monitoring list services --host=icingam-003.x.com --service=icinga2-ido-mysql --columns "service_output" --format='$service_output$'
DB IDO connection is temporarily disabled on this cluster instance.

icingam-002 is supposed to have “paused” state and icingam-003 (Active endpoint) should be “connected”

From Active DB host

mysql> SELECT status_update_time, endpoint_name FROM icinga_programstatus;
+---------------------+------------------------------------------+
| status_update_time  | endpoint_name                            |
+---------------------+------------------------------------------+
| 2025-03-14 23:33:54 | icingam-003.x.com |
+---------------------+------------------------------------------+
1 row in set (0.00 sec)

From Inactive DB host

mysql>  SELECT status_update_time, endpoint_name FROM icinga_programstatus;
+---------------------+------------------------------------------+
| status_update_time  | endpoint_name                            |
+---------------------+------------------------------------------+
| 2025-03-14 21:48:54 | icingam-003.x.com |
+---------------------+------------------------------------------+
1 row in set (0.00 sec)