Hello,
I’m observing “Pending queries” messages on the both master whenever they take over the IDO ownership. I don’t understand why.
I’ve “enable_ha = true” on both masters. I’ve “ido” check setup to be assigned to both masters but for some reason it only shows up on the config master. It should apply to secondary master which does take the IDO ownership. See the logs below. Checking on the IDO DB. it does show that only secondary is connected. WhenI stop “icinga2” service on the secondary, config master takes over and DB shows that it’s tje only one connected. However it will spit out “Pending queries” messaeges.
Have someone seen this before? Please advise, Thanks
Config master
[2025-03-07 23:46:03 +0000] information/IdoMysqlConnection: Pending queries: 15 (Input: 4/s; Output: 2/s)
[2025-03-07 23:47:33 +0000] information/DbConnection: Pausing IDO connection: ido-mysql
[2025-03-07 23:47:33 +0000] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga'.
[2025-03-07 23:47:33 +0000] information/IdoMysqlConnection: 'ido-mysql' paused.
Secondary master
[2025-03-07 23:47:36 +0000] information/DbConnection: Pausing IDO connection: ido-mysql
[2025-03-07 23:47:36 +0000] information/IdoMysqlConnection: 'ido-mysql' paused.
[2025-03-07 23:47:36 +0000] information/DbConnection: 'ido-mysql' stopped.
[2025-03-07 23:47:36 +0000] information/DbConnection: 'ido-mysql' started.
[2025-03-07 23:47:36 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2025-03-07 23:47:36 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
[2025-03-07 23:47:36 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 2.98446s ago (< failover timeout of 30s). Retrying.
[2025-03-07 23:47:46 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 12.9825s ago (< failover timeout of 30s). Retrying.
[2025-03-07 23:47:56 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 22.9813s ago (< failover timeout of 30s). Retrying.
[2025-03-07 23:48:06 +0000] information/IdoMysqlConnection: Last update by endpoint '01' was 32.9827s ago. Taking over 'ido-mysql' in HA zone 'master'.
[2025-03-07 23:48:06 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.15.1')
[2025-03-07 23:48:07 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga' in 0.096422 second(s).
[2025-03-07 23:48:16 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 8/s; Output: 8/s)
[2025-03-07 23:53:16 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 4/s; Output: 3/s)
- Version used (
icinga2 --version
)
# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.5-1)
Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Ubuntu
Platform version: 24.04.2 LTS (Noble Numbat)
Kernel: Linux
Kernel version: 6.8.0-54-generic
Architecture: x86_64
Build information:
Compiler: GNU 13.3.0
Build host: runner-hh8q3bz2-project-575-concurrent-0
OpenSSL version: OpenSSL 3.0.13 30 Jan 2024
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
- Enabled features (
icinga2 feature list
)
# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 journald livestatus notification opentsdb perfdata syslog
Enabled features: api checker ido-mysql mainlog
- Icinga Web 2 version and modules (System - About)
IcingaWeb2 2.12-2
- Config validation (
icinga2 daemon -C
)
# icinga2 daemon -C
[2025-03-08 00:41:15 +0000] information/cli: Icinga application loader (version: r2.14.5-1)
[2025-03-08 00:41:15 +0000] information/cli: Loading configuration file(s).
[2025-03-08 00:41:15 +0000] information/ConfigItem: Committing config item(s).
[2025-03-08 00:41:15 +0000] information/ApiListener: My API identity: m
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/notifications.conf: 23:1-23:48) for type 'Notification' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/notifications.conf: 11:1-11:45) for type 'Notification' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/zones.d/global-templates/services.conf: 35:1-35:21) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'ssh' (in /etc/icinga2/zones.d/global-templates/services.conf: 49:1-49:19) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-templates/services.conf: 59:1-59:65) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-templates/services.conf: 67:1-67:53) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 6 Hosts.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 4 Zones.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 6 Endpoints.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 User.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 246 CheckCommands.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 18 Services.
[2025-03-08 00:41:15 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-03-08 00:41:15 +0000] information/cli: Finished validating the configuration file(s).
apply Service "icinga2-ido-mysql" {
import "generic-service"
check_command = "ido"
vars.ido_type = "IdoMysqlConnection"
vars.ido_name = "ido-mysql"
assign where ( host.name == "m" || host.name == "01")
}