IDO - Mysql 8 - replica error

Icinga 2
1

I’ve setup master-master replication on 2x Mysql 8 hosts. When I start replicas/slaves, I see this error. Have anyone see this before and know the cause of it? I understand error 1032 can be skipped in this case but I would like to know why it exists at the first place. Thanks

2025-03-13T23:47:48.194429Z 562 [ERROR] [MY-010584] [Repl] Replica SQL for channel '': Worker 1 failed executing transaction 'ANONYMOUS' at source log mysql-bin.000001, end_log_pos 5583912; Could not execute Delete_rows event on table icinga.icinga_programstatus; Can't find record in 'icinga_programstatus', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's source log mysql-bin.000001, end_log_pos 5583912, Error_code: MY-001032

# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.5-1)

Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 24.04.2 LTS (Noble Numbat)
  Kernel: Linux
  Kernel version: 6.8.0-54-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 13.3.0
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 3.0.13 30 Jan 2024

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 journald livestatus notification opentsdb perfdata syslog
Enabled features: api checker ido-mysql mainlog

# icinga2 daemon -C
[2025-03-08 00:41:15 +0000] information/cli: Icinga application loader (version: r2.14.5-1)
[2025-03-08 00:41:15 +0000] information/cli: Loading configuration file(s).
[2025-03-08 00:41:15 +0000] information/ConfigItem: Committing config item(s).
[2025-03-08 00:41:15 +0000] information/ApiListener: My API identity: m
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/notifications.conf: 23:1-23:48) for type 'Notification' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/zones.d/global-templates/notifications.conf: 11:1-11:45) for type 'Notification' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/zones.d/global-templates/services.conf: 35:1-35:21) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule 'ssh' (in /etc/icinga2/zones.d/global-templates/services.conf: 49:1-49:19) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-templates/services.conf: 59:1-59:65) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-templates/services.conf: 67:1-67:53) for type 'Service' does not match anywhere!
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 6 Hosts.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 4 Zones.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 6 Endpoints.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 User.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 246 CheckCommands.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2025-03-08 00:41:15 +0000] information/ConfigItem: Instantiated 18 Services.
[2025-03-08 00:41:15 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-03-08 00:41:15 +0000] information/cli: Finished validating the configuration file(s).
2

Could you please describe your MySQL server setup a bit more in detail? If I got it right, you have a master-master replication on top and replicas below the masters?

Nevertheless, I experienced such errors in the past with master-master setups if writes happen to different SQL servers, but not with Icinga 2. My personal suggestion would be to avoid master-master setups and use a master-backup concept, which may not automagically fall over, but is IMO way more stable.

In case your curios, compare the binlog or otherwise acknowledge that the MySQL server had a bad day and roll back the log and restart the synchronization.

1 Like
3

I’ve “enable_ha” enabled on both masters. I’ve 2 nodes running mysql 8 on them Only one of them (FQDN hóstiame) is added in the IDO config on both masters. I set up “icinga” and “icingaweb2” databases as per documentation. It worked as expected as one of the masters will write to one DB host.

For IDO HA, I enabled replica on both DB servers so each server is a master and a replica of another. That’s when this 1032 error was seen on the inactive DB host replica. At that time, the table "icinga_programstatus’ was empty ( because it was inactive and not updated by icinga). So mysql failed to delete it during replica sync with master throwing 1032 error. It does look like the replication is working though.