Icingadb crashing with error invalid connection#012can't perform "INSERT INTO \"state history\

Icinga 2 Icinga DB
1

icingadb has been running fine for several months. It started failing today (Jul 29 2025) with this error. I restart icingadb but it fails with the same error within a few minutes

I did a search and tried this command ‘icingadb-redis-cli flushall’ which returned OK but it didn’t fix the problem.

I believe I ran the icingadb-migrate command but I am not sure how to check.

Here’s the error message from /var/log/message. I added line endings since it is one long string

I think there may be a duplicate key issue but I am not sure how this happened.

Jul 29 12:39:50 bld-lvn-bca-60 icingadb[3415113]: invalid connection#012can't perform "INSERT 
INTO \"state_history\" (\"event_time\", \"hard_state\", \"previous_hard_state\", \"long_output\", 
\"scheduling_source\", \"check_source\", \"environment_id\", \"host_id\", \"service_id\", 
\"soft_state\", \"max_check_attempts\", \"endpoint_id\", \"id\", \"state_type\", 
\"previous_soft_state\", \"check_attempt\", \"output\", \"object_type\") VALUES 
(:event_time,:hard_state,:previous_hard_state,:long_output,:scheduling_source,:check_source,:
environment_id,:host_id,:service_id,:soft_state,:max_check_attempts,:endpoint_id,:id,:state_type,
:previous_soft_state,:check_attempt,:output,:object_type) ON DUPLICATE KEY UPDATE \"id\" = 
VALUES(\"id\")"#012github.com/icinga/icinga-go-
library/database.CantPerformQuery#012#011github.com/icinga/icinga-go-
library@v0.4.0/database/utils.go:16#012github.com/icinga/icinga-go-library/database.
(*DB).NamedBulkExec.func1.
(*DB).NamedBulkExec.func1.1.2.1#012#011github.com/icinga/icinga-go-
library@v0.4.0/database/db.go:535#012github.com/icinga/icinga-go-
library/retry.WithBackoff#012#011github.com/icinga/icinga-go-
library@v0.4.0/retry/retry.go:65#012github.com/icinga/icinga-go-library/database.
(*DB).NamedBulkExec.func1.
(*DB).NamedBulkExec.func1.1.2#012#011github.com/icinga/icinga-go-
library@v0.4.0/database/db.go:530#012golang.org/x/sync/errgroup.
(*Group).Go.func1#012#011golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78#012runtime.go
exit#012#011runtime/asm_amd64.s:1700#012retry deadline 
exceeded#012github.com/icinga/icinga-go-
library/retry.WithBackoff#012#011github.com/icinga/icinga-go-
library@v0.4.0/retry/retry.go:100#012github.com/icinga/icinga-go-library/database.
(*DB).NamedBulkExec.func1.
(*DB).NamedBulkExec.func1.1.2#012#011github.com/icinga/icinga-go-
library@v0.4.0/database/db.go:530#012golang.org/x/sync/errgroup.
(*Group).Go.func1#012#011golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78#012runtime.go
exit#012#011runtime/asm_amd64.s:1700

Icinga Web 2 Version	2.12.4
Git commit	219b11789beb0484b7993fa9ea838edd211ca5b2
PHP Version	7.2.24
Git commit date	2025-03-26

Loaded Libraries
icinga/icinga-php-library	    0.14.2
icinga/icinga-php-thirdparty	0.12.1

Loaded Modules
director		1.11.4	Configure
doc		        2.12.4	Configure
icingadb		1.1.3	Configure
graphite	    1.2.4	Configure
incubator		0.22.0	Configure
migrate	    	2.12.4	Configure
monitoring    	2.12.4	Configure
setup		    2.12.4	Configure
2

Thanks for posting.

First, please tell us which Icinga DB version you are using. Furthermore, the Icinga 2 version and relational database server are also helpful.

Based on your stack trace - the github.com/icinga/icinga-go-library@v0.4.0 part -, I would assume you are using Icinga DB v1.2.1 - which is a bit outdated and which was superseded by version 1.3.0 and 1.4.0. Please consider upgrading.

On first glance, the error implies a SQL error.

Please provide a bit more context by posting more of the logs - let’s say the last ten lines before the crash or even more, if there are some log messages with an ERROR severity. Since you are using an older Icinga DB version, you have to set --output verbose, as documented here. Btw, this is no longer necessary for errors since Icinga DB 1.4.0.

Are you using an HA setup or is this a single node setup?

The icingadb-migrate command migrates an Icinga 2 IDO database to Icinga DB. Schema upgrades must be applied manually.

3

I did run the migrate command, it was just a while ago

I still have IDO running. Worst case I could remove icingadb database and migrate again.

This is a single node setup, not an HA setup

I thought I had included the relevant version information from System → About. Here’s a list of the packages

icinga2-2.14.5-1.el8.x86_64
icinga2-bin-2.14.5-1.el8.x86_64
icinga2-common-2.14.5-1.el8.x86_64
icinga2-ido-mysql-2.14.5-1.el8.x86_64
icingacli-2.12.4-1.el8.noarch
icingadb-1.2.1-1.el8.x86_64
icingadb-redis-7.2.7-1.el8.x86_64
icingadb-web-1.1.3-1.el8.noarch
icinga-director-1.11.4-1.el8.noarch
icinga-director-daemon-1.11.4-1.el8.noarch
icinga-director-php-1.11.4-1.el8.noarch
icinga-director-web-1.11.4-1.el8.noarch
icinga-graphite-1.2.4-1.el8.noarch
icinga-graphite-web-1.2.4-1.el8.noarch
icinga-l10n-1.3.0-1.el8.noarch
icinga-php-incubator-0.22.0-1.1.el8.noarch
icinga-php-library-0.14.2-1.el8.noarch
icinga-php-thirdparty-0.12.1-1.el8.noarch
icingaweb2-2.12.4-1.el8.noarch
icingaweb2-common-2.12.4-1.el8.noarch

and the mariadb packages

mariadb-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-backup-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-common-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-connector-c-3.1.11-2.el8_3.x86_64
mariadb-connector-c-config-3.1.11-2.el8_3.noarch
mariadb-connector-c-devel-3.1.11-2.el8_3.x86_64
mariadb-connector-odbc-3.1.12-1.el8.x86_64
mariadb-devel-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-errmsg-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-gssapi-server-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-server-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-server-utils-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64
mariadb-test-10.5.27-1.module+el8.10.0+22630+cacda3f1.x86_64

You asked for more log output. I am not sure how much you want

journalctl --unit icingadb.service --output verbose > /tmp/output
Wed 2025-07-30 06:06:05.729791 PDT [s=5584c0767f8b462f9c1afd07c998c664;i=958a8;b=3826bae12bdc4b89b08ab08a0b6b477a;m=47961b8ba33;t=63b2530050c3b;x=a65a09c333a2c6dd]
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=3826bae12bdc4b89b08ab08a0b6b477a
    _MACHINE_ID=ba720aadd5454a2882f9fdb11949fe37
    _HOSTNAME=bld-lvn-bca-60
    _CAP_EFFECTIVE=0
    _TRANSPORT=journal
    SYSLOG_IDENTIFIER=icingadb
    _UID=961
    _GID=958
    _COMM=icingadb
    _EXE=/usr/sbin/icingadb
    _CMDLINE=/usr/sbin/icingadb --config /etc/icingadb/config.yml
    _SYSTEMD_CGROUP=/system.slice/icingadb.service
    _SYSTEMD_UNIT=icingadb.service
    PRIORITY=4
    MESSAGE=database: Can't execute query. Retrying
    _PID=3748055
    _SYSTEMD_INVOCATION_ID=eb2d690f5e444e0ebce746b5df83ff98
    ICINGADB_ERROR=can't perform "INSERT INTO \"state_history\" (\"object_type\", \"previous_hard_state\", \"long_output\", \"id\", \"event_time\", \"max_check_attempts\", \"check_source\"
, \"scheduling_source\", \"environment_id\", \"endpoint_id\", \"service_id\", \"hard_state\", \"previous_soft_state\", \"check_attempt\", \"output\", \"host_id\", \"state_type\", \"soft_st
ate\") VALUES (:object_type,:previous_hard_state,:long_output,:id,:event_time,:max_check_attempts,:check_source,:scheduling_source,:environment_id,:endpoint_id,:service_id,:hard_state,:pre
vious_soft_state,:check_attempt,:output,:host_id,:state_type,:soft_state) ON DUPLICATE KEY UPDATE \"id\" = VALUES(\"id\")": invalid connection
    _SOURCE_REALTIME_TIMESTAMP=1753880765729791
Wed 2025-07-30 06:06:06.263335 PDT [s=5584c0767f8b462f9c1afd07c998c664;i=958a9;b=3826bae12bdc4b89b08ab08a0b6b477a;m=47961c0de2e;t=63b25300d3037;x=eb90f0e0a1a2a52b]
    PRIORITY=6
    _SYSTEMD_SLICE=system.slice
    _BOOT_ID=3826bae12bdc4b89b08ab08a0b6b477a
    _MACHINE_ID=ba720aadd5454a2882f9fdb11949fe37
    _HOSTNAME=bld-lvn-bca-60
    _CAP_EFFECTIVE=0
    _TRANSPORT=journal
    SYSLOG_IDENTIFIER=icingadb
    _UID=961
    _GID=958
    _COMM=icingadb
    _EXE=/usr/sbin/icingadb
    _CMDLINE=/usr/sbin/icingadb --config /etc/icingadb/config.yml
    _SYSTEMD_CGROUP=/system.slice/icingadb.service
    _SYSTEMD_UNIT=icingadb.service
    MESSAGE=heartbeat: Received Icinga heartbeat
    ICINGADB_ENVIRONMENT=df198f3eb95387df0c82e598f0155eddaaeafef5
    _PID=3748055
    _SYSTEMD_INVOCATION_ID=eb2d690f5e444e0ebce746b5df83ff98

This from /var/log/messages from the restart of all the icinga related services, filtered for ‘icingadb’

Jul 30 06:05:35 XXXXX icingadb[3747419]: Starting Icinga DB daemon (v1.2.1)
Jul 30 06:05:35 XXXXX icingadb[3747419]: Connecting to database at 'mysql://icingadb@localhost:3306/icingadb'
Jul 30 06:05:35 XXXXX icingadb[3747419]: Connecting to Redis at 'redis://localhost:6380'
Jul 30 06:05:38 XXXXX icingadb[3747419]: Waiting for Icinga 2 to write into Redis, please make sure you have started Icinga 2 and the Icinga DB feature is enabled
Jul 30 06:05:59 XXXXX systemd[1]: icingadb.service: Succeeded.
Jul 30 06:05:59 XXXXX systemd[1]: icingadb-redis.service: Succeeded.
Jul 30 06:06:04 XXXXX icingadb[3748055]: Starting Icinga DB daemon (v1.2.1)
Jul 30 06:06:04 XXXXX icingadb[3748055]: Connecting to database at 'mysql://icingadb@localhost:3306/icingadb'
Jul 30 06:06:04 XXXXX icingadb[3748055]: Connecting to Redis at 'redis://localhost:6380'
Jul 30 06:06:05 XXXXX icingadb[3748055]: Starting history sync
Jul 30 06:06:05 XXXXX icingadb[3748055]: database: Can't execute query. Retrying
Jul 30 06:06:06 XXXXX icingadb[3748055]: heartbeat: Received Icinga heartbeat
Jul 30 06:06:06 XXXXX icingadb[3748055]: Taking over
Jul 30 06:06:06 XXXXX icingadb[3748055]: Starting config sync
Jul 30 06:06:06 XXXXX icingadb[3748055]: Starting initial state sync
Jul 30 06:06:06 XXXXX icingadb[3748055]: Starting overdue sync
Jul 30 06:06:06 XXXXX icingadb[3748055]: config-sync: Inserting 1 items of type comment
Jul 30 06:06:06 XXXXX icingadb[3748055]: config-sync: Updating 238 items of type host state
Jul 30 06:06:06 XXXXX icingadb[3748055]: Starting config runtime updates sync
Jul 30 06:06:06 XXXXX icingadb[3748055]: config-sync: Finished config sync in 120.335491ms
Jul 30 06:06:06 XXXXX icingadb[3748055]: config-sync: Updating 1880 items of type service state
Jul 30 06:06:06 XXXXX icingadb[3748055]: Starting history retention
Jul 30 06:06:06 XXXXX icingadb[3748055]: Starting state runtime updates sync
Jul 30 06:06:06 XXXXX icingadb[3748055]: config-sync: Finished initial state sync in 278.495901ms
Jul 30 06:06:25 XXXXX icingadb[3748055]: history-sync: Synced 1 acknowledgement history items
Jul 30 06:06:25 XXXXX icingadb[3748055]: history-sync: Synced 37 notification history items
Jul 30 06:08:45 XXXXX icingadb[3748055]: history-sync: Synced 1 notification history items
Jul 30 06:11:06 XXXXX icingadb[3748055]: invalid connection#012can't perform "INSERT INTO \"state_history\" (\"object_type\", \"previous_hard_state\", \"long_output\", \"id\", \"event_time\", \"max_check_attempts\", \"check_source\", \"scheduling_source\", \"environment_id\", \"endpoint_id\", \"service_id\", \"hard_state\", \"previous_soft_state\", \"check_attempt\", \"output\", \"host_id\", \"state_type\", \"soft_state\") VALUES (:object_type,:previous_hard_state,:long_output,:id,:event_time,:max_check_attempts,:check_source,:scheduling_source,:environment_id,:endpoint_id,:service_id,:hard_state,:previous_soft_state,:check_attempt,:output,:host_id,:state_type,:soft_state) ON DUPLICATE KEY UPDATE \"id\" = VALUES(\"id\")"#012github.com/icinga/icinga-go-library/database.CantPerformQuery#012#011github.com/icinga/icinga-go-library@v0.4.0/database/utils.go:16#012github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1#012#011github.com/icinga/icinga-go-library@v0.4.0/database/db.go:535#012github.com/icinga/icinga-go-library/retry.WithBackoff#012#011github.com/icinga/icinga-go-library@v0.4.0/retry/retry.go:65#012github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2#012#011github.com/icinga/icinga-go-library@v0.4.0/database/db.go:530#012golang.org/x/sync/errgroup.(*Group).Go.func1#012#011golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78#012runtime.goexit#012#011runtime/asm_amd64.s:1700#012retry deadline exceeded#012github.com/icinga/icinga-go-library/retry.WithBackoff#012#011github.com/icinga/icinga-go-library@v0.4.0/retry/retry.go:100#012github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2#012#011github.com/icinga/icinga-go-library@v0.4.0/database/db.go:530#012golang.org/x/sync/errgroup.(*Group).Go.func1#012#011golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78#012runtime.goexit#012#011runtime/asm_amd64.s:1700
Jul 30 06:11:06 XXXXX systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Jul 30 06:11:06 XXXXX systemd[1]: icingadb.service: Failed with result 'exit-code'.
4

Thanks for your detailed response.

There seems to be a problem with the MariaDB connection. The crashing error invalid connection also reflects this.

We had a similar issue a while back, but your setups are quite different. However, what did the MariaDB do during these times? Were there outages or schema upgrades? And how are you doing relational database backups? By default, MySQL/MariaDB locks the database during these times (docs).

Even if this might not be the answer you want to hear, I would strongly urge you to upgrade to a recent version. There were some major refactorings in the meantime. Furthermore, some of your Icinga components have known security issues.

5

This is the end of /var/log/mariadb/mariadb.log

2025-07-30  6:10:59 687 [Warning] Aborted connection 687 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:00 688 [Warning] Aborted connection 688 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:01 689 [Warning] Aborted connection 689 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:02 691 [Warning] Aborted connection 691 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:03 697 [Warning] Aborted connection 697 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:03 709 [Warning] Aborted connection 709 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:04 710 [Warning] Aborted connection 710 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:05 711 [Warning] Aborted connection 711 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got a packet bigger than 'max_allowed_packet' bytes)
2025-07-30  6:11:06 14 [Warning] Aborted connection 14 to db: 'icingadb' user: 'icingadb' host: 'localhost' (Got an error reading communication packets)

I will work on upgrading packages but it is difficult because this systems is RHEL 8 and I have to build the packages myself because I don’t have a paid subscription.

Do I need to wipe the icingadb and re-migrate?

6

The “Got a packet bigger than ‘max_allowed_packet’ bytes” part looks promising. Please take a look at the specific section of our operations manual and ensure your MariaDB’s configuration has at least 64M set.

This makes things a bit harder, I am sorry hearing this.

I hope not. For upgrading, please follow the upgrading docs. For jumping from 1.2.1 to 1.4.0, one schema upgrade is necessary.

7

Adding

max_allowed_packet = 67108864

seems to have fixed the problem
Thanks very much for your help