IcingaDB 'max_allowed_packet' exceeded

Icinga 2 Icinga DB
1

Hello

Since the update to IcingaDB v1.2.1, we are experiencing Daemon failures in the IcingaDB every 5 minutes.

Dec 22 10:13:58 skinner2.backbone.admin icingadb[1507116]: Error 1105 (HY000): Parameter of prepared statement which is set through mysql_send_long_data() is longer than 'max_allowed_packet' bytes
                                                           can't perform "INSERT INTO \"state_history\" (\"state_type\", \"hard_state\", \"previous_soft_state\", \"check_attempt\", \"soft_state\", \"long_output\", \"object_type\", \"host_id\", \"check_source\", \"endpoint_id\", \"service_id\", \"event_time\", \"previous_hard_state\", \"output\", \"max_check_attempts\", \"id\", \"scheduling_source\", \"environment_id\") VALUES (:state_type,:hard_state,:previous_soft_state,:check_attempt,:soft_state,:long_output,:object_type,:host_id,:check_source,:endpoint_id,:service_id,:event_time,:previous_hard_state,:output,:max_check_attempts,:id,:scheduling_source,:environment_id) ON DUPLICATE KEY UPDATE \"id\" = VALUES(\"id\")"
                                                           github.com/icinga/icinga-go-library/database.CantPerformQuery
                                                                   github.com/icinga/icinga-go-library@v0.4.0/database/utils.go:16
                                                           github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2.1
                                                                   github.com/icinga/icinga-go-library@v0.4.0/database/db.go:535
                                                           github.com/icinga/icinga-go-library/retry.WithBackoff
                                                                   github.com/icinga/icinga-go-library@v0.4.0/retry/retry.go:65
                                                           github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
                                                                   github.com/icinga/icinga-go-library@v0.4.0/database/db.go:530
                                                           golang.org/x/sync/errgroup.(*Group).Go.func1
                                                                   golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78
                                                           runtime.goexit
                                                                   runtime/asm_amd64.s:1700
                                                           retry deadline exceeded
                                                           github.com/icinga/icinga-go-library/retry.WithBackoff
                                                                   github.com/icinga/icinga-go-library@v0.4.0/retry/retry.go:100
                                                           github.com/icinga/icinga-go-library/database.(*DB).NamedBulkExec.func1.(*DB).NamedBulkExec.func1.1.2
                                                                   github.com/icinga/icinga-go-library@v0.4.0/database/db.go:530
                                                           golang.org/x/sync/errgroup.(*Group).Go.func1
                                                                   golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78
                                                           runtime.goexit
                                                                   runtime/asm_amd64.s:1700

On our server side, we already set ‘max_allowed_packet’ to 64MB and on the client side, we are at 64MB as well.

  • Icinga DB Web version: 2.12.2
  • Icinga Web 2 version: 1.1.13
  • Web browser: Firefox 128.0.5esr
  • Icinga 2 version: r2.14.3-1
  • Icinga DB version: v1.2.1
  • PHP version used: 8.1.2-1ubuntu2.20
  • Server operating system and version: Ubuntu 22.04.5 LTS

How shall we proceed?

2

FTR: our icingadb-redis-server had a memory usage of around ~300MB.

root@skinner1:~# redis-cli -p 6380 info | grep used_memory_human
used_memory_human:304.15M

After flushing out the Redis Store, our system no longer crashes.

redis-cli -p 6380 flushall

I’m assuming that during the upgrade the Redis Server was running on, while the IcingaDB was down and stored up too much data in the “buffer” and IcingaDB was no longer capable of syncing this data.

However, issue resolved.

2 Likes
3

Thanks for both reporting and solving this issue.
As a crash is undesired, I have taken another look at this issue.

Icinga DB always sets max_allowed_packet to 64 MiB, as this is what the underlying MySQL client library does, https://github.com/go-sql-driver/mysql/blob/4395c45fd098a81c5251667cda111f94c693ab14/dsn.go#L88. However, if I read the following MySQL bug - https://bugs.mysql.com/bug.php?id=83958 - correctly, prepared statements may easily exceed this limit, as it seems to be the case for your crash.

Due to the temporary unavailability of Icinga DB, the backlog stored in the Redis outgrew this limit, resulting in a later crash. I created an issue for this, as I think we should be able to mitigate this bug within our code base.

1 Like