IcingaDB disconnects from Redis

  • Icinga DB Web version: 1.0.0-rc2
  • Icinga Web 2 version: 2.9.5
  • Icinga 2 version: r2.13.2-1
  • Icinga DB version: v1.0.0-rc2-dev
  • Icinga DB Redis: 6.2.6-2.focal
  • PHP version used: 7.4.3
  • Icinga PHP Third Library: 0.7.0
  • Icinga PHP Third Party: 0.10.0
  • Server operating system and version: Ubuntu 20.04.3 LTS

I have managed to get IcingaDB started and the web interface shows everything as online, but this ends up being only temporary. About 30 seconds after it has started, IcingaDB stops and the following is recorded:

● icingadb.service - Icinga DB
     Loaded: loaded (/lib/systemd/system/icingadb.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2021-11-24 21:00:37 UTC; 7min ago
    Process: 17851 ExecStart=/usr/sbin/icingadb --config /etc/icingadb/config.yml (code=exited, status=1/FAILURE)
   Main PID: 17851 (code=exited, status=1/FAILURE)

Nov 24 21:00:02 vmmon-icingamaster-dev systemd[1]: Starting Icinga DB...
Nov 24 21:00:02 vmmon-icingamaster-dev systemd[1]: Started Icinga DB.
Nov 24 21:00:02 vmmon-icingamaster-dev icingadb[17851]: Starting Icinga DB
Nov 24 21:00:02 vmmon-icingamaster-dev icingadb[17851]: Connecting to database
Nov 24 21:00:02 vmmon-icingamaster-dev icingadb[17851]: Connecting to Redis
Nov 24 21:00:02 vmmon-icingamaster-dev icingadb[17851]: Starting history sync
Nov 24 21:00:03 vmmon-icingamaster-dev icingadb[17851]: heartbeat: Received Icinga heartbeat
Nov 24 21:00:03 vmmon-icingamaster-dev icingadb[17851]: Taking over
Nov 24 21:00:03 vmmon-icingamaster-dev icingadb[17851]: Starting config sync
Nov 24 21:00:03 vmmon-icingamaster-dev icingadb[17851]: Starting initial state sync
Nov 24 21:00:03 vmmon-icingamaster-dev icingadb[17851]: Starting overdue sync
Nov 24 21:00:04 vmmon-icingamaster-dev icingadb[17851]: config-sync: Updating 3 items of type host state
Nov 24 21:00:04 vmmon-icingamaster-dev icingadb[17851]: config-sync: Updating 13 items of type service state
Nov 24 21:00:04 vmmon-icingamaster-dev icingadb[17851]: Starting config runtime updates sync
Nov 24 21:00:04 vmmon-icingamaster-dev icingadb[17851]: config-sync: Finished config sync in 137.94895ms
Nov 24 21:00:04 vmmon-icingamaster-dev icingadb[17851]: config-sync: Finished initial state sync in 140.929412ms
Nov 24 21:00:04 vmmon-icingamaster-dev icingadb[17851]: Starting state runtime updates sync
Nov 24 21:00:37 vmmon-icingamaster-dev icingadb[17851]: redis: connection pool timeout
                                                        can't execute Redis script
                                                        github.com/icinga/icingadb/pkg/icingadb/overdue.Sync.sync
                                                                /builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/overdue/sync.go:190
                                                        github.com/icinga/icingadb/pkg/icingadb/overdue.Sync.Sync.func4
                                                                /builds/packaging/deb-icingadb/build/icingadb/pkg/icingadb/overdue/sync.go:73
                                                        golang.org/x/sync/errgroup.(*Group).Go.func1
                                                                /home/build/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57
                                                        runtime.goexit
                                                                /usr/local/go/src/runtime/asm_amd64.s:1371
Nov 24 21:00:37 vmmon-icingamaster-dev systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Nov 24 21:00:37 vmmon-icingamaster-dev systemd[1]: icingadb.service: Failed with result 'exit-code'.

I ran Redis manually with verbose logging and it doesn’t seem like Redis is the source of the issue:

17835:M 24 Nov 2021 21:00:03.904 - Accepted 127.0.0.1:44476
17835:M 24 Nov 2021 21:00:03.911 - Accepted 127.0.0.1:44478
17835:M 24 Nov 2021 21:00:06.687 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:11.705 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:16.724 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:21.742 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:26.760 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:31.777 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:36.795 - DB 0: 31 keys (0 volatile) in 32 slots HT.
17835:M 24 Nov 2021 21:00:37.016 - Client closed connection
17835:M 24 Nov 2021 21:00:37.016 - Client closed connection
17835:M 24 Nov 2021 21:00:37.017 - Client closed connection

I also verified that the Redis config has the timeout directive uncommented and set to 0 to disable the setting.

Any ideas on where to look further?

Hi,

Do you have sensitive data in your configuration? If not, could you please create a Redis dump via icingadb-redis-cli SAVE and provide us with the resulting file /var/lib/icingadb-redis/dump.rdb?

All the best,
Eric

dump.zip (593.1 KB)
@elippmann Here’s the dump.rdb file from the redis install.

Hey @thozook,

we’ve most likely found the cause for your issue and fixed it on our master branch (GitHub Reference).

This issue only appears on machines with 1 CPU, which means you have 2 options:

  • Increase the core count of your machine to 2
  • Install the latest Icinga DB master from our snapshot repositories

Greetings
Noah