Icingadb service random failures

Icinga 2
1

Hello,

icingadb service will fail randomly. It has happened a couple of time since deplyoment couple of months ago. Redis service and MariaDB services have been up and running. Restarting the icingdb service does fix it until it happens again randomly. What does this error mean and how troubleshoot this? Is it a known issue or bug ? Thanks

systemctl status icingadb
● icingadb.service - Icinga DB
   Loaded: loaded (/usr/lib/systemd/system/icingadb.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2025-09-16 03:23:01 PDT; 1 weeks 6 days ago
  Process: 721776 ExecStart=/usr/sbin/icingadb --config /etc/icingadb/config.yml (code=exited, status=1/FAILURE)
 Main PID: 721776 (code=exited, status=1/FAILURE)


 icingadb[721776]: read tcp 10.4.182.22:50536->10.4.182.52:6380: i/o timeout#012can't perform "[xread count 4096 block 1000 streams icinga:history:stream:state 1758014208760-0]"#012github.com/icinga/icingadb/pkg/icingaredis.WrapCmdErr#012#011github.com/icinga/icingadb/pkg/icingaredis/utils.go:121#012github.com/icinga/icingadb/pkg/icingaredis.(*Client).XReadUntilResult#012#011github.com/icinga/icingadb/pkg/icingaredis/client.go:204#012github.com/icinga/icingadb/pkg/icingadb/history.Sync.readFromRedis#012#011github.com/icinga/icingadb/pkg/icingadb/history/sync.go:114#012github.com/icinga/icingadb/pkg/icingadb/history.Sync.Sync.func1#012#011github.com/icinga/icingadb/pkg/icingadb/history/sync.go:83#012golang.org/x/sync/errgroup.(*Group).Go.func1#012#011golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78#012runtime.goexit#012#011runtime/asm_amd64.s:1695#012can't read history#012github.com/icinga/icingadb/pkg/icingadb/history.Sync.readFromRedis#012#011github.com/icinga/icingadb/pkg/icingadb/history/sync.go:116#012github.com/icinga/icingadb/pkg/icingadb/history.Sync.Sync.func1#012#011github.com/icinga/icingadb/pkg/icingadb/history/sync.go:83#012golang.org/x/sync/errgroup.(*Group).Go.func1#012#011golang.org/x/sync@v0.7.0/errgroup/errgroup.go:78#012runtime.goexit#012#011runtime/asm_amd64.s:1695


# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.2-1)

Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Red Hat Enterprise Linux
  Platform version: 8.10 (Ootpa)
  Kernel: Linux
  Kernel version: 4.18.0-553.5.1.el8_10.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.5.0
  Build host: unknown
  OpenSSL version: OpenSSL 1.1.1k  FIPS 25 Mar 2021

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker icingadb mainlog notification

[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 User.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 13 TimePeriods.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 137 Services.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 4 Zones.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 5 NotificationCommands.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 4 Notifications.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 FileLogger.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 33 Hosts.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 4 Endpoints.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 ApiUser.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 ApiListener.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 374 CheckCommands.
[2025-09-29 11:46:29 -0700] information/ConfigItem: Instantiated 1 IcingaDB.
[2025-09-29 11:46:29 -0700] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-09-29 11:46:29 -0700] information/cli: Finished validating the configuration file(s).

# rpm -qa | grep -i icinga
icingacli-2.12.1-1.el8.noarch
icingaweb2-common-2.12.1-1.el8.noarch
icinga2-2.14.2-1.el8.x86_64
icinga2-common-2.14.2-1.el8.x86_64
icinga-l10n-1.3.0-1.el8.noarch
php-icinga-2.12.1-1.el8.noarch
icingadb-1.2.0-1.el8.x86_64
icingaweb2-2.12.1-1.el8.noarch
icinga2-bin-2.14.2-1.el8.x86_64
icinga-php-thirdparty-0.12.1-1.el8.noarch
icinga-php-library-0.13.2-1.el8.noarch
icingadb-web-1.1.2-1.el8.noarch
2

Hi @monigacom,
Is the Redis service running on a different machine?
The error message suggests (from a short glance) a timeout in the underlying network connection. Does that sound like a reasonable cause?

3

Thanks for creating this thread. Please note that your Icinga DB version is outdated by two versions. Same applies to Icinga 2, which has at least two relevant CVEs.

Since there were some lots of bug fixes in the last Icinga DB versions, I would urge you to upgrade.

4

Yes, the Redis service is running on a different server. Yeah, network connectivity issue could be a possibility.