Unexpected Redis schema version: "5" (expected "6")

[HenryyXVIII]

A week ago I recognized that my icingadb doesnt really point on my redis.
In the Icingaweb2 module tab was the ip from the icinga master wich we i changed to the IP-address of the DB-Node.
Since that change my IcingaDB Service crash randomly multiple times a day.

---

The Error Message

Aug 17 11:57:25 RZHROMONITORDB icingadb[130103]: config-sync: Inserting 1187 items of type host
Aug 17 11:57:24 RZHROMONITORDB icingadb[130103]: Starting initial state sync
Aug 17 11:57:24 RZHROMONITORDB icingadb[130103]: Starting overdue sync
Aug 17 11:57:24 RZHROMONITORDB icingadb[130103]: Starting config sync
Aug 17 11:57:24 RZHROMONITORDB icingadb[130103]: Taking over
Aug 17 11:57:24 RZHROMONITORDB icingadb[130103]: heartbeat: Received Icinga heartbeat
Aug 17 11:57:24 RZHROMONITORDB icingadb[130103]: Starting history sync
Aug 17 11:56:58 RZHROMONITORDB icingadb[130103]: Waiting for Icinga 2 to write into Redis, please make sure you have started Icinga 2 and the Icinga DB feature is enabled
Aug 17 11:56:55 RZHROMONITORDB icingadb[130103]: Connecting to Redis at 'redis://localhost:6380'
Aug 17 11:56:55 RZHROMONITORDB icingadb[130103]: Connecting to database at 'mysql://icingadb@localhost:3306/icingadb'
Aug 17 11:56:55 RZHROMONITORDB icingadb[130103]: Starting Icinga DB daemon (v1.5.1)
Aug 17 11:56:55 RZHROMONITORDB systemd[1]: Started icingadb.service - Icinga DB.
Aug 17 11:56:55 RZHROMONITORDB systemd[1]: Starting icingadb.service - Icinga DB...
Aug 17 11:55:02 RZHROMONITORDB systemd[1]: icingadb.service: Consumed 33.484s CPU time.
Aug 17 11:55:02 RZHROMONITORDB systemd[1]: icingadb.service: Failed with result 'exit-code'.
Aug 17 11:55:02 RZHROMONITORDB systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 11:55:02 RZHROMONITORDB icingadb[128582]: unexpected Redis schema version: "5" (expected "6"), please make sure you are running compatible versions of Icinga 2 and Icinga DB
Aug 17 11:55:02 RZHROMONITORDB icingadb[128582]: Icinga 2 started a new config dump, waiting for it to complete
Aug 17 11:55:02 RZHROMONITORDB icingadb[128582]: config-sync: Deleting 11002 items of type notification recipient

After restarting the icingadb.service the db is running again but crash after several hours.

! corrention !

The icingadb carshes if someon deploys with the Icinga-Director. 20 seconds after rollout the icingadb.services crashes

Montag, 17. August 2026
[admin.xy modify service "systemd"		17:13:18
[admin.abc] modify host "Host162"		11:54:46
[admin.abc] modify host "Host162"		10:50:56

╰─# journalctl -u icingadb --since "2026-08-17 10:00" -f | grep Failed -C 3
Aug 17 10:51:13 RZHROMONITORDB icingadb[23493]: config-sync: Updating 1187 items of type host state
Aug 17 10:51:14 RZHROMONITORDB icingadb[23493]: unexpected Redis schema version: "5" (expected "6"), please make sure you are running compatible versions of Icinga 2 and Icinga DB
Aug 17 10:51:14 RZHROMONITORDB systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 10:51:14 RZHROMONITORDB systemd[1]: icingadb.service: Failed with result 'exit-code'.
Aug 17 10:51:14 RZHROMONITORDB systemd[1]: icingadb.service: Consumed 1h 9min 21.985s CPU time.
Aug 17 10:59:24 RZHROMONITORDB systemd[1]: Starting icingadb.service - Icinga DB...
Aug 17 10:59:24 RZHROMONITORDB icingadb[127288]: Starting Icinga DB daemon (v1.5.1)
--
Aug 17 11:55:02 RZHROMONITORDB icingadb[128582]: Icinga 2 started a new config dump, waiting for it to complete
Aug 17 11:55:02 RZHROMONITORDB icingadb[128582]: unexpected Redis schema version: "5" (expected "6"), please make sure you are running compatible versions of Icinga 2 and Icinga DB
Aug 17 11:55:02 RZHROMONITORDB systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 11:55:02 RZHROMONITORDB systemd[1]: icingadb.service: Failed with result 'exit-code'.
Aug 17 11:55:02 RZHROMONITORDB systemd[1]: icingadb.service: Consumed 33.484s CPU time.
Aug 17 11:56:55 RZHROMONITORDB systemd[1]: Starting icingadb.service - Icinga DB...
Aug 17 11:56:55 RZHROMONITORDB systemd[1]: Started icingadb.service - Icinga DB.
--
Aug 17 17:13:33 RZHROMONITORDB icingadb[130103]: Icinga 2 started a new config dump, waiting for it to complete
Aug 17 17:13:33 RZHROMONITORDB icingadb[130103]: unexpected Redis schema version: "5" (expected "6"), please make sure you are running compatible versions of Icinga 2 and Icinga DB
Aug 17 17:13:33 RZHROMONITORDB systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 17:13:33 RZHROMONITORDB systemd[1]: icingadb.service: Failed with result 'exit-code'.
Aug 17 17:13:33 RZHROMONITORDB systemd[1]: icingadb.service: Consumed 4min 47.235s CPU time.
Aug 17 17:16:41 RZHROMONITORDB systemd[1]: Starting icingadb.service - Icinga DB...
Aug 17 17:16:41 RZHROMONITORDB icingadb[139584]: Starting Icinga DB daemon (v1.5.1)

---

The MariaDB schema is 1.4.0 and i migrated the MariaDB several months befor.

I have split up my setup to a master node, DB Node (MariaDB, Redis, IcingaDB, Influx) and Icingaweb2 Node.

The Icingadb module is enabled on both the Icinga Master and the DB Node.

Befor I thought i fixed it couse there are two entrys in the Redis.ini pointing at the master and the DB but now we recognized that this happen on more than one system that after deploying the Icingdb crashes (my case) or need some time to load (client)

---

Versions

icinga2 - The Icinga 2 network monitoring daemon (version: r2.16.4-1)
Icinga DB version: v1.5.1 (both master and DB)
mariadb Ver 15.1
icingadb-redis 8.2.8-1+debian12
icingadb-redis-server 7.0.15-1+debian12

---

Maybe there are some old apckets or other apps installed on the master interferring with the redis/IcingaDB. After restarting the icingdb on the DB host its start waiting for a hearthbeat and only after you resatart the master it begins working (screenshot).

config on the Icinga master

╰─# cat /etc/icinga2/features-enabled/icingadb.conf

object IcingaDB "icingadb" {
  host = "10.1.68.3" #DB-Host
  port = 6380
  //password = "xxx"
}

We also stopped the icinga2.service and the icingadb.service and flushed the redis server without anny succes.

icingadb-redis-cli flushall

Did i miss something?

[HenryyXVIII]

Update

after researching i have dicoverd that on every node the icingadb feature is enabled.

Server Node	IP Address	Features-enabled	Services
Icinga-Master	10.1.68.1	api, checker, icingadb, influxdb2, mainlog, notification, syslog	Icinga2
Icinga-DB	10.1.68.2	api, checker, icingadb, mainlog	Icingadb, Redis, Mariadb, Influx2
Icinga-Web2	10.1.68.3	api, checker, icingadb, mainlog	Icingaweb2, Apache

Currently I have comment out the config from the Icingaweb2 (like how it was befor) and restarted everything.

Now the DB isnt crashing but after rolling out a config in the director, all services and hosts, under the tab Overview/Dashboard disappear for several seconds/minutes and then appear again.

Logs

Aug 19 14:17:22 RZHROMONITORDB icingadb[240535]: runtime-updates: Upserted 1 HostState items
Aug 19 14:17:25 RZHROMONITORDB icingadb[240535]: Icinga 2 started a new config dump, waiting for it to complete
Aug 19 14:17:25 RZHROMONITORDB icingadb[240535]: history-sync: Synced 17 state history items
Aug 19 14:17:25 RZHROMONITORDB icingadb[240535]: Starting config sync
Aug 19 14:17:25 RZHROMONITORDB icingadb[240535]: Starting initial state sync
Aug 19 14:17:25 RZHROMONITORDB icingadb[240535]: Starting overdue sync
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Updating 1187 items of type host state
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: Icinga 2 started a new config dump, waiting for it to complete
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Aborted initial state sync after 1.528961097s
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Aborted config sync after 1.529004427s
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: Starting config sync
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: Starting initial state sync
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: Starting overdue sync
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 1187 items of type host
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 1187 items of type host state
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 2075 items of type host customvar
Aug 19 14:17:26 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 3402 items of type hostgroup member
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 694 items of type downtime
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 230 items of type comment
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 11256 items of type service
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 8865 items of type servicegroup member
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 15095 items of type notification user
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 724 items of type endpoint
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 1 items of type usergroup
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 11000 items of type notification recipient
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 3 items of type icon image
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 1 items of type notification usergroup
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 14785 items of type service customvar
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 1043 items of type customvar
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 1494 items of type customvar flat
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Updating 1 items of type zone
Aug 19 14:17:27 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 724 items of type zone
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 6 items of type checkcommand envvar
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 25 items of type checkcommand
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 35 items of type checkcommand customvar
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: config-sync: Updating 1498 items of type service state
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 9756 items of type service state
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 171 items of type checkcommand argument
Aug 19 14:17:28 RZHROMONITORDB icingadb[240535]: database: Can't execute query. Retrying        error="can't perform \"INSERT INTO \\\"service_state\\\" (\\\"check_source\\\", \\\"execution_time\\\", \\\"last_update\\\", \\\"latency\\\", \\\"in_downtime\\\", >
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: database: Can't execute query. Retrying        error="can't perform \"INSERT INTO \\\"service_state\\\" (\\\"check_source\\\", \\\"execution_time\\\", \\\"last_update\\\", \\\"latency\\\", \\\"in_downtime\\\", >
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: config-sync: Deleting 89 items of type notification
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: database: Can't execute query. Retrying        error="can't perform \"INSERT INTO \\\"service_state\\\" (\\\"check_source\\\", \\\"execution_time\\\", \\\"last_update\\\", \\\"latency\\\", \\\"in_downtime\\\", >
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: Starting config runtime updates sync
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: config-sync: Finished config sync in 2.241622792s
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: database: Can't execute query. Retrying        error="can't perform \"INSERT INTO \\\"service_state\\\" (\\\"check_source\\\", \\\"execution_time\\\", \\\"last_update\\\", \\\"latency\\\", \\\"in_downtime\\\", >
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: database: Query retried successfully after error
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: database: Query retried successfully after error
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: database: Query retried successfully after error
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: Starting history retention
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: config-sync: Finished initial state sync in 2.812867548s
Aug 19 14:17:29 RZHROMONITORDB icingadb[240535]: Starting state runtime updates sync
Aug 19 14:17:45 RZHROMONITORDB icingadb[240535]: history-sync: Synced 6 state history items
Aug 19 14:17:46 RZHROMONITORDB icingadb[240535]: overdue-sync: Synced 1 host overdue indicators

I still dont know the difference between icingadb (feature), icingdb (module) [Icinga DB WEB], icingadb.service, icingadb-redis-server.service, icingadb-redis.service, redis.service.

is there any diference beween moduel:icingadb and the feature:Icingadb-web. And are those the reason for the redis crash (schema 5 expected 6)?

[moreamazingnick]

@HenryyXVIII

Column 1	Column 2	Column 3	Column 4
Node	IP Address	Features-enabled	Services
Icinga-Master	10.1.68.1	api, checker, icingadb, influxdb2, mainlog, notification, syslog	Icinga2
Icinga-DB	10.1.68.2	api, checker, icingadb, mainlog	Icingadb, Redis, Mariadb, Influx2
Icinga-Web2	10.1.68.3	api, checker, icingadb, mainlog	Icingaweb2, Apache

just out of curiosity, did you configure a 3 master icinga cluster?

[HenryyXVIII]

No, it’s one Master, but the Icinga Master (Core), the database, and Icinga-web are spread across three VMs, so it’s a single Master setup with separate hosts for the database and Icinga-web2.

[moreamazingnick]

the easiest way is if icingadb icinga2 and icingaweb2 runs on the same machine.

you most likely have some problem in your config files or some database.

so in order to debug that you need to describe your setup in full. each component each database, each config file

It says feature enabled icingadb on all three machines, the feature enable is part of icinga2 so, there is some problem here since icingadb should run on the master not on a satellite or somewhere else.

stick to one machine for a start. if you feel comportable with icinga, you can try to outsource components.

But my preference would be one machine, but you might can outsource apache2 and icingaweb2.

you also have no real improvement in splitting that in vms. if you want to encapsulate each components, you can use the helmcharts and kubernetes, docker etc.

[moreamazingnick]

@HenryyXVIII

I still dont know the difference between icingadb (feature), icingdb (module) [Icinga DB WEB], icingadb.service, icingadb-redis-server.service, icingadb-redis.service, redis.service.

is there any diference beween moduel:icingadb and the feature:Icingadb-web. And are those the reason for the redis crash (schema 5 expected 6)?

• icinga2 is the core, check execution engine
• icingadb like the history database, successor of ido/monitoring
• icingadb-web is the module for icingaweb2 to display the data from icingadb

and now the confusing part if you install icingadb-web, in icingaweb2 the module is called icingadb

• icinga2 executes the check
• icinga2 writes it to a database and redis using icingadb feature
• icingaweb2 has a module called icingadb (icingadb-web) which reads from the database and from redis

depending on your os preference you can use one of these excelent guides to get started:

Install Guide Fedora 44 Server Icinga2 + IcingaDB + IcingaDB-Web + Director

Install Guide Fedora 44 Server Icinga2 + IcingaDB + IcingaDB-Web + Director A try of a more structured way of a first time install. Basics I skip the part with accessing the machine. First thing most people would do is updating the os. root@testsystem:~# dnf update -y && dnf upgrade -y Second in a Fedora System WGET is not installed so we do it dnf install wget -y Then adding the repos for installing icinga2 and components. Most common people would visit Https://packages.icinga.com Ch…

Install Guide Debian 13 (Trixie) Icinga2 + IcingaDB + IcingaDB-Web + Director

Install Guide Debian 13 (Trixie) Icinga2 + IcingaDB + IcingaDB-Web + Director A try of a more structured way of a first time install. Basics I skip the part with accessing the machine. First thing most people would do is updating the os. root@testsystem:~# apt update -y && apt upgrade -y Then adding the repos for installing icinga2 and components. Most common people would visit Https://packages.icinga.com Choose the distro you want and apply the steps for the repository setup in the syst…

Installing Icingaweb2 and Icinga DB Web on Ubuntu 26.04 | NETWAYS

[HenryyXVIII]

Thanks for clarifying the single components, it get quite complicated when there is a feature called icingadb, a module and a daemon.

In my case I adopted the system in this state. I only set up for testing my own all in one VM Icinga2 setup couse I dint know much about K8s and Helcharts only some Docker.

But it seems that the error is fixed.

I have disabled the icingadb feature on the DB and Icingaweb host.

Both where activated.

The config was commented out but on the icinga web not and there was de Adress of the icingadb-redis server.

So someon thought that he has configured the module instead of had the feature.