Icingadb keeps crashing

Hi guys,

I’m having some issues with my Icinga server since more than a week.
Icinga is installed on a vm on a promox server; For more than a week now, the server has lots of IO delay and that seems linked to the icingadb because if I stop it, the IO delay goes down.

If I let it run, most of the time, the icingadb crashes.

Logs show that icinga-director.service can’t start properly

oct. 27 08:51:08 icinga-master-p-01 icingadirector[350811]: RuntimeException in /usr/share/icingaweb2/modules/director/library/Director/Daemon/DaemonDb.php:162 with message: DB is locked by a running daemon instance, will retry
oct. 27 08:51:48 icinga-master-p-01 systemd[1]: icinga-director.service: Watchdog timeout (limit 10s)!
oct. 27 08:51:48 icinga-master-p-01 systemd[1]: icinga-director.service: Killing process 350811 (icingacli) with signal SIGABRT.
oct. 27 08:51:48 icinga-master-p-01 systemd[1]: icinga-director.service: Main process exited, code=killed, status=6/ABRT
oct. 27 08:51:48 icinga-master-p-01 systemd[1]: icinga-director.service: Failed with result ‘watchdog’.
oct. 27 08:52:19 icinga-master-p-01 systemd[1]: icinga-director.service: Scheduled restart job, restart counter is at 202.

And icingadb logs show a lot of “icingadb[358638]: heartbeat: Previous heartbeat not read from channel”

The disk space seems to have grown quite a bit as well

Any ideas ?

  • Icinga DB Web version (System - About): 1.2.3
  • Icinga Web 2 version (System - About): 2.12.5
  • Web browser: Google Chrome
  • Icinga 2 version (icinga2 --version): r2.15.1-1
  • Icinga DB version (icingadb --version): v1.4.0
  • PHP version used (php --version): 8.2.29
  • Server operating system and version: Debian 12

Thanks

Hi @padri45 ,
is the SQL database ( MariaDB / MYSQL or Postgres ) on the same VM as Icinga 2 and Icinga DB ?