Stale downtime entries that can't be removed

We use API to create and remove downtime entries every once in a while.
All works well, but those entries are sometimes visible in web interface long after they should be gone. They have negative amount of time in “Expires” field and when I try to delete those manually I get “Can’t send external Icinga command: 404 Objects not found”.
Indeed those entries are not displayed when I use cli tool like icinga2 object list --type Downtime. The only way to remove those stale downtimes is to restart icinga daemon.
I’d be grateful for some hints on this and I’ll happily respond to any questions.

HI Aquaz,

Welcome to the forum.

It is a interesting thing. some questions:

  • How busy is your host ?
  • any cpu / mem shortage?
  • Did you update to the latest version?

You could possibly enable debug logging and create and delete a downtime and see what gets logged.

It’s actually not busy at all, and there’s plenty of resources available on this machine.
We’re currently using icinga2 2.11.2 with icingaweb2 2.7.3
I see that there’s an update available, we’ll take a look.
The problem with this issue is it doesn’t happen every time, more like randomly for what I can tell right now, but we’ll try to gather more information.