Debugging zombie icinga2

About every 4-6 weeks, Icinga Web 2 UI informs us that the backend is not running. I check systemctl and icinga2 is running, with very few children (should have dozens and dozens of plugins running), and a disconnect message in the mysql logs.

In every case, a systemctl restart icinga2 fixes it.

I read through troubleshooting, and to turn on debug I have to do a restart. But since that fixes it, that doesn’t help at all. Also Debug logs about 15M a minute, so it is not practical to leave it running 24/7.

So, questions.

  1. Is there an API or CLI way to turn on debugging?
  2. Is there something else I can do to debug the situation as it happens?

Currently running 2.12.3-1.bionic on Ubuntu.

Not sure if this works when icinga2 is already in such a broken state, but you can do the following with the console:

# icinga2 console --connect 'https://apiuser:password@icinga-server:5665/'
=> get_objects(FileLogger)[0].severity="debug"

You can also execute other commands via the console to verify if it is still working at all like log("Is icinga2 still writing to the log?").

Thank you Dirk, I’ll try that that next time it happens.

<1> => get_objects(FileLogger)[0].severity=“debug”

If this should be a question if this is the normal output, then yes it is because you are just setting a value.

I haven’t gotten anywhere with debugging, so I’m going with writing a watchdog to restart: