Services notifications getting disabled

Icinga 2
,
1

Hello,

For some unknown reason, some of the services ( not all ) on multiple hosts end up having notification disabled on them at some point. I would like to track this down why that is occurring.
I couldn’t find anything odd as far as the configuration ( no director here ) is concerned. Icinga2 api is open from a few limited hosts. How do I debug or tracke this down when it happens again?

Please advise,
Thanks

2

Hi @monigacom ,

When looking at these problem I start with where the original config is, it sounds like it is on disk for you. Then I compare that with the icinga2 object list on the masters and if I need to any endpoints dowstream.

The icinga config is pre load by Icinga, the object list is post load by Icinga so they should match.

I find this is a good way to ensure that config is deploy’d correctly to all endpoints in a cluster.

If you find that is all correct and that your on disk config is changing you could try turning it into a git repo with a cron based auto commit. Then at least you’ll know what changed and when.

1 Like
3

For the corresponding services is a good idea. Be sure to run icinga2 daemon -C --dump-objects before, to have the up-to-date versions of each object available, as this command does not query live data (only the API does)

I would also enable the debug log and take a look in there.
If you can’t force this problem and it “just happens” the debug log can fill up your space pretty quick, depending on the size of your setup and the messages logged.

We have done the following for this case (to “always” be able to have a look at a debug log):
edited /etc/icinga2/features-available/debuglog.conf

object FileLogger "debug-file" {
  severity = "debug"
  path = LogDir + "/debug_log/debug.log"
}

addition to /etc/logrotate.d/icinga2

/var/log/icinga2/debug_log/debug.log {
        hourly
        rotate 72
        maxsize 10G
        compress
        su icinga icinga
        missingok
        notifempty
        postrotate
                /bin/kill -USR1 $(cat /run/icinga2/icinga2.pid 2> /dev/null) 2> /dev/null || true
        endscript
}

This keeps ~72 hours of debug logs in our setup, to have them in case we need them.
Adjust it to your needs and/or available space.