the latest version of Icinga2 has broken my monitoring setup due to the fact that a definition for check_systemd has now been added to ITL by default. However, for those of us using the community check_systemd before, this creates a conflict.
To start, here are some of my system details:
version: icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.1-1)
os: Ubuntu 18.04.5 LTS
enabled features: api checker mainlog
output of icinga2 daemon -C
[2021-08-19 16:28:14 +0200] information/cli: Icinga application loader (version: r2.13.1-1)
[2021-08-19 16:28:14 +0200] information/cli: Loading configuration file(s).
[2021-08-19 16:28:14 +0200] critical/config: Error: Object 'systemd' of type 'CheckCommand' re-defined: in /var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf: 6:1-6:29; previous definition: in /usr/share/icinga2/include/plugins-contrib.d/systemd.conf: 3:1-3:29
Location: in /var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf: 6:1-6:29
/var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf(4): #
/var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf(5):
/var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf(6): object CheckCommand "systemd" {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf(7): command = [ "sudo", PluginContribDir + "/check_systemd" ]
/var/lib/icinga2/api/zones/global-checkcommands/_etc/checkcommand-systemd.conf(8):
[2021-08-19 16:28:14 +0200] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
I had seen this before on my icinga2 master but fixed it by deleting the ITL config file. However, it seems that with every icinga update the updater attempts to recreate the ITL version of the check command, causing update issues every time.
Since I am using a slightly modified version of the check_systemd command, how can I avoid running into this issue on every icinga2 package update? I’d prefer not to have to rename my checkcommand as that would have serious impact on my entire monitored infrastructure. Simply disabling ITL altogether is not an option, since I am using many other checks provided by the (fantastic!) ITL library.
Obviously that is a solution, but rather cumbersome to implement. Also, this would leave the door open for a similar incident in the future if more checks are added to the ITL which conflict with any addons I may already have.
Would it not be better/safer is there is a way to enable only specific check commands from ITL instead of using an all or nothing approach? That way people could whitelist the ones they are using and not have any conflict when updates are rolled out.
From a systems management perspective, you could just remove the file /usr/share/icinga2/include/plugins-contrib.d/systemd.conf and use configuration management (or a simple cron job) to ensure that file is no longer there (along with any other checks you don’t want).
I think it might be easier to re-create (probably easily done via cp) the ITL objects that you actually use, reconfigure your checks to use your copies, and then omit the ITL entry in your icinga2.conf file.
I can’t find any documentation on omitting select objects from the ITL library, so unless the ability already exists, this is probably your best bet.
With that said, I think it would be nice to have a way to omit stuff from a library we don’t want.
Maybe something like the default being
include_recursive <itl>
# Or you can comment the above and uncomment below
# include <itl/systemd.conf>
# of course, in your case you don't want the above entry, you would just include the things that you want
I tried deleting the file you mentioned and worked, but only after I had already attempted to install the updates (apt upgrade) and that process failed. Once removed, everything started working again (this was only on the master).
Today however apparently a new version appeared and the problem was recreated as the updated installer once again placed the checkcommand definition file in the contrib dir.
I ended up using ansible to rename my own plugin and that fixed it permanently; I just hope this doesn’t occur again when another new command makes it to ITL status. So being able to selectively enable/disable plugins from ITL would be great.
I also can’t imagine I am the only one with this issue, since I assume many people are monitoring their systemd services? Either way, thanks for your advice, issue solved (for now). I was just hoping for a different solution!
One thought, in addition to taking advantage of PluginContribDir, is to prefix CheckCommand names to avoid collision. I believe I brought this up a while ago (before this site), but it seems the solution will be up to your implementation.
What did you modify? Is it worth for the community? You can create a pull request for it. If not, try to create an empty file for Icinga’s version, or rename your command.
I just modified the output a little to be more consistent with the rest of all my checks, so not really worth a pull request I think. I ended up renaming my custom command with a custom prefix to avoid the naming conflict with the (now) standard check_systemd command.
