Reload icinga2 process inside container

Im trying to get icinga2 to work inside our kubernetes setup. Changing files e.g inside zones.d is a non-option, so instance needs to be talked to exclusively via the API (config packages). The issue is the reload icinga2 does after validating a new config item.

Here is my limited understanding of whats happening: The reload seems to be triggered by a SIGHUP, after which StartReloadProcess() (in /lib/base/application.cpp) is run and a new process is forked (process->Run(&ReloadProcessCallback) ). This process parent PID will be the PID of its creator. This new process will get an additional flag --reload-internal . The parent processes terminate soon after and on a centos VM, systemd becomes the new parent (PID 1) of the icinga2 process. This reparenting doesn’t seem to work inside a docker container, so the container exits as soon as icinga2 terminates.

I assume the “new” icinga2 process shortly exists as a zombie process and gets SIGKILL . This issue is usually solved by running a init process such as dumb-init or tini inside the container. In this case tough i simply can’t get it to work. No matter the setup, the container simply dies.

This is probably just as much a docker question but if you guys have got a icinga2 docker setup running, help would be greatly appreciated.

I am not much of a Docker/ k8s expert but from what I do know and understand about the forking of process and Docker, I believe that the best way for such a “reload” would be to launch a new container.

Place all the config files in a persistent volume and have the containers mount that volume to read the config, when you want to reload, each container will treat the launch as a “start” but in essence will be a reload, so you gain the benefit of the docker/k8s but have the data safe.

If you have any config issues on startup, you will see the docker failing and restarting, and you can access the logs ( or better yet check them on a vm to see the pass the verification stage).

I’m not really a Docker expert, but from what I hear is that people keep using some workarounds for this. supervisorctl, daemonize mode instead of foreground, etc.

2.11 comes with a new umbrella process solving the latter, you might want to give the RC1 a whirl in your test setup to see whether it solves your problems already.

Cheers,
Michael

Just tried it, the 2.11 RC1 solves my issue.
The first two processes remain even after a reload.

Cool, thanks. You can follow the progress towards 2.11 here.