Im trying to get icinga2 to work inside our kubernetes setup. Changing files e.g inside zones.d is a non-option, so instance needs to be talked to exclusively via the API (config packages). The issue is the reload icinga2 does after validating a new config item.
Here is my limited understanding of whats happening: The reload seems to be triggered by a SIGHUP, after which StartReloadProcess() (in /lib/base/application.cpp) is run and a new process is forked (process->Run(&ReloadProcessCallback) ). This process parent PID will be the PID of its creator. This new process will get an additional flag --reload-internal . The parent processes terminate soon after and on a centos VM, systemd becomes the new parent (PID 1) of the icinga2 process. This reparenting doesn’t seem to work inside a docker container, so the container exits as soon as icinga2 terminates.
I assume the “new” icinga2 process shortly exists as a zombie process and gets SIGKILL . This issue is usually solved by running a init process such as dumb-init or tini inside the container. In this case tough i simply can’t get it to work. No matter the setup, the container simply dies.
This is probably just as much a docker question but if you guys have got a icinga2 docker setup running, help would be greatly appreciated.
I am not much of a Docker/ k8s expert but from what I do know and understand about the forking of process and Docker, I believe that the best way for such a “reload” would be to launch a new container.
Place all the config files in a persistent volume and have the containers mount that volume to read the config, when you want to reload, each container will treat the launch as a “start” but in essence will be a reload, so you gain the benefit of the docker/k8s but have the data safe.
If you have any config issues on startup, you will see the docker failing and restarting, and you can access the logs ( or better yet check them on a vm to see the pass the verification stage).
I’m not really a Docker expert, but from what I hear is that people keep using some workarounds for this. supervisorctl, daemonize mode instead of foreground, etc.
2.11 comes with a new umbrella process solving the latter, you might want to give the RC1 a whirl in your test setup to see whether it solves your problems already.