Icinga2 issue when deploying new changes

Icinga 2
1

Hello,

After upgrading our icinga2 to the version : 2.12.4 we are facing an issue with the deployment of new configurations…

About our monitoring : we have 2 pollers in a zone (master - master) with 1 deployment endpoint. We have 1 web server that host icingaweb2 and icingadirector and we have a database server that host all of our databases for icinga, director… All these servers are on Gentoo Linux, icingaweb2 is on version 2.11.3 and icingadirector in 1.8.0.

Since the update the behaviour is like this :

Normal run : both icinga2 services are fine, 3 process per server are working properly and everything is fine.

When we push our first modifications via icingadirector, new processes are created on both icinga2 servers and replace the old process as it should be according to the documentation.

But after that first deployment our issue kick in: if we push a second configuration via icingadirector, some process are created BUT then are stucked… the only solution is to “kill -9” on thoses “stuck” processes to get the deployment back…

Can anyone help debug this issue please ?

Thanks.

1 Like
2

Hello,

Up anyone ? Still stuck on this issue

Thanks

3

Maybe give us a bit of a clue by telling us what these “stuck” processes
actually are and which machine/s in your setup they are running on.

Also, what appears in the icinga logs when this problem occurs?

Antony.

1 Like
4

Hello Antony,

When we push our configuration with icinga director, 2 new processes are created on our pollers (machines with the icinga2 package) so these stuck processes are running on them. According to icingaDirector documentation (How It Works - Icinga Director in section “How your configuration is going to be rendered”) it seems to me that when a new stage (configuration) is deployed, Icinga would launch new processes using the newest changes. Then icinga would kill the old process (using the old configuration).

We suspect this phase is not going as its should. Icinga is not being able to kill the old processes. Therefore leaving us with stuck “old process” and the new process.

Regarding the logs, it’s more likely what is not happening. The only difference between a deployment that is good and a bad one is the following :

[2021-08-16 11:35:16 +0200] information/Application: Received request to shut down.
[2021-08-16 11:35:17 +0200] information/Application: Shutting down…
[2021-08-16 11:35:17 +0200] information/CheckerComponent: ‘checker’ stopped.
[2021-08-16 11:35:17 +0200] information/NotificationComponent: ‘notification’ stopped.
[2021-08-16 11:35:17 +0200] information/GraphiteWriter: ‘graphite’ paused.
[2021-08-16 11:35:17 +0200] information/DbConnection: ‘ido-mysql’ stopped.
[2021-08-16 11:35:17 +0200] information/ApiListener: ‘api’ stopped.
[2021-08-16 11:35:17 +0200] information/FileLogger: ‘main-log’ started.
[2021-08-16 11:35:18 +0200] information/ApiListener: ‘api’ started.
[2021-08-16 11:35:18 +0200] information/ApiListener: Copying 11 zone configuration files for zone ‘director-global’ to ‘/var/lib/icinga2/api/zones/director-global’.
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/001-director-basics.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/commands.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/dependency_templates.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/host_templates.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/hostgroups.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/service_apply.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/service_templates.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/servicegroups.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/servicesets.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/timeperiods.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/director-global//director/user_templates.conf
[2021-08-16 11:35:18 +0200] information/ApiListener: Copying 8 zone configuration files for zone ‘’ to '/var/lib/icinga2/api/zones/’.
[2021-08-16 11:35:18 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/*******//_etc/commands.conf

It’s there when it’s Ok but not there when processes are stuck.

Thanks for the help.

5

Hello,

Returning from vacation today, our issue is still here.

Any idea on this ?

Thanks

6

Hello there,
I would like to ask you to refrain from bumping topics this way. This forum works on the basis of people answering because they want to help and bumping is generally seen as rude.
If you don’t get a response to a topic, it’s usually because there wasn’t anyone who could help you at this point.
If you need more help with a topic and want to put it on the front page again, a good way to do so is giving an update on what you have tried since your last post to give people a few pointers.
Maybe there will be someone able to help you from there.