Icinga2 stuck on reloading in the master node, after deploying

Icinga 2
1

Hi Icinga Community :upside_down_face:

since some time now, I am experiencing the following situation:
after a couple of deployments (using the director module) in a time period of less than an hour, the icinga2 service on the master node seems to not being able to reload anymore. Thus, the reloading operation times out and the reload process is killed as a result.
As a consequence, the icingadb service fails after waiting a long time for Icinga heartbeat.

Thanks a lot in advance for any help provided!

Icinga2 version used: r2.14.2-1
Operating System and version: Debian 12.5

2

what’s the output of

systemctl status icinga2

icinga2 daemon -C

3

Output of systemctl status icinga2 :

icinga2.service - Icinga host/service/network monitoring system
     Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/icinga2.service.d
             └─limits.conf, override.conf
     Active: active (running) since Mon 2024-06-24 07:41:13 CEST; 5h 7min ago
    Process: 54747 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
   Main PID: 54752 (icinga2)
     Status: "Startup finished."
      Tasks: 38
     Memory: 1.1G
        CPU: 32min 12.248s
     CGroup: /system.slice/icinga2.service
             ├─54752 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
             ├─93192 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log
             └─93201 /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon --close-stdio -e /var/log/icinga2/error.log

Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ConfigItem: Instantiated 1 ApiListener.
Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ConfigItem: Instantiated 270 CheckCommands.
Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ConfigItem: Instantiated 15 ServiceGroups.
Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ConfigItem: Instantiated 2 TimePeriods.
Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ConfigItem: Instantiated 8220 Services.
Jun 24 07:41:13 xxxxxxxx icinga2[54778]: [2024-06-24 07:41:13 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
Jun 24 07:41:13 xxxxxxxx icinga2[54752]: [2024-06-24 07:41:13 +0200] information/cli: Closing console log.
Jun 24 07:41:13 xxxxxxxx systemd[1]: Started icinga2.service - Icinga host/service/network monitoring system.
Jun 24 11:46:20 xxxxxxxx systemd[1]: icinga2.service: Reload operation timed out. Killing reload process.

Output of icinga2 daemon -C :

[2024-06-27 12:48:20 +0200] information/cli: Icinga application loader (version: r2.14.2-1)
[2024-06-27 12:48:20 +0200] information/cli: Loading configuration file(s).
[2024-06-27 12:48:20 +0200] information/ConfigItem: Committing config item(s).
[2024-06-27 12:48:20 +0200] information/ApiListener: My API identity: xxxxxxx 
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 NotificationCommand.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 8428 Notifications.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 15 HostGroups.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 783 Hosts.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 Influxdb2Writer.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 751 Zones.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 2 Users.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 760 Endpoints.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 270 CheckCommands.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 15 ServiceGroups.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 2 TimePeriods.
[2024-06-27 12:48:21 +0200] information/ConfigItem: Instantiated 8428 Services.
[2024-06-27 12:48:21 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2024-06-27 12:48:21 +0200] information/cli: Finished validating the configuration file(s).
4

In case the file /var/lib/icinga2/api/zones-stage-startup-last-failed.log exists, check the content.

Have you enabled the debug log and checked that?

1 Like
5

The debug log is enabled, but the file /var/lib/icinga2/api/zones-stage-startup-last-failed.log does not exist.