Failed to start Icinga host/service/network monitoring system. 'activating (start) since…'

The Problem

Last week, the icinga2 service on one of our machines hung during a restart. The checks are completing and reporting on the Icinga Web server but the service status returns (the since…15min ago will change but the service never changes to active).

systemctl status -l icinga2
● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
   Active: activating (start) since Tue 2022-08-09 15:50:25 EDT; 15min ago

What We’ve Tried So Far

I have tried to delete the comments as per this post:

cd /var/lib/icinga2/api/packages/_api
ls -la $(cat active-stage)/conf.d/comments
rm $(cat active-stage)/conf.d/comments/*

I have deleted the data under the api directories as per this post and this post:

rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*

The Scope

We only have one virtual machine among >300 that has this issue. I checked the troubleshooting documentation first, but did not find an answer there.

Configuration Information

* Version used: r2.13.4-1

* Operating System and version: CentOS Linux 7 (Core)

* Kernel version: 3.10.0-1160.71.1.el7.x86_64

* Enabled features: api checker mainlog

* Config validation (icinga2 daemon -C)

[2022-08-09 15:59:12 -0400] information/cli: Icinga application loader (version: r2.13.4-1)
[2022-08-09 15:59:12 -0400] information/cli: Loading configuration file(s).
[2022-08-09 15:59:12 -0400] information/ConfigItem: Committing config item(s).
[2022-08-09 15:59:12 -0400] information/ApiListener: My API identity: REDACTED
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 UserGroup.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 User.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 11 Services.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 4 Zones.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 12 Notifications.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 Host.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 2 HostGroups.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 Downtime.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 2 Endpoints.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 FileLogger.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 250 CheckCommands.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 ApiListener.
[2022-08-09 15:59:12 -0400] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-08-09 15:59:12 -0400] information/cli: Finished validating the configuration file(s).

* Endpoint

Object 'REDACTED' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 3:1-3:24
  * __name = "REDACTED"
  * host = ""
  * log_duration = 86400
  * name = "REDACTED"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 3
    * last_column = 24
    * last_line = 3
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "REDACTED" ]
    % = modified in '/etc/icinga2/zones.conf', lines 3:1-3:24
  * type = "Endpoint"
  * zone = ""

Object 'REDACTED' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 6:1-6:44
  * __name = "REDACTED"
  * host = "REDACTED"
    % = modified in '/etc/icinga2/zones.conf', lines 7:3-7:22
  * log_duration = 86400
  * name = "REDACTED"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 44
    * last_line = 6
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "REDACTED" ]
    % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:44
  * type = "Endpoint"
  * zone = ""

* Zone

  % declared in '/etc/icinga2/zones.conf', lines 23:1-23:20
  * __name = "master"
  * endpoints = [ "REDACTED" ]
    % = modified in '/etc/icinga2/zones.conf', lines 24:3-24:47
  * global = false
  * name = "master"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 23
    * last_column = 20
    * last_line = 23
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "master" ]
    % = modified in '/etc/icinga2/zones.conf', lines 23:1-23:20
  * type = "Zone"
  * zone = ""

Object 'global-commands' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 15:1-15:29
  * __name = "global-commands"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 16:3-16:15
  * name = "global-commands"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 29
    * last_line = 15
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-commands" ]
    % = modified in '/etc/icinga2/zones.conf', lines 15:1-15:29
  * type = "Zone"
  * zone = ""

* Application information:

* General paths:

Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

* Internal paths:

Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

1 Like