The Problem
Last week, the icinga2 service on one of our machines hung during a restart. The checks are completing and reporting on the Icinga Web server but the service status returns (the since…15min ago will change but the service never changes to active).
systemctl status -l icinga2
● icinga2.service - Icinga host/service/network monitoring system
Loaded: loaded (/usr/lib/systemd/system/icinga2.service; enabled; vendor preset: disabled)
Active: activating (start) since Tue 2022-08-09 15:50:25 EDT; 15min ago
What We’ve Tried So Far
I have tried to delete the comments as per this post:
cd /var/lib/icinga2/api/packages/_api
ls -la $(cat active-stage)/conf.d/comments
rm $(cat active-stage)/conf.d/comments/*
I have deleted the data under the api directories as per this post and this post:
rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*
The Scope
We only have one virtual machine among >300 that has this issue. I checked the troubleshooting documentation first, but did not find an answer there.
Configuration Information
* Version used: r2.13.4-1
* Operating System and version: CentOS Linux 7 (Core)
* Kernel version: 3.10.0-1160.71.1.el7.x86_64
* Enabled features: api checker mainlog
* Config validation (icinga2 daemon -C)
[2022-08-09 15:59:12 -0400] information/cli: Icinga application loader (version: r2.13.4-1)
[2022-08-09 15:59:12 -0400] information/cli: Loading configuration file(s).
[2022-08-09 15:59:12 -0400] information/ConfigItem: Committing config item(s).
[2022-08-09 15:59:12 -0400] information/ApiListener: My API identity: REDACTED
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 UserGroup.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 User.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 11 Services.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 4 Zones.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 12 Notifications.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 Host.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 2 HostGroups.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 Downtime.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 2 Endpoints.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 FileLogger.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 250 CheckCommands.
[2022-08-09 15:59:12 -0400] information/ConfigItem: Instantiated 1 ApiListener.
[2022-08-09 15:59:12 -0400] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-08-09 15:59:12 -0400] information/cli: Finished validating the configuration file(s).
* Endpoint
Object 'REDACTED' of type 'Endpoint':
% declared in '/etc/icinga2/zones.conf', lines 3:1-3:24
* __name = "REDACTED"
* host = ""
* log_duration = 86400
* name = "REDACTED"
* package = "_etc"
* port = "5665"
* source_location
* first_column = 1
* first_line = 3
* last_column = 24
* last_line = 3
* path = "/etc/icinga2/zones.conf"
* templates = [ "REDACTED" ]
% = modified in '/etc/icinga2/zones.conf', lines 3:1-3:24
* type = "Endpoint"
* zone = ""
Object 'REDACTED' of type 'Endpoint':
% declared in '/etc/icinga2/zones.conf', lines 6:1-6:44
* __name = "REDACTED"
* host = "REDACTED"
% = modified in '/etc/icinga2/zones.conf', lines 7:3-7:22
* log_duration = 86400
* name = "REDACTED"
* package = "_etc"
* port = "5665"
* source_location
* first_column = 1
* first_line = 6
* last_column = 44
* last_line = 6
* path = "/etc/icinga2/zones.conf"
* templates = [ "REDACTED" ]
% = modified in '/etc/icinga2/zones.conf', lines 6:1-6:44
* type = "Endpoint"
* zone = ""
* Zone
% declared in '/etc/icinga2/zones.conf', lines 23:1-23:20
* __name = "master"
* endpoints = [ "REDACTED" ]
% = modified in '/etc/icinga2/zones.conf', lines 24:3-24:47
* global = false
* name = "master"
* package = "_etc"
* parent = ""
* source_location
* first_column = 1
* first_line = 23
* last_column = 20
* last_line = 23
* path = "/etc/icinga2/zones.conf"
* templates = [ "master" ]
% = modified in '/etc/icinga2/zones.conf', lines 23:1-23:20
* type = "Zone"
* zone = ""
Object 'global-commands' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 15:1-15:29
* __name = "global-commands"
* endpoints = null
* global = true
% = modified in '/etc/icinga2/zones.conf', lines 16:3-16:15
* name = "global-commands"
* package = "_etc"
* parent = ""
* source_location
* first_column = 1
* first_line = 15
* last_column = 29
* last_line = 15
* path = "/etc/icinga2/zones.conf"
* templates = [ "global-commands" ]
% = modified in '/etc/icinga2/zones.conf', lines 15:1-15:29
* type = "Zone"
* zone = ""
* Application information:
* General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
* Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid