Services in Pending state - Need help to find the root cause

Some of the services are stuck in the “Pending” state and I need help to find the root cause. The zones they are in are connected and many other servces in those zones are “Ok” “Check-now” didn’t help nor restarting Icinga2 service on satellites and the masters. I increased the MaxConsurrentCheck on the satellites constant.conf but that didn’t help.

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.15.0-1)

Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 24.04.3 LTS (Noble Numbat)
  Kernel: Linux
  Kernel version: 6.8.0-78-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 13.3.0
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 3.0.13 30 Jan 2024

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
# icinga2 daemon -C
[2025-09-05 13:16:18 -0700] information/cli: Icinga application loader (version: r2.15.0-1)
[2025-09-05 13:16:18 -0700] information/cli: Loading configuration file(s).
[2025-09-05 13:16:18 -0700] information/ConfigItem: Committing config item(s).
[2025-09-05 13:16:18 -0700] information/ApiListener: My API identity: x.y.com
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 126526 Notifications.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 7 NotificationCommands.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 57 HostGroups.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 8017 Hosts.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 3 Downtimes.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 FileLogger.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 9 Zones.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 14 Endpoints.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 3 ApiUsers.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 ApiListener.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 383 CheckCommands.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 133 Users.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 110 UserGroups.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 108 ServiceGroups.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 14 TimePeriods.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 48190 Services.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 4 ScheduledDowntimes.
[2025-09-05 13:16:28 -0700] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-09-05 13:16:28 -0700] information/cli: Finished validating the configuration file(s).

On satellites, ‘icinga2 object list --type Host ’ doesn’t return any result for the hosts having services in "Pending’ whereas it does on masters and for other hosts. No relevant info in the debug log that I could find.

Never mind. I got it fixed. It was a config sync issue. I had to remove content in /var/lib/icinga2/api/zones and restart icinga2 service on the satellites.

1 Like