Active Checks stopped working after 2.11 upgrade (from 2.8)

Icinga 2
1

Hey,

I’ve upgraded my sattelite/master from 2.8 to 2.11 and noticed all of my Active Checks stop working - none of them is executing.

Both of my servers are on same version:

icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.2-1)

My zones hierarchy

object Zone “master” {
endpoints = [ “icinga2-master-1c” ]
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}
object Endpoint “UAT2-stp-1a.nova.local” {
host = “172.26.43.41”
}
object Zone “UAT2” {
parent = “master”
endpoints = [ “UAT2-stp-1a.nova.local” ]
}

My host definition that calls an active check is stored in /etc/icinga2/conf.d/<some_path_here>

On my sattelite I can see event that forcing check is received from master:
[2020-02-14 15:54:52 +0800] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2020-02-14 15:54:52 +0800] notice/ApiListener: Setting log position for identity ‘icinga2-master-1c’: 2020/02/14 15:47:18
[2020-02-14 15:54:57 +0800] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-02-14 15:54:57 +0800] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-02-14 15:54:57 +0800] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2020-02-14 15:54:57 +0800] notice/ApiListener: Setting log position for identity ‘icinga2-master-1c’: 2020/02/14 15:47:18
[2020-02-14 15:54:57 +0800] notice/ApiListener: Updating object authority for objects at endpoint ‘UAT2-stp-1a.nova.local’.
[2020-02-14 15:54:57 +0800] debug/ApiListener: Not connecting to Endpoint ‘UAT2-stp-1a.nova.local’ because that’s us.
[2020-02-14 15:54:57 +0800] debug/ApiListener: Not connecting to Endpoint ‘icinga2-master-1c’ because we’re already connected to it.
[2020-02-14 15:54:57 +0800] notice/ApiListener: Current zone master: UAT2-stp-1a.nova.local
[2020-02-14 15:54:57 +0800] notice/ApiListener: Connected endpoints: icinga2-master-1c (1)
[2020-02-14 15:54:57 +0800] notice/JsonRpcConnection: Received ‘event::Heartbeat’ message from identity ‘icinga2-master-1c’.
[2020-02-14 15:54:57 +0800] notice/JsonRpcConnection: Received ‘event::SetForceNextCheck’ message from identity ‘icinga2-master-1c’.
[2020-02-14 15:54:57 +0800] notice/JsonRpcConnection: Received ‘event::SetNextCheck’ message from identity ‘icinga2-master-1c’.
[2020-02-14 15:55:02 +0800] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2020-02-14 15:55:02 +0800] notice/ApiListener: Setting log position for identity ‘icinga2-master-1c’: 2020/02/14 15:54:57
[2020-02-14 15:55:07 +0800] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2020-02-14 15:55:07 +0800] notice/ApiListener: Setting log position for identity ‘icinga2-master-1c’: 2020/02/14 15:54:57
[2020-02-14 15:55:07 +0800] notice/ApiListener: Updating object authority for objects at endpoint ‘UAT2-stp-1a.nova.local’.
[2020-02-14 15:55:07 +0800] debug/ApiListener: Not connecting to Endpoint ‘UAT2-stp-1a.nova.local’ because that’s us.
[2020-02-14 15:55:07 +0800] debug/ApiListener: Not connecting to Endpoint ‘icinga2-master-1c’ because we’re already connected to it.
[2020-02-14 15:55:07 +0800] notice/ApiListener: Current zone master: UAT2-stp-1a.nova.local
[2020-02-14 15:55:07 +0800] notice/ApiListener: Connected endpoints: icinga2-master-1c (1)

But nothing is received back on master, any ideas - am I done something wrong with zones design and that’s the reason for it? Of course check_command etc exists (linux load) same like all binaries etc. Icinga2 Web interface simply counting to new check execution, doing nothing and last checked is date in the past (when I upgraded).

Thanks
D

2

Hi,

If that host does a command_endpoint check, it needs a zone. The configuration in conf.d does not have such. The easiest way to fix this is

mkdir -p /etc/icinga2/zones.d/master
mv /etc/icinga2/conf.d/<hostpath>.conf /etc/icinga2/zones.d/master
systemctl reload icinga2

The change is mentioned here: https://icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/#agent-hosts-with-command-endpoint-require-a-zone

Cheers,
Michael

3

Hi @dnsmichi

I saw this change and tried this - no luck.

Now moved all content of conf.d into zones/master and still no luck.

Thanks
D

4

Hi,

then please proceed in the troubleshooting docs with why checks are not executed. Specifically, we are interested in the full object configuration from icinga2 object list ... and its runtime state via the REST API.

https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#checks-are-not-executed

Cheers,
Michael

5

Hi @dnsmichi

I’ve done all basics - ex. confirmed binary with correct permissions, investigated debug log (not executed) and went through other steps.

Here you have Host & Endpoint objects: https://pastebin.com/m0cB0Csr
Here you have Service endpoint: https://pastebin.com/sUxWHvtd

As mentioned moved whole config to zones.d folder and my zones are declared in zones.conf

Thanks
Dariusz