Satellite Node stuck in PENDING

Icinga Director
1

Hi icinga team,

I have a satellite node that just sits in PENDING after being deployed using ‘director’. All other hosts I have defined are working fine, including the other satellites.

The current FAQ answers for ‘node stuck in pending’ do not provide me with a clear answer.

Infos

icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.3-1)
Latest icingaweb2, director, ipl, and all that’s required for installation

zones.d/awsir-sdmon-satellite/hosts.conf
object Host "awsir-sdmon-sat01.somedomain.net" {
display_name = "awsir-sdmon-sat01"
address = "awsir-sdmon-sat01.somedomain.net"
check_command = "icinga"
max_check_attempts = "3"
check_interval = 5m
enable_notifications = true
enable_active_checks = true
enable_passive_checks = true
enable_perfdata = true
volatile = false
zone = "awsir-sdmon-satellite"
notes = "Host with the icinga2 agent installed"
icon_image = "tux.png"
groups = [ "sdmonitoring" ]

}

It is the endpoint for a satellite zone:

object Zone "awsir-sdmon-satellite" {
parent = "master"
endpoints = [ "awsir-sdmon-sat01.somedomain.net" ]
}
object Endpoint "awsir-sdmon-sat01.somedomain.net" {
host = "awsir-sdmon-sat01.somedomain.net"
port = "5665"
log_duration = 1d
}

Output of

  icinga2 object list --type Host --name awsir-sdmon-sat01.somedomain.net

(somedomain.net is text replacement in this file for the real domain)

  enable_perfdata = true
  % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469- 61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 8:5-8:26
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ "sdmonitoring" ]
  % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/awsir-sdmon-satellite/hosts.conf', lines 6:5-6:31
  * icon_image = "tux.png"
  % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 32:5-32:26
  * icon_image_alt = ""
  * max_check_attempts = 3
  % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 3:5-3:28
  * name = "awsir-sdmon-sat01.somedomain.net"
  * notes = "Host with the icinga2 agent installed"
  % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 31:5-31:51
    * notes_url = ""
    * package = "director"
    * retry_interval = 60
    * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 41
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/awsir-sdmon-satellite/hosts.conf"
    * templates = [ "awsir-sdmon-sat01.somedomain.net", "Icinga Satellite", "Base Template" ]
    % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/awsir-sdmon-satellite/hosts.conf', lines 1:0-1:41
    % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 27:1-27:32
    % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 1:0-1:28
    * type = "Host"
    * vars = null
    * volatile = false
    % = modified in '/var/lib/icinga2/api/packages/director/48351e56-3d8a-4ed2-8469-61a1a9c6c33a/zones.d/director-global/host_templates.conf', lines 9:5-9:20
    * zone = "awsir-sdmon-satellite"

Can anyone suggest actions/solution.

Kind regards
Peter Goudman

2

Over a week and no one to help?
I notice on the satellite it keeps going around in an infinite configuration loop?

I have no idea why that would be happening.

Does this stuff really work?

3

Did you check icinga’s logfiles? Common errors are typos, hostname/zone/endpoint mismatch not only because they are case sensitive. Is the agent’s certificate signed?

4

Hello Roland.

I managed to end the infinite loop by stopping the agent, and deleting:

/var/lib/icinga2/api/zones
/var/lib/icinga2/api/zones-stage

The satellite node just stays in ‘pending’ state on icingaweb. (I also use director)
icinga2 version: 2.11.3-1

I have everything set up for HA Master and a number of satellite zones. All nodes are running the icinga2 agent.

The satellite certs were signed using a ticket when I ran the icinga node wizard.

Here is my zone.conf files (no typos that I can see).

awsir-sdmon-sat01.somedomain.net
 #####################################
/etc/icinga2/zones.conf

object Endpoint "enf-sdmon-mstr01.somedomain.net" {
        // This endpoint will connnect to us
}
object Endpoint "awsir-sdmon-mstr02.somedomain.net" {
        // This endpoint will connnect to us
}
object Endpoint "awsir-sdmon-sat01.somedomain.net" {
        // This server
}
object Zone "master" {
        endpoints = [  "enf-sdmon-mstr01.somedomain.net", "awsir-sdmon-mstr02.somedomain.net" ]
}
object Zone "awsir-sdmon-satellite" {
        endpoints = [ "awsir-sdmon-sat01.somedomain.net" ]
        parent = "master"
}
object Zone "global-templates" {
        global = true
}
object Zone "director-global" {
        global = true
}

enf-sdmon-mstr01.somedomain.net
#####################################
/etc/icinga2/zones.conf

// Primary master
object Endpoint "enf-sdmon-mstr01.somedomain.net" {
  // This server
}
// Secondary master
object Endpoint "awsir-sdmon-mstr02.somedomain.net" {
  // Actively connect to the secondary master
  host = "awsir-sdmon-mstr02.somedomain.net"
}
// Satellite Endpoints
object Endpoint "enf-emea01-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "enf-emea01-sat01.somedomain.net"
}
object Endpoint "enf-emea02-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "enf-emea02-sat01.somedomain.net"
}
object Endpoint "awsir-sdmon-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "awsir-sdmon-sat01.somedomain.net"
}
// MASTER ZONE
object Zone "master" {
  endpoints = [ "enf-sdmon-mstr01.somedomain.net", "awsir-sdmon-mstr02.somedomain.net" ]
}
// ENF EMEA01 Satellite
object Zone "enf-emea01-satellite" {
  endpoints = [ "enf-emea01-sat01.somedomain.net" ]
  parent = "master"
}
// ENF EMEA02 Satellite
object Zone "enf-emea02-satellite" {
  endpoints = [ "enf-emea02-sat01.somedomain.net" ]
  parent = "master"
// AWSIR Satellite
object Zone "awsir-sdmon-satellite" {
  endpoints = [ "awsir-sdmon-sat01.somedomain.net" ]
  parent = "master"
}
object Zone "global-templates" {
        global = true
}
object Zone "director-global" {
        global = true

awsir-sdmon-mstr02.somedomain.net
#####################################
/etc/icinga2/zones.conf

// primary master
object Endpoint "enf-sdmon-mstr01.somedomain.net" {
  // First master already connects to us
}
// Secondary master
object Endpoint "awsir-sdmon-mstr02.somedomain.net" {
  // This server

}
// Satellite Endpoints
object Endpoint "enf-emea01-sat01.somedomain.net" {
  // Actively connect to satellite
//  host = "enf-emea01-sat01.somedomain.net"
}
object Endpoint "enf-emea02-sat01.somedomain.net" {
  // Actively connect to satellite
//  host = "enf-emea02-sat01.somedomain.net"
}
object Endpoint "awsir-sdmon-sat01.somedomain.net" {
  // Actively connect to satellite
//  host = "awsir-sdmon-sat01.somedomain.net"
}
// MASTER ZONE
object Zone "master" {
  endpoints = [ "enf-sdmon-mstr01.somedomain.net", "awsir-sdmon-mstr02.somedomain.net" ]
}
// ENF EMEA01 Satellite
object Zone "enf-emea01-satellite" {
  endpoints = [ "enf-emea01-sat01.somedomain.net" ]
  parent = "master"
}
// ENF EMEA02 Satellite
object Zone "enf-emea02-satellite" {
  endpoints = [ "enf-emea02-sat01.somedomain.net" ]
  parent = "master"
}
// AWSIR Satellite
object Zone "awsir-sdmon-satellite" {
  endpoints = [ "awsir-sdmon-sat01.somedomain.net" ]
  parent = "master"
}
object Zone "global-templates" {
        global = true
}
object Zone "director-global" {
        global = true
}

Log file from Satellite
##################

   [2020-07-14 10:20:38 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);                                                                                                 [0/5684]
    [2020-07-14 10:24:00 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
    [2020-07-14 10:25:48 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
    [2020-07-14 10:25:48 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
    [2020-07-14 10:26:55 +0000] warning/JsonRpcConnection: API client disconnected for identity 'enf-sdmon-mstr01.xxxxxx.net'
    [2020-07-14 10:26:55 +0000] warning/ApiListener: Removing API client for endpoint 'enf-sdmon-mstr01.xxxxxx.net'. 0 API clients left.
    [2020-07-14 10:26:55 +0000] information/ApiListener: New client connection for identity 'enf-sdmon-mstr01.xxxxxx.net' from [10.44.136.61]:42030
    [2020-07-14 10:26:55 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'enf-sdmon-mstr01.xxxxxx.net'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Sending config updates for endpoint 'enf-sdmon-mstr01.xxxxxx.net' in zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Finished sending config file updates for endpoint 'enf-sdmon-mstr01.xxxxxx.net' in zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Syncing runtime objects to endpoint 'enf-sdmon-mstr01.xxxxxx.net'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'enf-sdmon-mstr01.xxxxxx.net'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'enf-sdmon-mstr01.xxxxxx.net' in zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Sending replay log for endpoint 'enf-sdmon-mstr01.xxxxxx.net' in zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Finished sending replay log for endpoint 'enf-sdmon-mstr01.xxxxxx.net' in zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Finished syncing endpoint 'enf-sdmon-mstr01.xxxxxx.net' in zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Applying config update from endpoint 'enf-sdmon-mstr01.xxxxxx.net' of zone 'master'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Received configuration for zone 'awsir-sdmon-satellite' from endpoint 'enf-sdmon-mstr01.xxxxxx.net'. Comparing the timestamp and checksums.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite//director/hosts.conf' for zone 'awsir-sdmon-satellite'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite' (299 Bytes).
    [2020-07-14 10:26:55 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'enf-sdmon-mstr01.xxxxxx.net'. Comparing the timestamp and checksums.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/hostgroups.conf' for zone 'director-global'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_apply.conf' for zone 'director-global'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
    [2020-07-14 10:26:55 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (3431 Bytes).
    [2020-07-14 10:26:55 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'enf-sdmon-mstr01.xxxxxx.net' do not qualify for production, not triggering reload.
    [2020-07-14 10:29:00 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
    [2020-07-14 10:30:58 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
    [2020-07-14 10:30:58 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);

Thanks you for replying, I am stuck as to why this node never goes from pending state on icingaweb2.

Btw, this is a fresh installation, everything installed is latests. I can delete dir’s if needed as I do not have any other nodes set up yet. I really want to see all my satelliltes up and having checks performed before setting up other nodes.

Peter

5

just as a try:

systemctl stop icinga2
rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*
systemctl start icinga2
6

Maybe you are affected by a bug which was fixed with V2.11.4. So I’d recommend to update anyway (but please keep in mind the version constrains).

7

Hi Roland,

tried that, here is the icinga2.log output: -

 [2020-07-14 13:04:02 +0000] information/FileLogger: 'main-log' started.
 [2020-07-14 13:04:02 +0000] information/FileLogger: 'debug-file' started.
 [2020-07-14 13:04:02 +0000] information/ApiListener: 'api' started.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
 [2020-07-14 13:04:02 +0000] information/ConfigItem: Activated all objects.
 [2020-07-14 13:04:02 +0000] information/ApiListener: New client connection for identity 'awsir-sdmon-mstr02.######.net' from [172.31.19.165]:40858
 [2020-07-14 13:04:02 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'awsir-sdmon-mstr02.######.net'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Sending config updates for endpoint 'awsir-sdmon-mstr02.######.net' in zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Finished sending config file updates for endpoint 'awsir-sdmon-mstr02.######.net' in zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Syncing runtime objects to endpoint 'awsir-sdmon-mstr02.######.net'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'awsir-sdmon-mstr02.######.net'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'awsir-sdmon-mstr02.######.net' in zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Sending replay log for endpoint 'awsir-sdmon-mstr02.######.net' in zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Finished sending replay log for endpoint 'awsir-sdmon-mstr02.######.net' in zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Finished syncing endpoint 'awsir-sdmon-mstr02.######.net' in zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Applying config update from endpoint 'awsir-sdmon-mstr02.######.net' of zone 'master'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Received configuration for zone 'awsir-sdmon-satellite' from endpoint 'awsir-sdmon-mstr02.######.net'. Comparing the timestamp and checksums.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite'
 . Current timestamp '2020-07-14 13:01:49 +0000' (1594731709.533785) >= received timestamp '2020-07-14 13:01:49 +0000' (1594731709.533785).
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite//director/hosts.conf' for zone 'awsir-sdmon-satellite'.                             [2020-07-14 13:04:02 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite' (254 Bytes).
 [2020-07-14 13:04:02 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'awsir-sdmon-mstr02.######.net'. Comparing the timestamp and checksums.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global'. Curr
 ent timestamp '2020-07-14 13:01:49 +0000' (1594731709.531988) >= received timestamp '2020-07-14 13:01:49 +0000' (1594731709.531988).
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/hostgroups.conf' for zone 'director-global'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_apply.conf' for zone 'director-global'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
 [2020-07-14 13:04:02 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (3681 Bytes).
 [2020-07-14 13:04:02 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'awsir-sdmon-mstr02.######.net' do not qualify for production, not triggering reload.
 [2020-07-14 13:04:09 +0000] information/ApiListener: New client connection for identity 'enf-sdmon-mstr01.######.net' from [10.44.136.61]:50292
 [2020-07-14 13:04:09 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'enf-sdmon-mstr01.######.net'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Sending config updates for endpoint 'enf-sdmon-mstr01.######.net' in zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Finished sending config file updates for endpoint 'enf-sdmon-mstr01.######.net' in zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Syncing runtime objects to endpoint 'enf-sdmon-mstr01.######.net'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'enf-sdmon-mstr01.######.net'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'enf-sdmon-mstr01.######.net' in zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Sending replay log for endpoint 'enf-sdmon-mstr01.######.net' in zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Finished sending replay log for endpoint 'enf-sdmon-mstr01.######.net' in zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Finished syncing endpoint 'enf-sdmon-mstr01.######.net' in zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Applying config update from endpoint 'enf-sdmon-mstr01.######.net' of zone 'master'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Received configuration for zone 'awsir-sdmon-satellite' from endpoint 'enf-sdmon-mstr01.######.net'. Comparing the timestamp and checksums.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite'
 . Current timestamp '2020-07-14 13:01:49 +0000' (1594731709.533785) >= received timestamp '2020-07-14 13:01:49 +0000' (1594731709.533785).
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite//director/hosts.conf' for zone 'awsir-sdmon-satellite'.                             [2020-07-14 13:04:09 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/awsir-sdmon-satellite' (254 Bytes).
 [2020-07-14 13:04:09 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'enf-sdmon-mstr01.######.net'. Comparing the timestamp and checksums.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global'. Current timestamp '2020-07-14 13:01:49 +0000' (1594731709.531988) >= received timestamp '2020-07-14 13:01:49 +0000' (1594731709.531988).
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/hostgroups.conf' for zone 'director-global'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_apply.conf' for zone 'director-global'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
 [2020-07-14 13:04:09 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (3681 Bytes).
 [2020-07-14 13:04:09 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'enf-sdmon-mstr01.######.net' do not qualify for production, not triggering reload.
 [2020-07-14 13:04:11 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.05/s (3/min 3/5min 3/15min);

Node still displays as pending in icingaweb2.
image

Services as pending well, there is an exclamation mark next to the node, but I have no idea why.

image
image913×153 10.5 KB

I inspected the node in icingaweb2, here is the output: -

      hosts "awsir-sdmon-sat01.######.net"
      Inspected via enf-sdmon-mstr01.######.net (https://director@enf-sdmon-mstr01.######.net:5665/v1/)

      Source Location
      The configuration for this object has been rendered by Icinga Director 10m 50s ago to zones.d/awsir-sdmon-satellite/hosts.conf:1

      Object Properties
      __name        "awsir-sdmon-sat01.######.net"
      acknowledgement       0
      acknowledgement_expiry        0
      action_url    ""
      active        true
      address       "10.70.23.99"
      address6      ""
      check_attempt 1
      check_command ssh
      check_interval        120
      check_period  ""
      check_timeout null
      command_endpoint      ""
      display_name  "awsir-sdmon-sat01"
      downtime_depth        0
      enable_active_checks  true                                                                                                                                                                                                                   enable_event_handler  false
      enable_flapping       false
      enable_notifications  true                                                                                                                                                                                                                   enable_passive_checks true
      enable_perfdata       true
      event_command ""                                                                                                                                                                                                                             flapping      false
      flapping_current      0
      flapping_last_change  0
      flapping_threshold    0
      flapping_threshold_high       30
      flapping_threshold_low        25
      force_next_check      false
      force_next_notification       false
      groups        sdmonitoring
      ha_mode       0
      handled       false
      icon_image    ""
      icon_image_alt        ""
      last_check    -1
      last_hard_state       1
      last_hard_state_change        1594728414.156514
      last_reachable        true
      last_state    1
      last_state_change     1594728414.156514
      last_state_down       0
      last_state_type       0
      last_state_unreachable        0
      last_state_up 0
      max_check_attempts    3
      name  "awsir-sdmon-sat01.######.net"
      next_check    1594731759.940619
      notes ""
      notes_url     ""
      original_attributes   null
      package       "director"
      paused        true
      previous_state_change 1594728414.156514
      problem       true
      retry_interval        120
      severity      24
      state 1
      state_type    0
      type  "Host"
      vars  null
      version       0
      volatile      false
      zone  "awsir-sdmon-satellite"
8

Hi Roland,

regarding the upgrade, I installed using this URL: https://icinga.com/docs/icinga2/latest/doc/02-installation/ which installs the version I currently have 2.11.3-1. How do I get version 2.11.4? Is there another repository for centos7?

Cheers
Peter

9

It’s always recommend to get packages directly from icinga.com in your case from here.

10

Yes that’s where I fetch them from using yum on my centos7 server. But no version 2.11.4 is listed When you install from that repo you get v2.11.3-1.

How do I get v2.11.4?

I tried this…

  [root@enf-sdmon-mstr01 ~]# yum update icinga2
  Loaded plugins: fastestmirror, rhnplugin
  This system is receiving updates from RHN Classic or Red Hat Satellite.
  Loading mirror speeds from cached hostfile
  centos-7-epel-main                                                                              | 1.0 kB  00:00:00
  centos-7-plus-x86_64                                                                            |  871 B  00:00:00
  centos_7                                                                                        | 1.1 kB  00:00:00
  icinga2                                                                                         |  871 B  00:00:00
  icingaweb2                                                                                      |  871 B  00:00:00
  jpackage-generic                                                                                |  871 B  00:00:00
  puppetlabs-el7-x86_64                                                                           |  871 B  00:00:00
  No packages marked for update

Thanks

11

The packages are here but I have no experience with CentOS.If you don’t get an answer here it might be a good idea to open an issue here.

12

Hi.
Just a small note (maybe i missed something):
The icingaweb2 output doesn’t match the host-config from here

For example:

This may be desired, but I just wanted to point out.

Greetings.

13

Hi Roland and Icinga Team,

as suggested, I have upgraded all my icinga2 instances to…

icinga2 -V
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.4-1)

Here is the current status of the satellite that stays in PENDING.
image

I am at a loss as to what to check/delete/upgrade next.

I was trying to make a case for using the Icinga Agent instead of nrpe, but this is not looking good, two weeks in pending with no checks being performed.

Desperate now, can anyone offer suggestions/help?

Thanks
Peter

14

I finally found the issue. After checking against a satellite that was working, I noticed the ‘checker’ feature was not enabled on the satellite stuck in pending.

Executing:

icinga2 feature enable checker
systemctl restart icinga2

After less that a minute the node was no longer in PENDING but now executing checks.

1 Like