New host is always in PENDING state

Icinga 2
1

Hello,

I’m new to icinga and I’m working with an already deployed installation that is running icinga version 2.8.1 (master and agents).

We added a new host and it is running icinga version 2.11.1

The new host showed up in icingaweb2 but it is always in PENDING state. All of its services are in PENDING state as well.

I’m not really sure what it means. I don’t see any errors on startup.

As I troubleshooted, I learned that 2.11 has more strict configuration syntax validation that results in slightly different configuration but I’m not sure what the problem really is.

Here’s the agent’s zone.conf:

/*
 * Ansible managed
 */

object Endpoint "icinga.a.mars.mydomain.net" {
        host = "icinga.a.mars.mydomain.net"
        port = "5665"
}
 
object Zone "master" {
        endpoints = [ "icinga.a.mars.mydomain.net" ]
}

object Endpoint "wq-4.a.mars.mydomain.net" {
}

object Zone "wq-4.a.mars.mydomain.net" {
        endpoints = [ "wq-4.a.mars.mydomain.net" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

On master in /etc/icinga2/zones.d/wq-4.a.mars.mydomain.net/hosts.conf

object Zone "wq-4.a.mars.mydomain.net" {
    endpoints = [ "wq-4.a.mars.mydomain.net" ]
    parent = "master"
}

object Endpoint "wq-4.a.mars.mydomain.net" {
    host = "wq-4.a.mars.mydomain.net"
}

object Host "wq-4.a.mars.mydomain.net" {
  import "sf-mars-host"
  address = "wq-4.a.mars.mydomain.net"
}

and /etc/icinga2/zones.conf:

/*
 * Generated by Icinga 2 node setup commands
 * on 2018-02-12 18:54:06 +0000
 */

object Zone "global-templates" {
        global = true
}

object Endpoint "icinga.a.mars.mydomain.net" {
}

object Zone "master" {
        endpoints = [ "icinga.a.mars.mydomain.net" ]
}

object Host "icinga.a.mars.mydomain.net" {
        address = "icinga.a.mars.mydomain.net"
        check_command = "hostalive"
        vars.slack_notifications = "enabled"
}

Not sure if anything else is relevant so I’ll keep it at that for now.

I’m not sure what is wrong exactly here.

I would appreciate any help!

2

Hello,

I’m new to icinga and I’m working with an already deployed installation
that is running icinga version 2.8.1 (master and agents).

We added a new host and it is running icinga version 2.11.1

Oooh :frowning: Not a good idea.

The new host showed up in icingaweb2 but it is always in PENDING state. All
of its services are in PENDING state as well.

See Distributed Monitoring - Icinga 2
#versions-and-upgrade

"The Icinga project aims to allow the following compatibility:

"master (2.11) >= satellite (2.10) >= agent (2.9)

"Older agent versions may work, but there’s no guarantee. Always keep in mind
that older versions are out of support and can contain bugs.

“In terms of an upgrade, ensure that the master is upgraded first, then
involved satellites, and last the Icinga agents. If you are on v2.10
currently, first upgrade the master instance(s) to 2.11, and then proceed with
the satellites.”

Regards,

Antony.

3

Well, as I troubleshooted this problem I ended up recreating a local test replica of production installation: master and one agent, both running icinga 2.11.1.

However, the same problem happens in this test setup. The one agent host is always in PENDING state. All of its services are in PENDING state as well.

So, I think we can probably rule out version mismatch as the cause of this problem.

4

What do the icinga log files tell you on the master and the agent?

Do they perhaps complain about connectivity problems, or untrusted
certificates?

Antony.

5

Here are the log files. They look good to me. At least I can’t see any explicit error, warning or suspicious messages.

(master) iliv@icinga-sf:~$ sudo cat /var/log/icinga2/icinga2.log

[2020-05-21 11:48:48 +0000] information/ApiListener: 'api' started.
[2020-05-21 11:48:48 +0000] information/ApiListener: Copying 1 zone configuration files for zone 'icinga-sf-client.lxd' to '/var/lib/icinga2/api/zones/icinga-sf-client.lxd'.
[2020-05-21 11:48:48 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-sf-client.lxd//_etc/hosts.conf
[2020-05-21 11:48:48 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-05-21 11:48:48 +0000] information/DbConnection: 'ido-pgsql' started.
[2020-05-21 11:48:48 +0000] information/NotificationComponent: 'notification' started.
[2020-05-21 11:48:48 +0000] information/CheckerComponent: 'checker' started.
[2020-05-21 11:48:48 +0000] information/ConfigItem: Activated all objects.
[2020-05-21 11:48:48 +0000] information/IdoPgsqlConnection: 'ido-pgsql' resumed.
[2020-05-21 11:48:48 +0000] information/DbConnection: Resuming IDO connection: ido-pgsql
[2020-05-21 11:48:48 +0000] information/IdoPgsqlConnection: PGSQL IDO instance id: 1 (schema version: '1.14.3')
[2020-05-21 11:48:49 +0000] information/IdoPgsqlConnection: Finished reconnecting to 'ido-pgsql' database 'icinga' in 0.617281 second(s).
[2020-05-21 11:48:57 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2020-05-21 11:48:57 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-05-21 11:48:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 0, rate: 0.2/s (12/min 12/5min 12/15min);
[2020-05-21 11:49:08 +0000] information/ApiListener: New client connection for identity 'icinga-sf-client.lxd' from [10.0.250.182]:56026
[2020-05-21 11:49:08 +0000] information/ApiListener: Sending config updates for endpoint 'icinga-sf-client.lxd' in zone 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished sending config file updates for endpoint 'icinga-sf-client.lxd' in zone 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Syncing runtime objects to endpoint 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-sf-client.lxd' in zone 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Sending replay log for endpoint 'icinga-sf-client.lxd' in zone 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished sending replay log for endpoint 'icinga-sf-client.lxd' in zone 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished syncing endpoint 'icinga-sf-client.lxd' in zone 'icinga-sf-client.lxd'.
[2020-05-21 11:49:08 +0000] information/JsonRpcConnection: Received certificate request for CN 'icinga-sf-client.lxd' signed by our CA.
[2020-05-21 11:49:08 +0000] information/JsonRpcConnection: The certificate for CN 'icinga-sf-client.lxd' is valid and uptodate. Skipping automated renewal.
[2020-05-21 11:52:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1, rate: 1.6/s (96/min 421/5min 421/15min);
[2020-05-21 11:53:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 5, rate: 1.6/s (96/min 489/5min 489/15min);
[2020-05-21 11:53:48 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-05-21 11:54:07 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-05-21 11:54:07 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 1/15min);
[2020-05-21 11:54:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 0, rate: 1.6/s (96/min 505/5min 540/15min);

(agent) iliv@icinga-sf-client:~$ sudo cat /var/log/icinga2/icinga2.log

[2020-05-21 11:49:07 +0000] information/ApiListener: 'api' started.
[2020-05-21 11:49:07 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-05-21 11:49:07 +0000] information/ApiListener: Reconnecting to endpoint 'icinga-sf.lxd' via host 'icinga-sf.lxd' and port '5665'
[2020-05-21 11:49:07 +0000] information/CheckerComponent: 'checker' started.
[2020-05-21 11:49:07 +0000] information/ConfigItem: Activated all objects.
[2020-05-21 11:49:08 +0000] information/ApiListener: New client connection for identity 'icinga-sf.lxd' to [10.0.250.123]:5665
[2020-05-21 11:49:08 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icinga-sf.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Sending config updates for endpoint 'icinga-sf.lxd' in zone 'master'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished sending config file updates for endpoint 'icinga-sf.lxd' in zone 'master'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Syncing runtime objects to endpoint 'icinga-sf.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-sf.lxd'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-sf.lxd' in zone 'master'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Sending replay log for endpoint 'icinga-sf.lxd' in zone 'master'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished sending replay log for endpoint 'icinga-sf.lxd' in zone 'master'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished syncing endpoint 'icinga-sf.lxd' in zone 'master'.
[2020-05-21 11:49:08 +0000] information/ApiListener: Finished reconnecting to endpoint 'icinga-sf.lxd' via host 'icinga-sf.lxd' and port '5665'
[2020-05-21 11:49:16 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-05-21 11:49:16 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
6

Zones and endpoint definition will not work in this way. They have to be defined in zones.conf only.

7

Does this rule apply to icinga > 2.11 or 2.8 as well? Because, it works with icinga 2.8.

8

Hi,

With the stricter configuration validation in 2.11.x we’ve also removed the zone in zone inclusion.
2.12. will readd this feature though because there are quite a lot installations which rely on that feature.

You may either change your configuration or install 2.12.0 RC1.

All the best,
Eric

9

Thank you for pointing that out. I just wanted to say that my problem was indeed icinga2 version 2.8 and 2.11 being incompatible in terms of this specific type of configuration.

1 Like