Host up and service pending

Hi everyone

I have a little problem with satellites checks
I’m running a ping and it seems it is working properly, I triedo to disconnect the pinged machine and it works, the only problem is that the host is showed as UP but the service is in pending state, that’s strange, the ping is working properly

My network is the following:

  • Network A → Master

-Network B → Satellite and other devices i want to ping ( also switches and routers so i can’t install agents in every device, niccolo.blueday.it in this case )

Master can not reach devices into network B

The objective is having satellites that performs checkes and give results to the master

Here are my configurations:

zones.conf master:

object Endpoint "monitor-master" {
}

object Endpoint "satellite-bd.blueday.it"{
        host = "195.39.202.134"
        log_duration = 0
}

object Zone "master" {
        endpoints = [ "monitor-master" ]
}

object Zone "satellite-bd"{
        parent = "master"
        endpoints = [ "satellite-bd.blueday.it" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

zones.conf satellite-bd:

object Endpoint "monitor-master" {
}

object Zone "master" {
        endpoints = [ "monitor-master" ]
}

object Endpoint "satellite-bd.blueday.it" {

}

object Zone "satellite-bd" {
        endpoints = [ "satellite-bd.blueday.it" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

then i created in the master two directories: one named master and one named satellite-bd

inside master i created a file named hosts.conf that contains:

object Host "satellite-bd.blueday.it" {

        check_command = "hostalive"
        address = "195.39.202.134"

}

Just to check that the satellite is reachable

inside satellite-bd i created a file named satellite-bd.conf that contains:

object Host "niccolo-bd.blueday.it" {
  import "generic-host"
  address = "192.168.0.69"
}

Here i where I perform my checkes on the devices.

I see the host is up but the service is stucked in pending

If someone could help me i would be very grateful

Best regards, Niccolò

Hi :slight_smile:

Which of the two hosts?

The satellite pinged by the master?
Or the client pinged by the satellite?

Please also show your service definition.
Is the satellite correctly connected to the master via the icinga2 node wizard?
Does it receive config from the master (received config is stored in /var/lib/icinga2/api/zones).

The log files icinga2.log and debug.log [needs to be activated] are also a good place to look for messages concerning a problem.

The satellite is correctly pinged by the master

This is the problem, the client is pinged correctly ( if i shut down the client it correctly signal me that the host is down ) but the service stays in pending state.

I found this error in satellite’s icinga2.log:

Blockquote
[2023-02-01 08:21:51 +0000] critical/config: Error: Import references unknown template: ‘generic-host’
Location: in /var/lib/icinga2/api/zones-stage//satellite-bd/_etc/satellite-bd.conf: 2:3-2:23
/var/lib/icinga2/api/zones-stage//satellite-bd/_etc/satellite-bd.conf(1): object Host “niccolo-bd.blueday.it” {
/var/lib/icinga2/api/zones-stage//satellite-bd/_etc/satellite-bd.conf(2): import “generic-host”
^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite-bd/_etc/satellite-bd.conf(3): address = “192.168.0.69”
/var/lib/icinga2/api/zones-stage//satellite-bd/_etc/satellite-bd.conf(4): }
[2023-02-01 08:21:51 +0000] critical/config: 1 error
[2023-02-01 08:21:51 +0000] critical/cli: Config validation failed. Re-run with ‘icinga2 daemon -C’ after fixing the config.

Should i specify the template also into the satellite?
I tried to add the template definition i have into the master in the satellites conf.d/templates.conf but it still doesn’t work

Put the template into a global zone.
But be careful with passwords!

Ok i resolved the issue on incinga2.log just remobing the import statement, it was just a trial so i can do it, in future i will put it in global zone
But the service keeps being pending

immagine

First, you should make sure the zones are connected. Best practice is to define cluster-zone checks e.g.:

apply Service "icinga_zone" {
   check_command = "cluster-zone"

   assign where get_object("Endpoint", host.name)
}

Pending checks are usually caused by not connected zones. To identify the reason you then need to check icinga2.log on the target machine as well as the parent node. Common errors are certificate issues or time synchronization.

Ok so i will put this check into the satellite zone,

I checked icinga2.log but it seems it’s all good

Blockquote
‘/var/lib/icinga2/api/zones-stage/director-global’. Current timestamp ‘2023-02-01 08:49:22 +0000’ (1675241362.436861) >= received timestamp ‘2023-02-01 08:49:22 +0000’ (1675241362.436861).
[2023-02-01 08:52:13 +0000] information/ApiListener: Received configuration for zone ‘director-global’ from endpoint ‘monitor-master’. Comparing the timestamp and checksums.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf’ for zone ‘director-global’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/director-global//director/commands.conf’ for zone ‘director-global’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf’ for zone ‘director-global’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf’ for zone ‘director-global’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/director-global//director/servicegroups.conf’ for zone ‘director-global’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf’ for zone ‘director-global’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones-stage/director-global’ (98125 Bytes).
[2023-02-01 08:52:13 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path ‘/var/lib/icinga2/api/zones-stage/satellite-bd’. Current timestamp ‘2023-02-01 08:49:22 +0000’ (1675241362.434354) >= received timestamp ‘2023-02-01 08:49:22 +0000’ (1675241362.434354).
[2023-02-01 08:52:13 +0000] information/ApiListener: Received configuration for zone ‘satellite-bd’ from endpoint ‘monitor-master’. Comparing the timestamp and checksums.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/satellite-bd//_etc/satellite-bd.conf’ for zone ‘satellite-bd’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Stage: Updating received configuration file ‘/var/lib/icinga2/api/zones-stage/satellite-bd//director/hosts.conf’ for zone ‘satellite-bd’.
[2023-02-01 08:52:13 +0000] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones-stage/satellite-bd’ (432 Bytes).
[2023-02-01 08:52:13 +0000] information/ApiListener: Received configuration updates (2) from endpoint ‘monitor-master’ are equal to production, skipping validation and reload.
[2023-02-01 08:52:18 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 08:52:18 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 08:52:18 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 3/s; Output: 3/s)
[2023-02-01 08:57:08 +0000] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2023-02-01 08:57:18 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 4/5min 4/15min);
[2023-02-01 08:57:18 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 08:57:18 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 3/s; Output: 3/s)
[2023-02-01 09:02:08 +0000] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2023-02-01 09:02:18 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 09:02:18 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 4/5min 8/15min);
[2023-02-01 09:02:28 +0000] information/IdoMysqlConnection: Pending queries: 5 (Input: 3/s; Output: 3/s)
[2023-02-01 09:07:08 +0000] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2023-02-01 09:07:18 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 4/5min 12/15min);
[2023-02-01 09:07:18 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 09:07:38 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 2/s; Output: 2/s)
[2023-02-01 09:12:08 +0000] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2023-02-01 09:12:18 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 4/5min 12/15min);
[2023-02-01 09:12:18 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 09:12:38 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 2/s; Output: 2/s)
[2023-02-01 09:17:08 +0000] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2023-02-01 09:17:28 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2023-02-01 09:17:28 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 4/5min 12/15min);
[2023-02-01 09:17:48 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 2/s; Output: 3/s)

It seems to me that is working correctly, but i noticed that maybe time syncronization is the problem,
there is 9:17:48 but it’s 10:17:48 in my time-zone

How can i sync time-zone?

Your satellite is receiving configuration, hence, it is connected, hence, you don’t have a time synchronization issue. Next guess, the service object(s) is/are missing at your satellite. You can check this using icinga2 object list.

I used that command and it listed all the commands that i have created into director.
I used grep to find all services, i cant find ping4 as service, it is only a CheckCommand

The file is very long, i have a lot of command defined in director so i can’t show you the entire output of the command

I tried to write:

Blockquote
apply Service “ping4” {
check_command = “cluster-zone”
assign where get_object(“Endpoint”, host.name)
}

into the master inside /etc/icinga2/zones.d/satellite-bd/satellite-bd.conf
but rerunning daemon it says:

Blockquote
[2023-02-01 11:08:37 +0100] critical/config: Error: An object with type ‘Service’ and name ‘monitor-master!ping4’ already exists (in /etc/icinga2/conf.d/services.conf: 26:1-26:21), new declaration: in /etc/icinga2/zones.d/satellite-bd/satellite-bd.conf: 6:1-6:21
Location: in /etc/icinga2/zones.d/satellite-bd/satellite-bd.conf: 6:1-6:21
/etc/icinga2/zones.d/satellite-bd/satellite-bd.conf(4): }
/etc/icinga2/zones.d/satellite-bd/satellite-bd.conf(5):
/etc/icinga2/zones.d/satellite-bd/satellite-bd.conf(6): apply Service “ping4”{
^^^^^^^^^^^^^^^^^^^^^

Blind guess, you have included /etc/icinga2/conf.d/ at your master but not at your satellite.

In general, it’s highly recommended to disable inclusion of /etc/icinga2/conf.d/ especially when using the director. Service objects should be arranged in global zone(s) instead.

You are perfectly right!

I will disable conf.d also in the master

How can i do it? Just creating a zone zones.d named as one of global zones and create a file service.conf?

The director uses director-global by default. But if you don’t like Windows service objects on Linux machines and vice versa, you could create e.g. windows-services or windows-global etc.

Ok so i have to copy all my services into my director global?

Thank you a lot!

No, you select director-global at Cluster Zone for your service definitions.

Is there a guid on how to do it?

Sorry but im new in in icinga and it’s my first satellite

Unfortunately, I don’t know good guidelines for the director. Maybe someone else can jump in.

now i commented recursive inclusion of /etc/icinga2/conf.d/ but it shows this error

Unable to detect your Icinga 2 Core version (DeployFormsBug7530.php:70)

There are several threads here about this error. Please check via search function.