Local device monitoring using satellites

Nick31 · January 30, 2023, 11:14am

Hi everyone,

I’m cloning an existing monitoring system but i have some trouble with satellites.

I created a test satellite before migrating the monitoring system, I’ve followed the documentation to connect the satellite with the master, it seems working but i can only execute commands into the server that contains the satellite and not check other devices connected to it.
It seems that the master is executing the check instead of the satellite.
For example i can check disk state of the satellite but i can’t ping other devices in the network from the satellite (it show me they aren’t reachable but they are).

If it can be useful for the troubleshooting i have icinga director installed and properly working

My network is structured in this way:

Network A - Master

Network B - Satellite and other devices i want to check

I followed the doc in section “distributed monitoring”:
https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#distributed-monitoring-configuration-modes

section “agent check with command endpoint”

Could you please help me?

Thanks a lot

Niccolò

Pooh · January 30, 2023, 12:57pm

It sounds to me like you might want to reconsider using top-down command
endpoint and maybe use config sync instead.

In command endpoint mode, the master is responsible for scheduling all service
checks, and telling each endpoint to perform the checks (and then the master
gets the results back again). If the master cannot communicate to the machine
which needs to be checked (network breakage somewhere, for example) then the
service check will not be performed.

In config sync mode, the master synchronises the configuration needed by each
node to that node, and then every machine performs its own scheduling, carries
out the service checks, and returns the results to the master. If there is a
network breakage somewhere, the service checks continue being performed, and
the results are sent back to the master once connectivity is restored.

Unless the majority of the machines you are monitoring are things like
switches, routers, access points, UPSs - basically things which Icinga2 cannot
be installed on - then I recommend using config sync.

If you can’t install Icinga2 on the things that need to be monitored (because
they’re not running a real O/S) then obviously you need to use command
endpoint, and perform SNMP and similar checks from the nearest machine which
does have Icinga2 installed on it.

If you are still puzzled about why your setup is not behaving as expected, I
suggest that you post the configuration of the Host, Endpoint and Zone for each
of the master, satellite and agent, as configured on each of the master,
satellite and agent.

Also, the output of:

icinga2 object list -name "exampleagent"

might give you useful information about where the service checks for that
agent are configured to be performed. Substitute the Host name of the agent
into the above command.

It should produce quite a bit of output, telling you about the Host, the Zone
and the Endpoint of that agent, plus details of each service check which has
been configured to run on it.

Finally, you mentioned “Network A” and “Network B”. Does Network A have
routing in to Network B such that all machines which need to be monitored in
Network B are accessible from Network A?

Antony.

Nick31 · January 30, 2023, 1:47pm

Thank you for the quick reply

You understood correctly, I frequently have to check devices where i can’t install Icinga2 and I will perform also SNMP checks.

I did not understand that with this configuration the master performs checkes, I’m following your advices and using the
I tried the config sync.

To do it I followed the doc but it still doesn’t seem to be correct.

Here are the Master and Satellite configurations:

zones.conf, i wrote it in both master and satellite:

then i created a folder named master that contains:

hosts.conf

services.conf

On the satellite instead:

I created a folder named satellite-bd that contains:

hosts.conf:

and services.conf that is empty at the moment

I hope i made myself clear
Niccolò

Pooh · January 30, 2023, 2:29pm

Firstly, you have called your master machine “monitor-master”, therefore that
is the name you should use for the folder on the master under
/etc/icinga2/zones.d

Secondly you have to configure the satellite to know about the master:
https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/
#agentsatellite-setup

Using the node wizard on the satellite may well be the easiest way to do this.

The satellite only needs to be configured in /etc/icinga2/zones.conf - you
should not need to create any subdirectories or new files on that machine.

The summary is:

The master needs to know about itself and the satellite - this usually goes
into /etc/icinga2/zones.conf
The satellite needs to know about itself and the master - this also goes
into /etc/icinga2/zones.conf
The agent needs to know about itself and the satellite - again in
/etc/icinga2/zones.conf

You then create a directory on the master with the same name as the Zone for
the satellite - in your case /etc/icinga2/zones.d/satellite-bd

Inside this directory you define the Host for the satellite (the filename
doesn’t matter), and you also create a file here defining the Zone and the
Endpoint of the agent.

Finally, you create a directory on the master with the same name as the Zone
of the agent - you don’t seem to have chosen this yet, so I shall assume it
would be /etc/icinga2/zones.d/niccolo-bd

Inside this directory you define the Host for the agent (the filename doesn’t
matter).

From then on everything is configured on the master only - you do not need to
define any service checks or similar on the satellite or the agent - the master
sends this information to the appropriate machines, once the above directory
structure is correct.

helped me to understand this rather better than just working from the standard
documentation.

Antony.

Pooh · January 30, 2023, 2:58pm

Note that everything I have just described about the directory structure for
master / satellite / agent is only true for the machines on which Icinga2 can
be installed.

Anything which does not run Icinga2 and only needs to be monitored by
something that is running Icinga2 only needs to be configured in the
directory on the master which has the Zone name of the machine which will do
the monitoring.

So, in my case, for example, I have a satellite named GatewayDE which performs
some checks on managed switches in its local network (not visible to the
master, because they are inside a NAT network link and the master is out in a
data centre).

In that case the definitions of the Host, Zone and Endpoint for every switch
are in a single file in the directory /etc/icinga2/zones.d/GatewayDE on the
master.

Hope that helps,

Antony.

Nick31 · January 30, 2023, 3:49pm

Thank you a lot

So now in zones.d i have two folders: monitor-master and satellite-bd,.
I have no agents, only satellite and i want to ping an host called niccolo-bd that is unreachable from the master ( just to try if configurations are working )

typically i will use this configuration to ping switches

Then

Inside zones.conf i set:

object Endpoint “monitor-master” {

}

object Endpoint “satellite-bd.blueday.it”{
host = “195.39.202.134”
log_duration = 0
}

object Zone ZoneName {

    endpoints = [ "monitor-master" ]

}
object Zone “satellite-bd”{

    parent = "master"
    endpoints = [ "satellite-bd.blueday.it" ]

}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

Inside monitor-master

a file containing

object Host “satellite-bd.blueday.it” {

    check_command = "hostalive"
    address = "195.39.202.134"
    vars.agent_endpoint = "satellite-bd.blueday.it"

}

Inside satellite-bd

a file containing

object Host “niccolo-bd.blueday.it” {

    check_command = "hostalive"
    address = "192.168.0.69"
    vars.agent_endpoint = "satellite-bd.blueday.it"

}

Have i understood correctly?

Niccolò

Pooh · January 30, 2023, 4:26pm

Inside zones.conf i set:

I assume you mean zones.conf on the master for now… you haven’t mentioned
zones.conf on the satellite, and that needs to exist as well

object Endpoint “monitor-master” {
}

Since that is empty, the satellite will not connect to it - that’s fine.

object Endpoint “satellite-bd.blueday.it”{
host = “195.39.202.134”
log_duration = 0
}

Specifying the host there means the master will try to connect to the
satellite. So far, so good.

object Zone ZoneName {

That “ZoneName” should be somethng like “monitor-master”, otherwise it looks
very generic and you are likely to get confused

    endpoints = [ "monitor-master" ]
}

object Zone “satellite-bd”{
parent = “master”

That should be the Zone name of the master, so if you use my suggestion of
calling that zone “monitor-master”, this should be: parent = “monitor-master”

    endpoints = [ "satellite-bd.blueday.it" ]
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

Inside monitor-master a file containing

object Host “satellite-bd.blueday.it” {
check_command = “hostalive”
address = “195.39.202.134”
vars.agent_endpoint = “satellite-bd.blueday.it”
}

I assume you want the master to check it can ping the staellite, not that the
satellite can ping its own IP address, so remove the agent_endpoint there.

Inside satellite-bd a file containing

object Host “niccolo-bd.blueday.it” {
check_command = “hostalive”
address = “192.168.0.69”
vars.agent_endpoint = “satellite-bd.blueday.it”
}

Remove the agent_endpoint again - this should be automatic, since you have put
this file under satellite-bd, therefore that’s where the configuration will go,
and be used.

Once everything is correct, you should be able to do a restart or a reload of
Icings2 on the master, and after a few seconds, find the following file has
appeared on the satellite:

/var/lib/icinga2/api/zones/satellite-bd/_etc/satellite-bd.conf

(or whatever you named the very last file listed above).

If that is not true, check to see whether either the master or the satellite
contains a file:

/var/lib/icinga2/api/zones-stage-startup-last-failed.log

If that exists, it would tell you more about what the problem is.

Antony.

Nick31 · January 31, 2023, 8:22am

Perfect, now it seems working properly, I found

/var/lib/icinga2/api/zones/satellite-bd/_etc/hosts.conf

and I can now reach the internal device into the network B
but there is still a problem, now the service of satellite is pending, i dont know why, I have not specified any service at the moment.

immagine

the device with 192.168.0.69 is in the network B, the same network of the satellite ( 192.168.0.49 )

in both montior-master and satellite-bd folders I have services.conf empty, so i should not have specified any service yet?

Sorry for the late reply and thank you a lot

Niccolò

Nick31 · January 31, 2023, 12:23pm

During the initial wizard I did not set the API maybe could that be the problem?

Niccolò