Help with Distributed Monitoring

Hi,

I want to monitor some security cameras (about 8 of them) and other devices like vending machines that are all connected to the internet. The cameras I have are IoT cameras, they each have an IP and a small computer inside. For obvious reasons, i don´t have access to the computer inside.

The problem is, they are not on the same network and there won´t be a VPN connection either.

I’ve heard about Distributed Monitoring with the "master > satelite > client node " setup.
But as far as I understand, you have to install Icinga on a client which is going to be the security camera right? And of course I’m not allowed to do that.

Or can you put a thin client inside who “collects” the checks and sends them to the master?

I would love if someone can give me some tips on how to do this. Im quiet familliar with Icinga, I’ve had Icinga monitoring for about two years in our Headquarters. I use Icinga Director for the configs, but chaning configs with nano for example is not going to be a problem. I just never went outside my network.
Or is there a guide i somehow missed ?

Kind regards,
Gabriel

In your case I’d suggest distributed monitoring. You’ll have on central icinga instance as master and for every other location on icinga instances configured as satellite. Those satellites will then run checks against the cameras in their local network. Such a small satellite does not need much resources, hence, a raspi or similar would be enough.

Thank you for your reply.

It was uncertain for me if that guide is really the one for me. English is also not my first language.

That sounds good, but how does the master Icinga get the satellite checks ? For example it checks hostalive, snmp etc. for the 8 cameras, how do I get those checks to show up on my master Icinga Dashboard?
I also have the module Business Process where i can visually see if theres a problem or not.

Does the port 5665 have to be allowed in the firewall? and if yes inbound and outbound?

What has to be configured in Icinga Director for this to work? As far as I can see, in the documentation it doesn’t use icinga director.

I would love if you could take your time to help me in more detail with this problem of mine.

You configure master and satellites manual via conf files. You then add host objects which represent your cameras to the director (and he will distribute corresponding conf files from the master to the satellites in the background).

There will be no change means you can continue using it as up to now.

This port needs to open at the master OR the satellites depending on your configuration. Means both sides can initiate the connection so it’s up to you to decide which.

It’s important to understand that zone and endpoint objects for master and satellites have to be defined outside the director means directly and only in zones.conf. Once defined (or updated) you need to run Kickstart Wizard within the director (to import those changes).

Most of us are volunteers here and usually have a fulltime job. So you may no expect to much.

I would love if you could take your time to help me in more detail with this problem of mine.

If you need help, there are Icinga » Partners and Icinga itself provides paid support. I can recommend the Linuxfabrik GmbH – Ihr Ansprechpartner für Linux und Open Source.

You configure master and satellites manual via conf files. You then add host objects which represent your cameras to the director (and he will distribute corresponding conf files from the master to the satellites in the background).

So i just add the host as if its in my local network and Icinga will know its from another zone and just collects them from the satellite if the zones and endpoints are correctly configured?

Most of us are volunteers here and usually have a fulltime job. So you may no expect to much.

No problem at all, i’m not expecting full support, just a litte more detailed, but you explained it quiet well in your reply so that helped me a lot.

If you need help, there are Icinga » Partners and Icinga itself provides paid support. I can recommend the Linuxfabrik GmbH – Ihr Ansprechpartner für Linux und Open Source.

Thank you so much, if I can’t get it to work, i will surely try to contact them :wink:

I’m gonna give it a shot and I’ll keep you updated.

Kind regards

Yes, by selecting the propriate zone for each host icinga knows to which satellite each host belongs. These satellites are then responsible for scheduling checks and report results to its parent means to your master.

So, I’ve also found this quick guide which I somewhat followed.

I’ve run icinga2 node wizard on both master and satellite, created the respected Zones and Endpoints and ran the kickstard wizard.

But for some reason, the Hosts I added (the satellite itself and the gateway) are stuck on “pending” (Soft 1/3). Any reason why?

Here the configs:

Satellite:
grafik
Sadly I can’t copy past from my Hyper V for some reasons…

Master:

/*
 * Generated by Icinga 2 node setup commands
 * on 2022-05-24 13:59:00 +0200
 */

object Endpoint "Icinga" {
}

object Endpoint "bkw-end-satellite" {
}

object Zone "master" {
        endpoints = [ "Icinga" ]
}

object Zone "bkw-end-satellite" {
        endpoints = [ "bkw-end-satellite" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Config Validation:

root@Icinga:/etc/icinga2/conf.d# icinga2 daemon -C
[2022-05-24 14:30:47 +0200] information/cli: Icinga application loader (version: r2.13.3-1)
[2022-05-24 14:30:47 +0200] information/cli: Loading configuration file(s).
[2022-05-24 14:30:47 +0200] information/ConfigItem: Committing config item(s).
[2022-05-24 14:30:47 +0200] information/ApiListener: My API identity: Icinga
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 2 HostGroups.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 3 Hosts.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 13 Notifications.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 4 Zones.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 2 Endpoints.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 User.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 14 Services.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-05-24 14:30:47 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-05-24 14:30:47 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-05-24 14:30:47 +0200] information/cli: Finished validating the configuration file(s).

the satellite itself:

grafik

Did you do the certificate signing? Any hints in icinga2.log e.g. connection errors? Is api on master enabled?

the icinga2.log … something with 127.0.0.1 not reachable?

> [2022-05-24 15:28:22 +0200] information/JsonRpcConnection: Received certificate request for CN 'bkw-end-satellite' not signed by our CA: self signed certificate (code 18)
> [2022-05-24 15:28:22 +0200] information/JsonRpcConnection: Certificate request for CN 'bkw-end-satellite' is pending. Waiting for approval.
> [2022-05-24 15:28:22 +0200] warning/JsonRpcConnection: API client disconnected for identity 'bkw-end-satellite'
> [2022-05-24 15:28:25 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
> [2022-05-24 15:28:25 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
> [2022-05-24 15:28:30 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
> [2022-05-24 15:28:30 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
> [2022-05-24 15:28:32 +0200] information/ApiListener: New client connection for identity 'bkw-end-satellite' from [::ffff:192.168.0.93]:44178 (certificate validation failed: code 18: self signed certificate)
> [2022-05-24 15:28:32 +0200] information/JsonRpcConnection: Received certificate request for CN 'bkw-end-satellite' not signed by our CA: self signed certificate (code 18)
> [2022-05-24 15:28:32 +0200] information/JsonRpcConnection: Certificate request for CN 'bkw-end-satellite' is pending. Waiting for approval.
> [2022-05-24 15:28:32 +0200] warning/JsonRpcConnection: API client disconnected for identity 'bkw-end-satellite'
> [2022-05-24 15:28:35 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
> [2022-05-24 15:28:35 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
> [2022-05-24 15:28:40 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
> [2022-05-24 15:28:40 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
> [2022-05-24 15:28:42 +0200] information/ApiListener: New client connection for identity 'bkw-end-satellite' from [::ffff:192.168.0.93]:44180 (certificate validation failed:

There was nothing in the guide that said I should sign anything?

The icinga2 pki ticket --cn ... didn’t work for me (something with --salt was needed) but in the Wizard the line where it ask to insert that ticket was (optional) and i didn’t insert it… maybe that is the case?

Api is on master enabled.
root@Icinga:/etc/icinga2/conf.d# icinga2 feature list Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog Enabled features: api checker icingadb ido-mysql mainlog notification

You have enabled icingadb.

Signing is described here. Without entering a previous generated ticket you have choosen On-Demand CSR Signing. Means you now need to sign the request at your master: run icinga2 ca list, select the appropriate fingerprint and sign with icinga2 ca sign <fingerprint>.

Ah I see, I signed it now with the fingerprint and disabled icingadb.

Log looks like this now:

[2022-05-24 16:40:15 +0200] information/ApiListener: New client connection from [::1]:52748 (no client certificate)
[2022-05-24 16:40:15 +0200] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [::1]:52748), user: bkwapi00, agent: , status: OK).
[2022-05-24 16:40:15 +0200] information/HttpServerConnection: HTTP client disconnected (from [::1]:52748)
[2022-05-24 16:40:16 +0200] information/ApiListener: New client connection from [::1]:52752 (no client certificate)
[2022-05-24 16:40:16 +0200] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [::1]:52752), user: bkwapi00, agent: , status: OK).
[2022-05-24 16:40:16 +0200] information/HttpServerConnection: HTTP client disconnected (from [::1]:52752)
[2022-05-24 16:40:17 +0200] information/ApiListener: New client connection from [::1]:52754 (no client certificate)
[2022-05-24 16:40:17 +0200] information/HttpServerConnection: Request: POST /v1/actions/reschedule-check (from [::1]:52754), user: bkwapi00, agent: , status: OK).
[2022-05-24 16:40:17 +0200] information/HttpServerConnection: HTTP client disconnected (from [::1]:52754)

Alright, thank god i made a lot of snapshots, i went trough the whole installation process again with giving the pki ticket directly and was still on pending…

The only thing the log shouted at me though, was the following:

[2022-05-25 08:48:42 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-05-25 08:48:42 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-05-25 08:48:47 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-05-25 08:48:47 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-05-25 08:48:52 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'

so after I googled what this Redis thing is, I have found out that its is a depedency you have to install…
after apt-get install icingadb-redis and starting it with systemctl start icingadb-redis i have gotten it to work…

grafik

Thanks to @rsx @rivad for helping me for the time being.