Issues with Distributed Monitoring

Hello All,

Trying to get a distributed setup going. Currently, it is 1 Master, 1 Satellite, 1 Agent as this is just a PoC I’m building out.

Master = dev-icinga
Satellite = dev-satellite
Agent = dev-mobrien

Right now, when looking in icingaweb2 ui, I can see all 3 hosts showing up in the hosts section, checks are working on the master and the satellite. The zones are setup so that dev-satellites is a child of master.

The way I’m trying to get this setup is that the master will control the checks happening on the satellites themselves, and then the satellites will control the checks on the agents in the envs they are deployed in.

The issue I’m running into right now, all the checks are showing up for the agent, but when they attempt to run they return "Unknown: Remote Icinga instance ‘dev-mobrien’ is not connected to ‘dev-satellite’.

I enabled debuglog on both the satellite and the agent hosts, and I can see that it seems to be having issues with the certificate. I sign the CA for the agent on the satellite, but then within a minute a new request is already showing up for the agent. This is what I’m seeing in the debug log on the agent.

[2024-07-09 13:27:09 -0400] information/ApiListener: New client connection for identity 'dev-satellite' from [::ffff:10.0.11.53]:54350
[2024-07-09 13:27:09 -0400] notice/ApiListener: New JSON-RPC client
[2024-07-09 13:27:09 -0400] debug/EndpointDbObject: update is_connected=1 for endpoint 'dev-satellite'
[2024-07-09 13:27:09 -0400] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'dev-satellite'.
[2024-07-09 13:27:09 -0400] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'dev-satellite'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Sending config updates for endpoint 'dev-satellite' in zone 'dev-satellites'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Finished sending config file updates for endpoint 'dev-satellite' in zone 'dev-satellites'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Syncing runtime objects to endpoint 'dev-satellite'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Finished syncing runtime objects to endpoint 'dev-satellite'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Finished sending runtime config updates for endpoint 'dev-satellite' in zone 'dev-satellites'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Sending replay log for endpoint 'dev-satellite' in zone 'dev-satellites'.
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720541930
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720543576
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720543617
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720545892
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/current
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replayed 0 messages.
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720541930
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720543576
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720543617
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/1720545892
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replaying log: /var/lib/icinga2/api/log/current
[2024-07-09 13:27:09 -0400] notice/ApiListener: Replayed 0 messages.
[2024-07-09 13:27:09 -0400] information/ApiListener: Finished sending replay log for endpoint 'dev-satellite' in zone 'dev-satellites'.
[2024-07-09 13:27:09 -0400] information/ApiListener: Finished syncing endpoint 'dev-satellite' in zone 'dev-satellites'.
[2024-07-09 13:27:09 -0400] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'dev-satellite': Error: End of file

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# 0x0000000000864F4A in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000ADE24F in /usr/lib64/icinga2/sbin/icinga2
 5# 0x0000000000AE544B in /usr/lib64/icinga2/sbin/icinga2

Certificates needs to be signed by the master only.

Ahh, ok. That fixed it, thank you!