Icinga2 - config master weird behavior

Icinga 2
1

Hello,
I see these errors from config master

[2026-09-29 00:53:41 -0700] information/ApiListener: Ignoring JSON-RPC connection from [::ffff:10.74.48.32]:47438. We're already connected to Endpoint 'master2'  (last message sent: 2026-09-29 00:51:27, last message received: 2026-09-29 00:51:51).[2026-09-29 00:53:51 -0700] information/ApiListener: Ignoring JSON-RPC connection from [::ffff:10.74.48.32]:55914. We're already connected to Endpoint 'master2'' (last message sent: 2026-09-29 00:51:27, last message received: 2026-09-29 00:51:51).

And these errors from the secondary master

[2026-09-29 00:53:41 -0700] critical/ApiListener: Error while syncing endpoint 'master1': Error: Cannot send message to already disconnected API client 'master1'!
[2026-09-29 00:53:51 -0700] critical/ApiListener: Error while syncing endpoint 'master1': Error: Cannot send message to already disconnected API client 'master1'!

So it seems secondary sees the config master as disconnected API client and the config master seems secondary as already connected API endpoint. When this problem occurs, I had to restart the icinga2 service on the config master and then everything works as normal. This has happened a couple of times in the last 20 days.

Any idea what could be the issue here? There is no firewall between the two master and they are on the same nework subnet.

Thanks

2

I am a bit confused about your setup.

I can see three different endpoints here: master2, master1 and master. Can you explain your setup a bit?

3

Oh that was a typo. I’ve corrected it. Sorry for the confusion.

I’ve Icinga2 2.15.1-1 running with Icingaweb2 2.12.5 on Ubuntu 24.04 LTS with 12 zones.
We are not using Director and icinga2 agent. 11K hosts and 56K services configured on the cluster.

4

I encountered another cluster communication problem , this time with a satellite

Config master reported multiple times until I restarted ‘icinga2’ service on the satellite.

[2026-09-30 14:41:42 -0700] warning/JsonRpcConnection: Error while sending JSON-RPC message for identity ‘satellite’ Error: Connection reset by peer [system:104 at /usr/include/boost/asio/detail/reactive_socket_send_op.hpp:137 in function ‘do_complete’]

Stacktrace:
0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
1# 0x0000652939A237DB in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
2# icinga::NetString::WriteStringToStream(boost::intrusive_ptr<icinga::Sharedicinga::AsioTlsStream > const&, icinga::String const&, boost::asio::basic_yield_contextboost::asio::executor) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
3# icinga::JsonRpcConnection::WriteOutgoingMessages(boost::asio::basic_yield_contextboost::asio::executor) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
4# 0x0000652939D74961 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
5# make_fcontext in /lib/x86_64-linux-gnu/libboost_context.so.1.83.0

The satellite log showed

[2026-09-30 15:01:12 -0700] information/JsonRpcConnection: Requesting new certificate for this Icinga instance from endpoint ‘master1’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Sending config updates for endpoint ‘master1’ in zone ‘master’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Finished sending config file updates for endpoint ‘master1’ in zone ‘master’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Syncing runtime objects to endpoint ‘master1’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Finished syncing runtime objects to endpoint ‘master1’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Finished sending runtime config updates for endpoint ‘master1’ in zone ‘master’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Sending replay log for endpoint ‘sjc-master1’ in zone ‘master’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Replayed 472 messages.
[2026-09-30 15:01:12 -0700] information/ApiListener: Finished sending replay log for endpoint 'master1 in zone ‘master’.
[2026-09-30 15:01:12 -0700] information/ApiListener: Finished syncing endpoint ‘master1’ in zone ‘master’.
[2026-09-30 15:01:12 -0700] information/ApiListener: New client connection for identity ‘master1’ from [::ffff:10.74.48.31]:50318
[2026-09-30 15:01:12 -0700] information/ApiListener: Ignoring JSON-RPC connection from [::ffff:10.74.48.31]:50318. We’re already connected to Endpoint ‘master1’ (last message sent: 2026-09-30 15:01:12, last message received: 2026-09-30 15:00:52).

Is this some sort of an issue with the cluster communcation over JSON-RPC on this version?

5

Wild guess, as I’m not 100% sure if the following would produce these messages.

Do you have the host attribute set in

  • the masters zones.conf files for the satellites endpoint object
  • AND
  • the satellites zones config for the masters endpoint objects
  • AND
  • each masters endpoint object in the masters zones.conf

I’m reffering to this: Endpoint connection direction

Meaning both sides try to connect to each other simultaneously.

I, personally, would setup a “bottom-up” connection direction, meaning the child node connects to the parent.

Master < Satellite (< Agent)