Satellite & Master connection reset by peer when synchronizing configuration files

We have a pair of master nodes, and several working zones each with a pair of satellite nodes. We are now in the process of setting up a new zone and have followed the same instructions, and the satellite gets an error whenever the master attempts to synchronize the zone configuration files to the satellite.

The main difference is that the OS images for the existing zones are on-prem, whereas the images for the new Satellites are AWS-based, but it’s triggering some kind of exception/error handling -

[2021-02-16 19:34:08 +0000] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'icinga-master01.xxxx': Error: Connection reset by peer

    (0) icinga2: icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executo
r> >, long) (+0x58b) [0x9f736b]
    (1) icinga2: icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long)
 (+0x95) [0x9e46b5]
    (2) icinga2: icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) (+0x10c) [0xa3703c]
    (3) /usr/lib64/icinga2/sbin/icinga2() [0xa3762a]
    (4) /usr/lib64/icinga2/sbin/icinga2() [0xa37ab8]
    (5) libboost_context.so.1.69.0: make_fcontext (+0x2f) [0x7fd34188218f]

The satellite node does not receive any files - there are no staging zone files or anything else created. The above error is only triggered when a configuration change is published, resulting in transferring that updated configs to the satellites. If not configuration change is published, there are no errors in the log files - so after this error it’ll reconnect to the masters intermittently, until another change is published when the error recurs. So it seems to be during the transfer of the configuration files this is triggered-

The satellite will connect to the master, download it’s new certificate (once signed at the master) and aside from disconnecting and reconnecting more frequently than the other zones (which we attribute to having no configuration, although it could be another symptom), we can’t seem to find any material setup difference in zone configuration, etc.

[2021-02-16 19:33:48 +0000] information/ApiListener: Reconnecting to endpoint ‘icinga-master01.xxxx’ via host ‘10.0.1.101’ and port ‘5665’
[2021-02-16 19:33:48 +0000] information/ApiListener: New client connection for identity ‘icinga-master01.xxxx’ to [10.0.1.101]:5665
[2021-02-16 19:33:48 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint ‘icinga-master01.xxxx’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Finished reconnecting to endpoint ‘icinga-master01.xxxx’ via host ‘10.0.1.101’ and port ‘5665’
[2021-02-16 19:33:48 +0000] information/ApiListener: Sending config updates for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Finished sending config file updates for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Syncing runtime objects to endpoint ‘icinga-master01.xxxx’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Finished syncing runtime objects to endpoint ‘icinga-master01.xxxx’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Finished sending runtime config updates for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Sending replay log for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Finished sending replay log for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:33:48 +0000] information/ApiListener: Finished syncing endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:08 +0000] information/JsonRpcConnection: No messages for identity ‘icinga-master02.xxxx’ have been received in the last 60 seconds.
[2021-02-16 19:34:08 +0000] warning/JsonRpcConnection: API client disconnected for identity ‘icinga-master02.xxxx’
[2021-02-16 19:34:08 +0000] warning/ApiListener: Removing API client for endpoint ‘icinga-master02.xxxx’. 0 API clients left.
[2021-02-16 19:34:08 +0000] warning/JsonRpcConnection: API client disconnected for identity ‘icinga-master01.xxxx’
[2021-02-16 19:34:08 +0000] warning/ApiListener: Removing API client for endpoint ‘icinga-master01.xxxx’. 0 API clients left.
[2021-02-16 19:34:11 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2021-02-16 19:34:11 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2021-02-16 19:34:18 +0000] information/ApiListener: Reconnecting to endpoint ‘icinga-master02.xxxx’ via host ‘10.0.2.102’ and port ‘5665’
[2021-02-16 19:34:18 +0000] information/ApiListener: Reconnecting to endpoint ‘icinga-master01.xxxx’ via host ‘10.0.1.101’ and port ‘5665’
[2021-02-16 19:34:18 +0000] information/ApiListener: New client connection for identity ‘icinga-master01.xxxx’ to [10.0.1.101]:5665
[2021-02-16 19:34:18 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint ‘icinga-master01.xxxx’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished reconnecting to endpoint ‘icinga-master01.xxxx’ via host ‘10.0.1.101’ and port ‘5665’
[2021-02-16 19:34:18 +0000] information/ApiListener: Sending config updates for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished sending config file updates for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Syncing runtime objects to endpoint ‘icinga-master01.xxxx’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished syncing runtime objects to endpoint ‘icinga-master01.xxxx’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished sending runtime config updates for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Sending replay log for endpoint ‘icinga-master01.xxxx’ in zone ‘I Master I’.
[2021-02-16 19:34:18 +0000] information/ApiListener: New client connection for identity ‘icinga-master02.xxxx’ to [10.0.2.102]:5665
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished sending replay log for endpoint ‘icinga-master01.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished syncing endpoint ‘icinga-master01.xxxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint ‘icinga-master02.xxxx’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished reconnecting to endpoint ‘icinga-master02.xxxx’ via host ‘10.0.2.102’ and port ‘5665’
[2021-02-16 19:34:18 +0000] information/ApiListener: Sending config updates for endpoint ‘icinga-master02.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished sending config file updates for endpoint ‘icinga-master02.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Syncing runtime objects to endpoint ‘icinga-master02.xxxx’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished syncing runtime objects to endpoint ‘icinga-master02.xxxx’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished sending runtime config updates for endpoint ‘icinga-master02.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Sending replay log for endpoint ‘icinga-master02.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished sending replay log for endpoint ‘icinga-master02.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:18 +0000] information/ApiListener: Finished syncing endpoint ‘icinga-master02.xxxx’ in zone ‘Master’.
[2021-02-16 19:34:38 +0000] warning/JsonRpcConnection: API client disconnected for identity ‘icinga-master02.xxxx’

  • Version used (icinga2 --version)
  • Operating System and version

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.3)

System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-1160.15.2.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: runner-hh8q3bz2-project-322-concurrent-0
OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

  • Enabled features (icinga2 feature list)

icinga2 feature list
Disabled features: compatlog elasticsearch gelf graphite icingadb influxdb livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker command debuglog mainlog

  • Config validation (icinga2 daemon -C)

icinga2 daemon -C
[2021-02-16 22:09:07 +0000] information/cli: Icinga application loader (version: 2.12.3)
[2021-02-16 22:09:07 +0000] information/cli: Loading configuration file(s).
[2021-02-16 22:09:07 +0000] information/ConfigItem: Committing config item(s).
[2021-02-16 22:09:07 +0000] information/ApiListener: My API identity: ops-mon01.e.xxxx
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 2 FileLoggers.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 4 Zones.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 4 Endpoints.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-02-16 22:09:07 +0000] information/ConfigItem: Instantiated 235 CheckCommands.
[2021-02-16 22:09:07 +0000] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2021-02-16 22:09:07 +0000] information/cli: Finished validating the configuration file(s).

  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes

object Endpoint “icinga-master01.xxxx” {
host = “10.0.1.101”
port = “5665”
}
object Endpoint “icinga-master02.xxxx” {
host = “10.0.2.102”
port = “5665”
}
object Zone “Master” {
endpoints = [ “icinga-master01.xxxx”, “icinga-master02.xxxx”, ]
}
object Endpoint “ops-mon01.e.xxxx” {
}
object Endpoint “ops-mon02.e.xxxx” {
}
object Zone “Zone-E” {
endpoints = [ “ops-mon01.e.xxxx”, “ops-mon02.e.xxxx” ]
parent = “Master”
}
object Zone “global-templates” {
global = true
}
object Zone “director-global” {
global = true
}