I’m trying to find out why Icinga2 endpoints in my installation keep randomly beeing disconnected. I could find some older posts about issues like this, but no real clue how to handle this.
[2019-04-23 09:11:36 +0200] warning/TlsStream: TLS stream was disconnected.
[2019-04-23 09:11:36 +0200] warning/JsonRpcConnection: API client disconnected for identity ‘masterhost.example’
[2019-04-23 09:11:36 +0200] warning/ApiListener: Removing API client for endpoint ‘masterhost.example’. 0 API clients left.
Icinga version on the master and the endpoints is: r2.10.4-1
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Debian GNU/Linux 9
Docker Image: jordan/icinga2:latest
The endpoints log does not show any attempts to re-initiate the connection again, although it is executing multiple checks every minute. The “cluster” check (https://icinga.com/docs/icinga2/latest/doc/10-icinga-template-library/#cluster) shows the endpoints as disconnected and I see the services “Last check” timestamp getting old.
The log of the master does not seem to show any related error. When searching /var/log/icinga2.log, the last entry containing the name of the disconneced host is:
[2019-04-23 08:09:48 +0000] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones/endpointhost.example’ (0 Bytes). Received timestamp ‘2019-04-23 08:09:48 +0000’ (1556006988.769867), Current timestamp ‘2019-04-23 08:00:38 +0000’ (1556006438.549674).
The Icinga setup consists of about 30 endpoints and I see 1-2 disconnecting every week. I have to restart the Icinga service on the endpoint, then it reconnects.
Any help would be gratefully appreciated.