We have observed strange behavior on Icinga2 master. All checks randomly stop working. On production setups this happens quite often. To resolve we need to change random config and apply it or restart icinga2 service on master instance. We are able to reproduce it in development environment with bulk registration of new hosts (10 for example - simultaneously). What happen is, it doesn’t come to reconnect between master and satellite.
Satellite log:
[2020-09-02 07:37:58 +0000] warning/JsonRpcConnection: API client disconnected for identity ‘icinga-master1.localdomain’
[2020-09-02 07:37:58 +0000] warning/ApiListener: Removing API client for endpoint ‘icinga-master1.localdomain’. 0 API clients left.
Master log:
[2020-09-01 09:31:40 +0000] information/ApiListener: Started new listener on ‘[0.0.0.0]:5665’
[2020-09-01 09:31:40 +0000] information/GraphiteWriter: ‘graphite’ resumed.
[2020-09-01 09:31:40 +0000] information/ExternalCommandListener: ‘command’ started.
[2020-09-01 09:31:40 +0000] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
(0) Activating object ‘command’ of type ‘ExternalCommandListener’
[2020-09-01 09:31:40 +0000] information/DbConnection: ‘ido-mysql’ started.
[2020-09-01 09:31:40 +0000] information/NotificationComponent: ‘notification’ started.
[2020-09-01 09:31:40 +0000] information/CheckerComponent: ‘checker’ started.
[2020-09-01 09:31:40 +0000] information/ConfigItem: Activated all objects.
[2020-09-01 09:31:40 +0000] information/GraphiteWriter: Finished reconnecting to Graphite in 0.000306129 second(s).
[2020-09-01 09:31:40 +0000] information/IdoMysqlConnection: ‘ido-mysql’ resumed.
[2020-09-01 09:31:40 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
If everything works as it should we get log below:
Master:
[2020-09-01 09:47:20 +0000] information/ApiListener: Started new listener on ‘[0.0.0.0]:5665’
[2020-09-01 09:47:20 +0000] information/ApiListener: Reconnecting to endpoint ‘icinga2-satellite1’ via host ‘icinga-master1.localdomain’ and port ‘5665’
[2020-09-01 09:47:20 +0000] information/GraphiteWriter: ‘graphite’ resumed.
[2020-09-01 09:47:20 +0000] information/ExternalCommandListener: ‘command’ started.
[2020-09-01 09:47:20 +0000] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
(0) Activating object ‘command’ of type ‘ExternalCommandListener’
[2020-09-01 09:47:20 +0000] information/DbConnection: ‘ido-mysql’ started.
[2020-09-01 09:47:20 +0000] information/NotificationComponent: ‘notification’ started.
[2020-09-01 09:47:20 +0000] information/CheckerComponent: ‘checker’ started.
[2020-09-01 09:47:20 +0000] information/ConfigItem: Activated all objects.
[2020-09-01 09:47:20 +0000] information/IdoMysqlConnection: ‘ido-mysql’ resumed.
[2020-09-01 09:47:20 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
Satellite:
[2020-09-02 07:42:58 +0000] warning/JsonRpcConnection: API client disconnected for identity ‘icinga-master1.localdomain’
[2020-09-02 07:42:58 +0000] warning/ApiListener: Removing API client for endpoint ‘icinga-master1.localdomain’. 0 API clients left.
[2020-09-02 07:42:59 +0000] information/ApiListener: New client connection for identity ‘icinga-master1.localdomain’ from []:41279
[2020-09-02 07:42:59 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint ‘icinga-master1.localdomain’.
[2020-09-02 07:42:59 +0000] information/ApiListener: Sending config updates for endpoint ‘icinga-master1.localdomain’ in zone ‘icinga-master1.localdomain’.
[2020-09-02 07:42:59 +0000] information/ApiListener: Finished sending config file updates for endpoint ‘icinga-master1.localdomain’ in zone ‘icinga-master1.localdomain’.
[2020-09-02 07:42:59 +0000] information/ApiListener: Syncing runtime objects to endpoint ‘icinga-master1.localdomain’.
[2020-09-02 07:42:59 +0000] information/ApiListener: Finished syncing runtime objects to endpoint ‘icinga-master1.localdomain’.
[2020-09-02 07:42:59 +0000] information/ApiListener: Finished sending runtime config updates for endpoint ‘icinga-master1.localdomain’ in zone ‘icinga-master1.localdomain’.
[2020-09-02 07:42:59 +0000] information/ApiListener: Sending replay log for endpoint ‘icinga-master1.localdomain’ in zone ‘icinga-master1.localdomain’.
…
We are using 2.11 version with 1 master + satellite + agents (using director as configuration tool). Same thing is also reproducible on 2.12 version