Hi all 
,
We are running a cluster setup with two master nodes in a master zone and 6 satellite nodes running on 3 zones, each zone having 2 satellite nodes in a production environment. Due to some hardware issue we shutdown our Master2 Node. After some time we decided to add Master2 Node again into the cluster but now Master2 won’t sync with Master1. We reinstalled Icinga2 again on Master2 Node and that didn’t work either. It does not create any file in /var/lib/icinga2/api/zones/ & /var/lib/icinga2/zones-stage.
Logs of Master2:
[2021-02-05 05:23:30 +0500] information/ApiListener: New client connection for identity 'Master01' from [X.X.X.X]:41388
[2021-02-05 05:23:30 +0500] information/ApiListener: Sending config updates for endpoint 'Master01' in zone 'master'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Finished sending config file updates for endpoint 'Master01' in zone 'master'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Syncing runtime objects to endpoint 'Master01'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Finished syncing runtime objects to endpoint 'Master01'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Finished sending runtime config updates for endpoint 'Master01' in zone 'master'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Sending replay log for endpoint 'Master01' in zone 'master'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Finished sending replay log for endpoint 'Master01' in zone 'master'.
[2021-02-05 05:23:30 +0500] information/ApiListener: Finished syncing endpoint 'Master01' in zone 'master'.
[2021-02-05 05:23:37 +0500] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2021-02-05 05:23:37 +0500] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2021-02-05 05:23:47 +0500] information/ApiListener: New client connection from [X.X.X.X]:41460 (no client certificate)
[2021-02-05 05:23:47 +0500] information/HttpServerConnection: Request: GET /v1 (from [X.X.X.X]:41460), user: <unauthenticated>, agent: check_http/v2.3.3 (nagios-plugins 2.3.3)).
[2021-02-05 05:23:47 +0500] warning/HttpServerConnection: Unauthorized request: GET /v1
[2021-02-05 05:23:47 +0500] information/HttpServerConnection: HTTP client disconnected (from [X.X.X.X]:41460)
[2021-02-05 05:24:30 +0500] information/JsonRpcConnection: No messages for identity 'Master01' have been received in the last 60 seconds.
Icinga2 version : 2.11.0-1
System Information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-957.27.2.el7.x86_64
Architecture: x86_64
Icinga2 Enabled Features on Master2:
Enabled features: api checker mainlog
icinga2 daemon -C gives following result:
[2021-02-26 19:16:26 +0500] information/cli: Icinga application loader (version: 2.11.0-1)
[2021-02-26 19:16:26 +0500] information/cli: Loading configuration file(s).
[2021-02-26 19:16:26 +0500] information/ConfigItem: Committing config item(s).
[2021-02-26 19:16:26 +0500] information/ApiListener: My API identity: tbe2-v01-img01
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 1 FileLogger.
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 6 Zones.
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 8 Endpoints.
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 1 ApiListener.
[2021-02-26 19:16:26 +0500] information/ConfigItem: Instantiated 234 CheckCommands.
[2021-02-26 19:16:26 +0500] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-02-26 19:16:26 +0500] information/cli: Finished validating the configuration file(s).
Thanks in advance,
Fasiha Fazal