Multi-Master sync issues

Icinga 2
, ,
1

Hi!

I’ve got some sync issues after I added a second master to my setup.
First, I set up a debian VM with icinga 2, following this how-to:
https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#high-availability-master-with-clients

I’ve set “accept_config = false” for API on the primary master (which also holds Director) and “accept_config = true” on the second one (director not installed).

This is what best describes the problem:

mon-master01:
icinga2 object list --type Endpoint | grep -i examplerouter

Object ‘examplerouter.network.local’ of type ‘Endpoint’

(SUCCESS)

mon-master02:
icinga2 object list --type Endpoint | grep -i examplerouter

-nothing-

(FAIL)

systemctl stop icinga2
rm -rf /var/lib/icinga2/api
systemctl start icinga2
icinga2 object list --type Endpoint | grep -i examplerouter

Object ‘examplerouter.network.local’ of type ‘Endpoint’

CONFIG

Identical on both nodes, only NodeName in constants.conf is different!

root@mon-master01:~# cat /etc/icinga2/zones.conf

object Endpoint "mon-master01.example.com" {
        host = "192.168.83.10"
}

object Endpoint "mon-master02.example.com" {
        host = "192.168.83.11"
}


object Zone "master" {
        endpoints = [ "mon-master01.example.com", "mon-master02.example.com" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

STATS:

[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1320 Services.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 580 Hosts.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 945 Dependencies.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 4 NotificationCommands.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1758 Notifications.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 333 Zones.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 332 Endpoints.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 User.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 215 CheckCommands.
[2019-07-29 19:01:30 +0200] information/ConfigItem: Instantiated 1 UserGroup.

During rollout phase, there are a lot of changes that are replicated from our CMDB to the Director API. There might be a deployment in director terms every 5 mins.

What am I doing wrong here?
Or is this a bug, maybe this one?
https://github.com/Icinga/icinga2/pull/7150

Comments are welcome! Thank you!

2

Hi,

the linked PR refers to runtime created objects via the API, the Director uses a different method with config packages.

I’m not sure what’s going on with your object list greps, in case you’d want to see whether Endpoint objects are created, query /v1/objects/endpoints via the REST API instead.

How’s the deployment done inside the Director, can you trace the deployment being synced from both logs, master1 and master2?

Cheers,
Michael

3

I wanted to show that the second master does not pull updates, it shows old data.
When I add a new endpoint, the config is not transferred to master02. If I delete the “api” directory and start Icinga2, it is pulled once.

Director is running on master01 and notifies the local daemon (at least it should, master01 lists all endpoints). The second master does not sync correctly.

mon-master01:

[2019-07-30 16:21:37 +0200] information/ApiListener: Reconnecting to endpoint ‘mon-master02.example.com’ via host ‘192.168.83.11’ and port ‘5665’
[2019-07-30 16:21:38 +0200] information/ApiListener: New client connection for identity ‘mon-master02.example.com’ to [192.168.83.11]:5665
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished reconnecting to endpoint ‘mon-master02.example.com’ via host ‘192.168.83.11’ and port ‘5665’
[2019-07-30 16:21:38 +0200] information/ApiListener: Sending config updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Syncing configuration files for zone ‘master’ to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished sending config file updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Syncing runtime objects to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished syncing runtime objects to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished sending runtime config updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Sending replay log for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished sending replay log for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished syncing endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Reconnecting to endpoint ‘mon-master02.example.com’ via host ‘192.168.83.11’ and port ‘5665’
[2019-07-30 16:26:36 +0200] information/ApiListener: New client connection for identity ‘mon-master02.example.com’ to [192.168.83.11]:5665
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished reconnecting to endpoint ‘mon-master02.example.com’ via host ‘192.168.83.11’ and port ‘5665’
[2019-07-30 16:26:36 +0200] information/ApiListener: Sending config updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Syncing configuration files for zone ‘master’ to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished sending config file updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Syncing runtime objects to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished syncing runtime objects to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished sending runtime config updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Sending replay log for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished sending replay log for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished syncing endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Reconnecting to endpoint ‘mon-master02.example.com’ via host ‘192.168.83.11’ and port ‘5665’
[2019-07-30 16:31:38 +0200] information/ApiListener: New client connection for identity ‘mon-master02.example.com’ to [192.168.83.11]:5665
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished reconnecting to endpoint ‘mon-master02.example.com’ via host ‘192.168.83.11’ and port ‘5665’
[2019-07-30 16:31:38 +0200] information/ApiListener: Sending config updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Syncing configuration files for zone ‘master’ to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished sending config file updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Syncing runtime objects to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished syncing runtime objects to endpoint ‘mon-master02.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished sending runtime config updates for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Sending replay log for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished sending replay log for endpoint ‘mon-master02.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished syncing endpoint ‘mon-master02.example.com’ in zone ‘master’.

mon-master02:

[2019-07-30 16:21:38 +0200] information/ApiListener: New client connection for identity ‘mon-master01.example.com’ from [192.168.83.10]:14772
[2019-07-30 16:21:38 +0200] information/ApiListener: Sending config updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Syncing configuration files for zone ‘master’ to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished sending config file updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Syncing runtime objects to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished syncing runtime objects to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished sending runtime config updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Sending replay log for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Applying config update from endpoint ‘mon-master01.example.com’ of zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished sending replay log for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:21:38 +0200] information/ApiListener: Finished syncing endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] warning/JsonRpcConnection: API client disconnected for identity ‘mon-master01.example.com’
[2019-07-30 16:26:36 +0200] warning/ApiListener: Removing API client for endpoint ‘mon-master01.example.com’. 0 API clients left.
[2019-07-30 16:26:36 +0200] information/ApiListener: New client connection for identity ‘mon-master01.example.com’ from [192.168.83.10]:19866
[2019-07-30 16:26:36 +0200] information/ApiListener: Sending config updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Syncing configuration files for zone ‘master’ to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished sending config file updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Syncing runtime objects to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished syncing runtime objects to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished sending runtime config updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Sending replay log for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Applying config update from endpoint ‘mon-master01.example.com’ of zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished sending replay log for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:26:36 +0200] information/ApiListener: Finished syncing endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] warning/JsonRpcConnection: API client disconnected for identity ‘mon-master01.example.com’
[2019-07-30 16:31:38 +0200] warning/ApiListener: Removing API client for endpoint ‘mon-master01.example.com’. 0 API clients left.
[2019-07-30 16:31:38 +0200] information/ApiListener: New client connection for identity ‘mon-master01.example.com’ from [192.168.83.10]:24890
[2019-07-30 16:31:38 +0200] information/ApiListener: Sending config updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Syncing configuration files for zone ‘master’ to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished sending config file updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Syncing runtime objects to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished syncing runtime objects to endpoint ‘mon-master01.example.com’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished sending runtime config updates for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Sending replay log for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Applying config update from endpoint ‘mon-master01.example.com’ of zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished sending replay log for endpoint ‘mon-master01.example.com’ in zone ‘master’.
[2019-07-30 16:31:38 +0200] information/ApiListener: Finished syncing endpoint ‘mon-master01.example.com’ in zone ‘master’.

How can I debug further? Director was running on master02 before and was removed / transferred to mon-master01 (prev. prod. system). All director files have been removed from mon-master02.
As far as I can tell, Director reloads Icinga locally. How can I check this? Maybe there are traces in the DB that are wrong now?

As far as I understand from the Icinga2 book and the docs, it should be sufficient to let Director deploy on mon-master01 and disable “accept_config” there, while enabling it on all other nodes. At least for me, this does not seem to be enough.

4

Hi,

the Director uses the config packages from the REST API. Each stage deployed inside such a package will trigger config validation and a reload if that succeeds. The config packages have been modeled around the Icinga Director requirements.

Since master02 was previously the API endpoint for the Director, this may indicate that it still thinks that it is authoritative for the deployment. The debug log should unveil more things on why specific zone syncs are ignored.

Cheers,
Michael

5

Hi,

I don’t need the history. Can I purge some folders and run a deployment from Director?
In the meantime I will try the debuglog.

Regards
Kevin

6

Currently, the cluster is stable. I’m not 100% sure but I might have only removed a folder under “api” the first time. Last time, I removed the whole API folder to force re-sync and was unable to reproduce the problem yet.