Icinga Director config sync / deploy between two masters

Dear all, I’m trying to set up a configuration with two masters. I set up the masters with the node wizard and then added each other in both /etc/icinga2/zones.conf files.

However, the Director elects a Config master, and deploys the configuration on only that master. If the config master goes down, or if some reason the “Active Endpoint” listed under Icinga Web 2’s Monitoring Heath page changes, the other master’s configuration is used. Is there a way to fix this so that the configuration stays synchronized between masters?

Thank you.

You can only have one config master, that being said the elected config master should sync its config to the other endpoint in its zone. What do your zones.conf and api.conf look like?

Hi, thank you for your answer. Here are my zones.conf and api.conf:
zones.conf on master 1
zones.conf on master 2
api.conf (same in both masters)

I really hope we can figure this out.

Looks normal to me. Keep an eye out for log messages from the ApiListener on your non config master. There should be a line like Applying config update from endpoint... or Ignoring config update for zone .... Log level information and warning respectively.

Hi, here are the log messages:

  1. From the non config master:
    [2019-06-27 15:34:33 +0200] information/ApiListener: Applying config update from endpoint ‘gnode3.ud.nohup.it’ of zone ‘master’.
    [2019-06-27 15:34:33 +0200] warning/ApiListener: Ignoring config update for zone ‘director-global’ because we have an authoritative version of the zone’s config.
    [2019-06-27 15:34:33 +0200] warning/ApiListener: Ignoring config update for zone ‘master’ because we have an authoritative version of the zone’s config.

  2. From the config master:
    [2019-06-27 15:34:35 +0200] information/ApiListener: Applying config update from endpoint ‘icinga2.ud.nohup.it’ of zone ‘master’.
    [2019-06-27 15:34:35 +0200] warning/ApiListener: Ignoring config update for zone ‘director-global’ because we have an authoritative version of the zone’s config.
    [2019-06-27 15:34:35 +0200] warning/ApiListener: Ignoring config update for zone ‘master’ because we have an authoritative version of the zone’s config.

Do you have any idea about what they mean?

Both endpoints either have configuration in /etc/icinga2/zones.d or in a deployed api config package which uses zones.d. I would guess that you’ve used the Director to deploy its package against both REST APIs. You can only do this for one endpoint, with a manual failover to the secondary host.

In order to clean things up, delete the director config package on the second master endpoint. This can be done with a curl request as explained in the API docs for Icinga 2.

Cheers,
Michael

Dear Michael, I removed the config package on the second master with a curl DELETE. The synchronization seems to work now (or at the very least, the following file is kept in sync: /var/lib/icinga2/api/zones/master/director/hosts.conf). However, when I disable the first master and the second master becomes the active endpoint, the list of hosts and services to check becomes empty. Do you have any idea?

This would imply that the secondary master does not have any configuration, or the IDO feature is not active on this endpoint. What’s the output of the monitoring health view in Icinga Web 2 for this?

Cheers,
Michael

Hi, here is the screenshot:

This is strange, since I do have ido-mysql enabled on both masters, and they use the same database.

And those missing monitoring objects are really active on this instance? You can query such via the REST API e.g. at the /v1/objects/hosts endpoint.

Cheers,
Michael

They are not, the REST call returns {“results”:[]}
On the config master instead, the same REST call properly returns all monitoring objects.

So, nothing is synced from the first master to the secondary master. Check the logs, is there really an entry on the secondary master that the config is synced?

Hi, I looked at the logs but it seems that the configuration is properly synced. All files in /var/lib/icinga2/api/zones/director-global/director and /var/lib/icinga2/api/zones/master/director are the same on both masters.

It is as if the configuration inside /var/lib/icinga2/api/zones is ignored by the second master.

Please show the full tree of that directory, e.g. ls -lR /var/lib/icinga2/api/zones. Also, run icinga2 daemon -C -x notice and look for the included directories.

1 Like

Hi, here are the dir trees (ls -laR /var/lib/icinga2/api/zones):
dirtree-icinga2.ud.nohup.it.txt (2.0 KB)
dirtree-gnode3.ud.nohup.it.txt (2.0 KB)

And here are the outputs from the icinga2 daemon -C -x notice on the two nodes:
icinga_notice-icinga2.ud.nohup.it.txt (10.3 KB)
icinga_notice-gnode3.ud.nohup.it.txt (6.3 KB)

It indeed seems like the configuration under /var/lib/icinga2/api/zones is ignored in the second master.

Ok, it still thinks it is authoritative for this zone. Purge the content of /var/lib/icinga2/api/zones/* on the secondary master and restart it.

1 Like

Thank you, it worked!!! I deleted the whole folder as you suggested, but I wonder if deleting

/var/lib/icinga2/api/zones/master/.authoritative

and

/var/lib/icinga2/api/zones/director-global/.authoritative

would have been enough. Oh well. I guess I’ll try it if I ever need to switch config masters :slight_smile:
Thank you for your precious help! I was totally stuck.

1 Like

Glad you’ve solved it. Deleting the two files would have been enough, but I wanted to ensure that Icinga re-dumps everything from a cluster config sync without manual interaction.

Instead of editing the topic’s subject, please pick a reply and mark this as solution. This helps others to search for solved topics, and also grants a badge for those being helpful the most.

Cheers,
Michael

1 Like

Thanks Michael. This steps helps me lot for debugging my master-master-installations.

Hello Michael,

I am having similar problems. I followed the instructions to create two masters from here:
high-availability-master-with-agents

I have also set up icingaweb2 and director on both masters all sharing a MySQL DB. But the sync does not seem to be happening on the second master. I checked on the community site and a post that recommends this:-

In order to clean things up, delete the director config package on the second master endpoint. This can be done with a curl request as explained in the API docs for Icinga 2.

I understand how to execute a curl API request but what is the URL for deleting the directory config package on my second master endpoint?

I hope you can provide some assistance.

Kind regards
Peter Goudman