Hello community,
I am administrating a rather big icinga2 environment with two masters in an active/active scenario where master1 and master2 are in zone master and all subordinate nodes have their own zones being their FQDNs (customer satellites).
My team colleague, who left the company a while ago, thought it was a great idea to make
/var/lib/icinga2/api/packages/director a glusterfs shared volume over the two masters. He then removed the credential from the configuration under Director → Icinga Infrastructure → Endpoints → master2 which also led to strange timeouts when deploying the config anew.
One day it came to be that the gluster daemon was announcing
Transport endpoint not connected
which led to many hosts and services being stale because part of the director’s configuration was not physically available on the filesystem.
I have removed the glusterfs configuration and re-entered the api credentials on master2.
However, today we created a host inside the director and it was only synced to master2, not to master1. Since master1 is currently also the active icingaweb2 host, the host is not showing in the Icingaweb2 Overview either.
I tried the suggestion from: Icinga Director config sync / deploy between two masters - #17 by francesco.cepparo
on master2 but the API call is giving me:
{
"error": 500.0,
"status": "Failed to delete package 'director-global'."
}
Before that I also tried to just delete the .authoritative files for the zone the host was created in. But neither of these approaches helped.
Currently master2 has more notification objects and seems to be the configuration I want to keep.
What would be the next steps in order to get the masters back in sync, to make master1 authoritative again and successfully remove the config package from master2 so that it only receives configs from master1 again?
Version Info:
Icinga 2.12.0-1
Icingaweb2 2.7.3
Director 1.6.2