Hello All!
I had an existing Icinga2 HA 2-master environment with 4 zones
I have since integrated IcingaDB into the mix, and have a few questions about HA and how the Mysql database for IcingaDB is involved.
Prior to IcingaDB, if any of the masters failed, the other would continue as normal as it had a local copy of the IDO database (ha=false)
With IcingaDB, both masters have to write to the same Mysql database as well as write to their own local redis database.(Im still not entirely sure what this redis database is for, which may be part of my issue, i think it stores current host and check info? 
)
The Mysql database is used to communicate cluster events (i think), like who is current owner, etc…
As it stands, the Mysql database for IcingaDB is running on the Master-1 server (Configuration Master)
I have the Master-2 server IcingaDB (config.yml) pointing to Master-1 as its database.
With this is setup, everything seems to be working great. Messages between the two masters about HA look good and all is well.
At least in my environment, the issue i have is that the Mysql server is a single point of failure with IcingaDB. Even if i were to offload that mysql role from the master and put it on a dedicated server, that dedicated server is now the point of failure.
I experimented with Mysql Replication and shipping a copy of the IcingaDB from Master-1 to Master-2 and enabled both to be mysql slaves of eachother, and then pointed icingadb at their local instance.
This worked for a little while but with a few manually triggered HA failovers, the db corrupted beyond what i was prepared to fix (had a backup)
So the real question is, how do i setup a HA IcingaDB where the MysqlDB on Master-1 or any other dedicated mysql server isnt going to bring IcingaDB down when they both think they are the primary?
Again, with the “old” IDO way, ha=false just works…
Is this possible with IcindaDB?