IcingaDB HA Mysql Database

Icinga DB
1

Hello All!

I had an existing Icinga2 HA 2-master environment with 4 zones

I have since integrated IcingaDB into the mix, and have a few questions about HA and how the Mysql database for IcingaDB is involved.

Prior to IcingaDB, if any of the masters failed, the other would continue as normal as it had a local copy of the IDO database (ha=false)

With IcingaDB, both masters have to write to the same Mysql database as well as write to their own local redis database.(Im still not entirely sure what this redis database is for, which may be part of my issue, i think it stores current host and check info? :wink: )

The Mysql database is used to communicate cluster events (i think), like who is current owner, etc…

As it stands, the Mysql database for IcingaDB is running on the Master-1 server (Configuration Master)

I have the Master-2 server IcingaDB (config.yml) pointing to Master-1 as its database.

With this is setup, everything seems to be working great. Messages between the two masters about HA look good and all is well.

At least in my environment, the issue i have is that the Mysql server is a single point of failure with IcingaDB. Even if i were to offload that mysql role from the master and put it on a dedicated server, that dedicated server is now the point of failure.

I experimented with Mysql Replication and shipping a copy of the IcingaDB from Master-1 to Master-2 and enabled both to be mysql slaves of eachother, and then pointed icingadb at their local instance.

This worked for a little while but with a few manually triggered HA failovers, the db corrupted beyond what i was prepared to fix (had a backup)

So the real question is, how do i setup a HA IcingaDB where the MysqlDB on Master-1 or any other dedicated mysql server isnt going to bring IcingaDB down when they both think they are the primary?

Again, with the “old” IDO way, ha=false just works…

Is this possible with IcindaDB?

2

I know my initial post was alot to consume, but does anyone have any insight or comments on how to fully protect a native IcingaDB installation from a Mysql outage?

Similar to IDO feature ha=false where both masters have a state copy

Any IcingaDB Devs out there that can chime in? I can take the heat :wink:

3

We’re currently deploying IcingaDB in a HA setup for the first time and are wondering the same.
Is it enough to have only one MySQL instance running on one of the masters for full HA?
Will the Redis Database on the other master cache all information until the MySQL DB is available again?

4

I stumpled upon your question while preparing migration from IDO to DB. As no one replied, I want at least drop my 2 cents.

I think using icinga-ido with ha=false in a 2-Node-Cluster-Setup is in general not a good idea - if anything fails, either icinga or mysql on one half of the setup, your ido-databases will differ.

I’m using a 3-node mariadb/galera cluster with ha=true and never had any issues for years (big setup).

With icinga-ido, you could also configure multiple db-backends. E.g., ido1 with ha=true would write to mysql-server 1 and ido2 with ha=true would write to mysql-server 2. I don’t know whether this is possible with IcingaDB.