Seeking advice on HA design

I have 2 data centres with a master node in each, I’m attempting to create a topology where if a datacentre is flooded, or goes offline the other master node should continue to monitor the entire network.

I’ve already setup Icinga2 with IcingaDB in a HA configuration, but where I’m stuck and seeking advice is on the IcingaDB system, as my current SQL server is a single point of failure.

Should I try using MariaDB with Galera and create a 3 node cluster to replicate data between all MariaDB instances? or is MariaDB’s inbuilt replication suitable?

If I do go down the route of multiple SQL servers, should each Icinga2 node write to the local SQL database?

As mentioned, I have infrastructure in 2 data centres, but the DC’s are some 600 kilometres apart from each other, I don’t really have a layer2 network between the two, so each icinga node is in a different subnet, most examples I’ve seen of DB clusters seem to have all the instances in the same local area network.

I’ve included a very basic topology diagram to help explain the topology I have to work with.

We use a Galera cluster and I have no idea how it’s setup as I rely on the DBAs for such decisions.

Maybe we have a DBA in the forum that can help with the DB decisions or you could ask the Community | Galera Cluster for MySQL .