Split-brain scenario for two HA masters across two datacenters

[matthew.smith (Matt)]

I was wrestling with the same problem recently, I needed the monitoring to remain up at both datacenters when one went offline.

This post is where I asked about this

Event streams between Icinga masters

Yes, each satellite can have their own icingadb instance (and webinterface). By this you can have separate webinterfaces for your satellite zones with only the visibility of the zone. The satellites will still forward their data to the master, where you then have the complete overview of your infrastructure.

The solution that worked was to have

• HA masters spanning DC1 and DC2 with it’s own db and icingaweb
• (HA|standalone) Satellite in DC1 with it’s own db and icingaweb
• (HA|standalone) Satellite in DC2 with it’s own db and icingaweb

Each of these is running Icinga2, Icingadb stack with 1 database per zone above, there is also a icingaweb per zone. The databases for each zone are independent from each other and contain different data.

The masters will have a complete view of both DC’s, in the event of a failure one DC goes dark and results/deployment to/from that DC is broken. If the DC is down deployment of new checks isn’t a priority :).
Each satellite will have a view of it’s own DC’s results all the time.

Note the above setup was designed to account for complete DC ip connectivity failure but the DC1 would still be active and transmitting non ip data, people onsite needed to know what was up local to them.