Icinga2 highavailability not working as expected

Icinga 2
1

Hi Team,
I’m seeing an issue with my icinga2 highavailability setup with two masters.

Below are the version details

Icinga2 - 2.11.2
Icingaweb2 - 2.7.3
PHP - 7.3.13
Apache - 2.4.41
OS - RHEL 7.6
Database - postgress

Below is my setup in Highavailaiblity with two masters

icinga2-master1.localdomain is the config master master node. with zone.d with zone name as bbmmaster
icinga2-master2.localdomain is the secondary master master node without config in zones.d. this node configured as satellite node to accept the connection and commands from master1

In both icinga2-master1.localdomain and icinga2-master2.localdomain having same zones.conf file - zones.conf

object Endpoint “icinga2-master1.localdomain” {
host = “xx.xx.xx.xx”
}

object Endpoint “icinga2-master2.localdomain” {
host = “xx.xx.xx.xx”
}
object Zone “mbbmaster” {
endpoints = [ “icinga2-master1.localdomain”, “icinga2-master2.localdomain” ]
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

In icinga2-master2.localdomain could able to see all the bbmmaster folder and underlying configurations in below folder
/var/lib/icinga2/api/zones-stage/

But the failover is happening in reverseway…I mean always master1 is showing as down, master2 is showing as up.

when we stop the services in master2 then it is failovering to master1

the checks being loadbalanced in both master1 and master2

Can you please help me to understand the configuration perspective any issues are there?

With Regards,
Krishna Rajapantula

2

Hi, where is it shown as down? cluster-zone check?

3

Thanks for the response.

In icingawebconsole it is showing as icingabackend not running.

With regards,
Krishna.

4

do you have ido2db on both masters activated? Both masters write to the same db?

5

Yes ido2db is enabled. It writes to individual dbs in both the masters installed seperately.
With Regards,
Krishna

6

Thats not possible. Both masters need to write to the same database.

There is an example:
https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#distributed-monitoring-scenarios-ha-master-agents

7

I beleive that is as per the best practice.but still we can write the data to individual dbs as per the below paragraphh. please correct me if my understanding is wrong.

Fyi…I have installed icinga2 in two new instances and successfully enabled HA with the same procedure as mentioned in my initial query. But this is not working in my initial two servers…HA failover is working in reverse way

In the icinga docs below is the paragraph I was referring to w.r.t ido2db.
One possibility is to use a dedicated MySQL cluster VIP (external application cluster) and leave the IDO feature with enabled HA capabilities. Alternatively, you can disable the HA feature and write to a local database on each node. Both methods require that you configure Icinga Web 2 accordingly (monitoring backend, IDO database, used transports, etc.).

With Regards,
Krishna.

8

No idea what this paragraph you are reffering to, exacly means, because there is no explanation how to configure the icingaweb2 etc. for this case.

Both methods require that you configure Icinga Web 2 accordingly (monitoring backend, IDO database, used transports, etc.).

I think thats where the magic is coming. I have no idea how HA should be possible, without syncing the databases.

Honestly i never read about this approach and i have no idea how to configure this.

Maybe someone else can bring some light into it, as I’am interested in this method and which drawbacks it has.

Do you use two icngaweb2? If not, how does your icingaweb2 know in which databse it has to look?

Best Regards,
Rafael

9

Yes I use two icingaweb2 …

10

Any help or pointers for troubleshooting this issue please

With Regards,
Krishna

11

“One possibility is to use a dedicated MySQL cluster VIP (external application
cluster) and leave the IDO feature with enabled HA capabilities.”

That means you need a single database to which both HA machines connect; you
cannot have separate databases on each server.

“Alternatively, you can disable the HA feature and write to a local database
on each node.”

That means you an have separate databases, but only if you disable HA.

You appear to be trying to do both - HA with separate databases, and this does
not work.

If I have misunderstood, please state clearly what problem you are currently
trying to solve.

Antony.

12

Thanks Antony for the response.

Im writing to individula DBs from both the masters by disabling HA for idodb.
otherfeatures i have enabled HA
I have created a zone called mbbmaster in icinga2-master1.localdomain(Master1) in zones.d directory.

I have same zones.conf file in both icinga2-master1.localdomain(Master1) and icinga2-master2.localdomain(Master2).

In my original post i have updated my zones.conf file.

I could see the mbbmaster zone objects are reflecting in icinga2-master2.localdomain(Master2) in /var/lib/icinga2/api/zones-stage/ folder…

I have two individual icingaweb2 runningin master1 and master2.

Initially as per the docs icinga2-master2.localdomain(Master2) has been setup as a satellite node to master1 server.

Now in the webconsole master1 server I see the status as icinga2 backend is not running, but in master2 i could see the status of all hosts and services.

If i stop the services in master2 then it is failovering to master1.

What im expecpecting: ideally master1 should be up and running and it should failover to master2. But in my caes it is in reverse.

with Regards,
Krishna

13

I have a similar set up, I think one issue is your zones.con file. It should not be identical on both master servers but rather the host parameter should be set on only one of the two masters. Within a zone whether it be a master zone or child zone the connection direction ie. host parameter should be set just on the one node.

14

Hi Denis,

Thanks for the update.

Yes I have noticed that in the icinga docs accordingly i have made changes in my zones.conf file in master1 and master2. In master2 i have not specified the host attribute. But still no luck.

Also i have found a below comment from the post which is something related to this issue.

"Both masters are “live”. They load balance the checks, notifications and calculate who will be the main node for enabling HA aware features (DB IDO enable_ha=true for example).

There is no configuration to elect a master instance, that happens automatically upon reconnect of those nodes. Note: This is different to your configuration master which holds the configuration in zones.d."

With Regards,
Krishna

15

Hi Krishna, There lies your problem, if you are using two DB’s your enable_ha setting should be false. You use enable_ha=true only when you use a single VIP DB instance.

16

Yes, I agree…I have already set the enable_ha = false for idodb as im using two individual dbs…Still it is same.

with Regards,
Krishna