Three Level distributed Monitoring - IDO feature with 2 Masters

Hey,
i’m testing around with Icinga2 and Saltstack and get stuck with the question how to configure the ido feature for my 2 icinga2 masters as HA zone.
I have 2 master as HA zone with a separate mariadb server and a separate server for icingaweb2.
For example:
icinga2-master01
icinga2-master02
icinga2-mariadb01
icinga2-web01 (icingaweb2)

How must the ido feature be set up?
Only activated for master01?
Activated for both masters writing into 1 db?
Activated for both masters writing into there separated db?

Can someone please help me out of this?

Thanks a lot!
Best regards,
Darkentik

I found the docs about the IDO and have a question.
https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#distributed-monitoring-high-availability-db-ido

While using top-down-config-sync, i thought only the masters must have IDO activated?
But i’m confused. Must all Icinga2 instances have ido activated?

Hi,

are you using pillar? if yes it’s not difficult:

  1. Create a jinja template file with the options from the doc like you want to use (maybe you want also use the automatic clean up feature)

  2. copy the file into /etc/icinga2/features-available/ - with the salt function: file.managed

  3. use the salt function “cmd.run” to run the (maybe you should test if the file is copies or isn’t enabled before with the onlyif or unless parameter - in the case you are doint the high state again and again)

icinga2 feature enable

About the database:
You wrote you are using one DB-Server and no local database. In this case the HA feature is valid for you - like written in the docs.
So you have to enable the parameter “enable_ha”. After this the two master agree with each other who is writting into the DB.

@stevie-sy Thanks for reply.
I’m more experienced with Saltstack than Icinga2. :slight_smile:
I found the optional parameter for enable_ha in the docs and always had have it build in my salt state but maybe a copy paste error caused that only one of 2 masters had this setting.
So i fixed it and gave both of them the parameter and now the backend ist up and running, icingaweb2 is happy, BUT in Icingaweb2 i got no host, no service.
My master02 was picked as the IDO feature holder and he pushes info into the mariadb. But the master01 is the active one in the HA zone with all config data structure under /etc/icinga2/zone.d/[stuff].
But this is a offtopic problem which i consider in another thread.

Back to topic:
The docs are unclear for me which icinga2 node has to use the IDO feature.
Must it be installed and activated on all machines that should be monitored with a icinga2 agent?
Or is it enough on my 2 masters using top-down-config-sync?

Thanks.

Hi @Darkentik,

only the masters need to have the IDO feature enabled.

Make sure both masters are connected to each other and the second master should have accept_config = true in /etc/icinga2/features-enabled/api.conf.

If you’re not sure on how to connect the masters to each other, make sure to have a look that the High-Availability Master with Agents section in our docs.

Greetings
Noah

@nhilverling Thanks for your reply.
I have configured in the zones.conf of master01 the master02 with “host” attribute to active connect to the second master.
Does the 2nd master only need “accept_config = true” or “accept_commands= = true” too?
This is a hint that i don’t read in the docs and as newbie to Icinga2 don’t got in mind while learning it.

Thanks for Support!

We have both parameter on every node in our cluster - that means master 1 (config-master incl Director), maste2, satellites, agents

@stevie-sy I think you missunderstand me.
Does the Master2 need accept_commands to be true or false?
Does the Master2 get commands from Master1?

The ApiListener looks on every our node like this:

object ApiListener "api" {
  ticket_salt = TicketSalt
  accept_config = true
  accept_commands = true
}

@Darkentik you don’t needed to set accept_commands on your second master. It’s only needed, if you want to use command_endpoint from your first master to your second one. This is not something many people do and I would not recommend it, if you’re just getting started with Icinga.

If your masters are still not connected or you can’t see anything in your Icinga Web 2 you should check the following things:

  • cat /var/log/icinga2/icinga.log | grep master-2 on your first master to see if the second master is even trying to connect. A common thing would be (certificate validation failed: code 18: self signed certificate). If you see this, you might have forgotten to sign the second masters certificate with icinga2 ca list and icinga2 ca sign <fingerprint>.

  • cat /var/log/icinga2/icinga.log | grep master-1 on your second master to see if there’re connection problems

@nhilverling Thanks for your answer.
My 2 Masters are now connected. The Problem was, that i forgot to rollout the api.conf for the 2nd master too where “accept_config = true”.

Just a thought: The active master, in my case master1, does not need “accept_config = true” because he is the active master or?
I have set its “accept_config = false” and “accept_commands = false”.

Any objections that i rename the topic so we can handle more stuff than the ido things here?
The Three-Level Setup is so much complex so i otherwise have to open some more threads for its detailed config.
I think what would be better now. Continue here or split all different .conf and HA stuff into multiple threads?

Yes, master-1 should not accept config from anywhere, because it should be the one providing config.

I think it might get too hard to follow this thread. So just mark this thread as resolved by accepting one answer as the solution and open new ones for your other problems :slight_smile:

Ok Thanks a lot for the good support!
I will continue in new threads. :smiley: