Cluster 2 icingaweb2 and one director instance together

Hello! I generally followed this guide here:

which works OK. I would like to cluster all three icinga instances created this way. I see people talk about a samba share, but I was unable to find any documentation except this:

which seems extraordinarily complex. Why not use a shared disk? But, what directory should be shared, the entire /etc/icinga2 directory?

Our requirements are to also have one master server with two other satellite servers, which I would add those using the director and add it to the icinga infrastructure? or should I be doing something else?

thank you for your help!

Hi!

I’m not sure what you mean with “clustering all three instances” … ?

If you have executed the How to then you would have three Master Instances with a Director which would be hard or non-clusterable because you would have three standalone Systems which you were trying to merge.

To better understand the concept of how a distributed icinga2 Setup would look on a larger scale i would recommend the following Distributed Monitoring - Icinga 2

As you would need to think of that you’ve might have created a setup like this multiple times.

.

The whole icinga2 API and director setup is there to replace the Config Sync in the first place because it takes care of the distribution of the config files.

But i might be wrong … if you could explain in more detail what you want to achieve it might be easier for others as well to help you :slight_smile:

Regards

David

+------------+        +--------------+       +------------+
| Sat 1     |          |     Sat 2   |       | Sat n      |
+------------+         +--------------+       +------------+
        \           /                    /
         \         /                    /
       +-------------+                  +-------------+
       |  Master 1   |       <===>       |  Master 2   |  (Master-Zone)
       +-------------+                  +-------------+
             ^                                 ^
             |   Icinga 2 REST API   :
             |                                    :
           +----------------------------+.      +------------------------+.  
           |       Icinga Director      |.       |   Icinga Director.   |
           +----------------------------+.      +------------------------+
                             \                                   /
                               \                              /
               +--------------------------------------------+.  
               |      load balancer      |
               +--------------------------------------------+
                             | 
                             | 
                  +--------------------------------------------+.  
                  |      users                               |
                  +--------------------------------------------+

I would like to do something like this. The Users would access either through an load balancer to the director. I would like the icinga Directors to have the same information.

one last thing, each of these boxes in the diagram are separate vm’s.

Hi !

I think i get the idea what you are trying to accomplish but … i think the most relevant part isn’t even touched at the moment.

       +---------------------------------+
       |                                 |
       |            Users                |
       |                                 |
       +--------------+------------------+
                      |
                      v
       +--------------+------------------+
       |                                 |
       |      Loadbalancer               |
       |                                 |
       +--------------+------------------+
                      |
                      v
       +--------------+--------+---------+
       |     multitude of Web2 + Director|
+------+     Module Instances            |
|      |                                 |
|      +---------------+-----------------+
|                      ^
|                      |
|      +---------------+-----------------+
|      |                                 |
|      | the whole master/satellite Zone |
|      | construct                       |
|      |                                 +------------------+
|      |                                 |                  ^
|      |                                 |                  |
|      |                                 |                  |
|      +---------------------------------+                  |
|                                                           |
|                                                           |
|      +---------------------------------+                  |
|      |                                 |                  v
|      |                                 +------------------+
+----->+        Databases / Galera or    |
       |        other HA setup you       |
       |        might like               |
       |                                 |
       +---------------------------------+

The whole director and icinga ido setup is based around databases.

The IDO is for the collected data from the checks and it is the base where icingaweb2 draws it’s representation data from. The second database ist the director which serves as a configuration database. from the loadbalancing and failover standpoint you need to have a failover for the not reachable nginx or apache2. And there is no problem there to have a multitude of icingaweb2 pointing at the same database for representation… now we come to the more painful part … you want to avoid at all costs to have a multitude of director databases with different states of content.

you can have a replicated director database if the database host which houses the ido & the directordb dies so you can have a failover switch to the replicated one but for the icinga2, the icinga web 2 and the director those resources need to be at least in their representation to the system one …

Because the director module has its own API it needs to write to one consistent target to create consistent config for icinga2 which is distributed to the satellites and endpoints.

So i think the point should be. How to use as an example galera mysql as a resource for the databases and how to setup ha failover for the icingaweb2.and its modules.

Regards

David

Sorry, I left that part of the diagram out. The director’s are pointing to proxysql, which points to a three node galera database. Are you saying that is all I need to do to cluster, is simply cluster on the database side?

As i said above the trickiest pony is the database replication. if it is possible in your setup is something i can’t say … i don’t know your exact Setup and your needs.

In the first post you implied that you would like to have a multitude of icingaweb2 instances where i guess you want to handle the usage/load of users working with the system …, as a viewpoint for handling incidents i think this would be fine.

If … you think to have a multitude of users adding config via the director … i think i wouldn’t recommend this, because the director isn’t made to verify multiple configurations from different users. Also this would be hard to backtrack if someone would deploy some sort of nonsense configuration which is syntactically correct … but otherwise wouldn’t make sense. Or even worse someone disabled systemwide the notification …

You have to keep in mind that the director has its own API … which interacts with the icinga2 core and core API …, So as a Fallback if the Original director instance dies this would be fine i think… but beware multiple people deploying from mulitple directors into the same database is a recipe for disaster…

If you keep this in mind, and have a plan for a loadbalanced web front end, then you just have to take care of the databases as i said.

Regards

David

Hi,

We do something similar to this - two masters, and two Docker containers with IcingaWeb + Director, and all of our databases live on a Galera cluster.

The services are split across two facilities and we both load balance and advertise the public IP address from both locations, so we achieve georedundancy in case of an outage at either location.

which seems extraordinarily complex. Why not use a shared disk? But, what directory should be shared, the entire /etc/icinga2 directory?

Thanks for reminding me…I wrote said Tutorial and have to say that indeed, it is quite complex. Of course you can achieve quite everything with a shared disc. My setup nowadays has a MooseFS-Share where most of the configuration-files and check-plugins are located.

I was thinking about your tourmaline when I attempted to cluster the director. We did not use a shared filesystem, and I was not sure that it would work correctly, but according to someone else on this thread,the database is the dependency. We are confident in our database set up, and have yet to experience an issue.

We implemented this solution by load balancing the director instances in a failover configuration, i.e the second instance is never reachable until the first instance is offline. We areusing a 3 node Galera database to store data fronted with proxysql.

1 Like