We initially configured MySQL Master-Master between the two sites, but we frequently get duplicate key errors so it seems this is not the right way to go.
In the documentation ( Distributed Setups - Icinga DB ) we see:
For each Redis® server you need to set up its own dedicated Icinga DB instance that connects to it, but the Icinga DB instances must write to the same database, which of course can be replicated or a cluster. So the steps from the standard Icinga DB installation documentation can be followed. However, as mentioned, the database only needs to be set up once.
It is mentioned here that MySQL Replication is OK, but it seems IcingaDB should write to the same master to prevent any issues. At the same time, the documentation suggests that only a single MySQL instance is fine as well. Because of this, I have the following questions:
What is the role of MySQL? I.e. if the data is written to Redis and MySQL, what does MySQL add that Redis does not?
What happens when the MySQL server goes down? Having only a single MySQL server means it is a SPOF. What breaks when MySQL is unavailable? How long can IcingaDB survive and how long until monitoring breaks?
In addition, will Icingaweb also be able to show data despite the MySQL backend being unavailable? I think it will, because it also connects to Redis, but I would like to be sure of this.
Thanks for giving Icinga DB a try and your questions.
In my personal experience, a master-master replication always comes with issues. Having a more conservative master-failover setup proved to be more stable. For similar reasons, PostgreSQL does not even support master-master.
Icinga 2 sends data to the Redis, using it as a queue. The Icinga DB daemon consumes this data and stores it in a relational database. Thus, the Redis is “just” a queue or ephemeral storage, while your MySQL stores the data for eternity, especially the history.
If you are building an HA setup with a single node MySQL, then it is not really HA; correct. So please have an HA database setup, e.g., using a good old two node MySQL setup, one active and one failover node.
If the relational database is gone, Icinga DB will have a hard time and give up after round about five minutes.
The Icinga DB Web module relies on the relational database for some data, especially the history. If your MySQL is down, Icinga DB Web will also be very unhappy.
So in a nutshell, your Icinga DB setup requires a relational database to be available. Hopefully my words may help you make more sense out of the graph at Distributed Setups - Icinga DB.
Thanks for your response. Very enlightening, especially that Redis is just meant as a “queue” before saving data to the database.
We have converted our database setup to active/backup, so data is always written to the same instance. Cheers!
Regarding Redis, I have noticed that the “active” Redis instance uses ± 140MB of RAM, and the Redis on the other master instance (where IcingaDB logs “Another instance is active” consumes 220MB of RAM. This appears to be a stable baseline, that does not decrease. So it seems that this data is not being flushed to the database and freed up. Fwiw this is with ± 600 hosts and ± 17.000 services monitored.
We have allocated 1GB of RAM to Redis.
Should we ensure that the Redis instance essentially is never allowed to reach 100%? Is there a recommendation for the Redis eviction policy? We have now configured noevict. Is this correct, or is it also fine to let Redis automatically clean up keys based on an LRU policy for example?
When writing that it “consumes 220MB of RAM”, do you mean that this is static? If so, I would guess that Redis does its memory management thing and keeps it reserved. Unless Redis starts nibbling away your memory, I would not worry.
Should be fine. However, I would advise to monitor the Redis server and Icinga DB itself. For the latter, please check out the icingadb check command.
While Icinga 2 writes into Redis, the Icinga DB daemon consumes the data and frees them up. Thus, there should be no need for a customized eviction policy.
The aforementioned icingadb check command creates multiple performance data metrics indicating the system state, including backlog sizes.
It doesn’t seem very extreme, I just expected it to go down if it is supposed to go to the database.
If these numbers don’t seem strange to you either I think we can assume everything is OK and as intended
We already had Redis memory monitoring but I will check out the icingadb check you mentioned as well!
