Hi,
After running Icinga for Kubernetes (I4K) for a week or so, the mariadb database became full, all connections were stalled, and ultimately the database was corrupted. Not a real issue as this was a proof of concept in a test environment.
However, now is the question: what are the best practices when designing and deploying I4K?
In our case, we have 50 clusters running each many pods and nodes, and I understand we’d better plan and size the I4K infrastructure properly.
Has anyone used I4K in an enterprise setup? What are the recommendations and findings for a smooth deployment?
Some more technical questions:
- Is I4K keeping a history of the environment, or just a real-time image? If it keeps a history, for how long, or how many versions of the environment does it keep? Are there parameters that can be set to ensure roll-over? What is the value of keeping historical data about what pods or nodes were running yesterday or two days ago, anyway?
- How much disk space is to be foreseen per cluster, for the database?
- For each cluster, how many DB connections does I4K use, and for how long? How many concurrent DB connections need to be foreseen? How long is each connection kept active? Or is each connection permanent, which is not what I think I observed? If the connections are not permanent, how often does I4K reconnect to the DB?
Thanks a lot in advance for sharing your experience! Ours so far has not been very conclusive, as the 200MB of DB space have been used up in less than a week, which was totally unexpected.
Wishing you all the best,
Jean
NB: The users love the information they see, and we really want to make I4K work! Kudos to the development team, and keep up the great job!