After running Icinga for Kubernetes (I4K) for a week or so, the mariadb database became full, all connections were stalled, and ultimately the database was corrupted. Not a real issue as this was a proof of concept in a test environment.
However, now is the question: what are the best practices when designing and deploying I4K?
In our case, we have 50 clusters running each many pods and nodes, and I understand we’d better plan and size the I4K infrastructure properly.
Has anyone used I4K in an enterprise setup? What are the recommendations and findings for a smooth deployment?
Some more technical questions:
Is I4K keeping a history of the environment, or just a real-time image? If it keeps a history, for how long, or how many versions of the environment does it keep? Are there parameters that can be set to ensure roll-over? What is the value of keeping historical data about what pods or nodes were running yesterday or two days ago, anyway?
How much disk space is to be foreseen per cluster, for the database?
For each cluster, how many DB connections does I4K use, and for how long? How many concurrent DB connections need to be foreseen? How long is each connection kept active? Or is each connection permanent, which is not what I think I observed? If the connections are not permanent, how often does I4K reconnect to the DB?
Thanks a lot in advance for sharing your experience! Ours so far has not been very conclusive, as the 200MB of DB space have been used up in less than a week, which was totally unexpected.
Wishing you all the best,
Jean
NB: The users love the information they see, and we really want to make I4K work! Kudos to the development team, and keep up the great job!
Unfortunately, I cannot answer your questions regarding clear recommendations. However, I will try to answer some of your technical questions.
At the moment Icinga for Kubernetes just reflects the current state of your environment without any history. There are a few exceptions to this as there are container logs and metrics from prometheus. And in fact the events also provide some kind of history. There is no configurable retention, but metrics and events have an automatic cleanup running hourly, removing every entry older than one day. Container logs are currently capped to a size of 64KiB per container.
I cannot provide you a clear disk space recommendation, but I’m afraid that 200MB of disk space is far too little for 50 clusters. For instance, it depends on the actual number of workloads running in your clusters and if prometheus is configured to provide metrics.
Per cluster the daemon uses at most 16 connections by default (configurable, see below), of which int(max_connections / 3) = 5 are kept idle. The rest are closed once returned to the pool. MariaDB will close sessions idle longer than wait_timeout (MariaDB’s wait_timeout, 8h default) and the daemon reconnects transparently.
With 50 clusters and the default settings there can be up to 800 concurrent connections (50 clusters x 16 conns). The MariaDB’s default for simultaneous client connections is 151 (MariaDB’s max_connections), so you will either have to raise the limit in MariaDB or reduce the max number of connections for the daemon as described below.
To reduce the max number of connections in config.yml:
database:
options:
max_connections: N
Alternatively set via environment variable: export ICINGA_FOR_KUBERNETES_DATABASE_OPTIONS_MAX_CONNECTIONS=N