Dear icinga friends,
I have an issue with my icinga2 director deployment system - while I run the deploy script provided in the Agent tab in icinga web on any of Debian nodes, the CPU load on the master reaches shortly up to 100% on all cores and it only goes back to normal after the host either has been added, or has failed to be added.
I am not sure what is causing the issue. Does anyone have a similar problem or a solution to this? Would be very thankful.
My setup:
Icinga Web 2 Version 2.9.3
Modules:
director |1.8.1|
doc |2.9.3|
grafana |1.4.2|
icinga-php-thirdparty |0.0.0|
incubator |0.6.0|
ipl |v0.5.0|
lazaroblanc |1.0.0|
migrate |2.9.3|
monitoring |2.9.3|
reactbundle |0.7.0|
test |2.9.3|
translation |2.9.3 |
Icinga 2 version r2.13.1-1
OS: Debian Buster 4.19.194-3
Webserver Apache
PHP version 7.3.29-1~deb10u1
IIRC, when a new api package is deployed (say, with Icinga director), Icinga2 spawns some extra processes that check the config (and then possibly take over the old processes, or otherwise hands off to the old ones and dies).
What you’re seeing is likely a side effect. When a new deployment is created, you might go from 2 or 3 Icinga2 processes to 5 or 6 (more or less, we average 7 on a config deployment per my alerts). If the deployment fails because of resource utilization, you may need to add additional CPU resources to handle the burst when validating/deploying configs.
Would you be so kind and please tell me what sort of CPU you are using? The CPU I have right now is Intel Core i7-8700. Icinga has around 3500 hosts deployed so far (and growing). Normal usage of CPU is around 12-20% on all 12 Threads, but the deploy just seem to kill it
Oh hrm, that’s pretty beefy. For comparison, we are virtualized, 8 cores on Xeon E5-2620.
We also have 32GB of RAM, which is the most used resource (on the master) since we heavily hit the API.
We don’t perform checks from the master though, all of those are performed from other agents. The master only accepts API results for ~7000 or so services.
