When deploying any configuration change to ICINGA2 stable version 2.10.5, sometimes the dashboard is blank. I have noticed that this happens when changes are being deployed which seems reasonable. The “Activity” option under Director show no pending changes but if I reapply the changes the dashboard comes up with the hosts & services but it triggers the notification for all of the hosts and services which were down earlier. I have notification set to “0”. Could someone please explain the behavior.
Hi,
a config deployment triggers a daemon reload, once the configuration is validated and ok. During this reload period, the configuration is dumped to the IDO database resource. This resource is used in Icinga Web for showing all the objects (config, state, history).
With applied permissions on e.g. hostgroups or custom variables, these attributes might be updated during the reload with an actual delete then insert again. Depending on the load and OS resources, this may take a while.
A more technical insight - the config dump cannot happen in parallel, as the schema has many sequential dependencies. That is why it takes a while to process all queries and update the SQL database. There’s work underway with a new backend called IcingaDB, but that’s not yet ready to test.
To get a better idea about possible improvements here, please share the output of icinga2 daemon -C and the details of your server (cpu, ram, etc.) as well as where the IDO database is located and whether you’ve tuned MySQL already.
Cheers,
Michael
Hello Michael,
Many Thanks for the detailed explanation. Please see the details below -
[root@xxxxxx conf.d]# icinga2 daemon -C
[2019-09-27 02:53:19 -0500] information/cli: Icinga application loader (version: 2.11.0-1)
[2019-09-27 02:53:19 -0500] information/cli: Loading configuration file(s).
[2019-09-27 02:53:19 -0500] information/ConfigItem: Committing config item(s).
[2019-09-27 02:53:19 -0500] information/ApiListener: My API identity: CHI-XXXXX.atkearney.com
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 2 FileLoggers.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 3 NotificationCommands.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 922 Notifications.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 11 HostGroups.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 688 Hosts.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 134 Zones.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 132 Endpoints.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 ApiUser.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 ApiListener.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 235 CheckCommands.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 1 TimePeriod.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 3 Users.
[2019-09-27 02:53:20 -0500] information/ConfigItem: Instantiated 234 Services.
[2019-09-27 02:53:20 -0500] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2019-09-27 02:53:20 -0500] information/cli: Finished validating the configuration file(s).
CPU - 1 vCPU
Memory - 2 GB.
I can see that the CPU utilization was close to 100% during the issue & I will increase the resources. Any insight about resource allocation on ICINGA server?
I have not touched MySQL yet. Any suggestions on this please?
Thanks in advance.
1 CPU is quite low, grant it 2 or 4 to stay safe with more additional monitoring data. RAM should also be extended, especially if you run MySQL on the same host … and maybe Graphite for metrics later on.
Here’s some hints on monitoring the Icinga host and getting more insights on tuning: https://icinga.com/docs/icinga2/latest/doc/08-advanced-topics/#monitoring-icinga-2
Cheers,
Michael