Icinga reloads config slowly? Try jemalloc

Al2Klimov · November 16, 2022, 11:35am

See

Icinga/icinga2/blob/master/doc/15-troubleshooting.md#icinga-startsrestartsreloads-very-slowly

# Icinga 2 Troubleshooting <a id="troubleshooting"></a>

## Required Information <a id="troubleshooting-information-required"></a>

Please ensure to provide any detail which may help reproduce and understand your issue.
Whether you ask on the [community channels](https://community.icinga.com) or you
create an issue at [GitHub](https://github.com/Icinga), make sure
that others can follow your explanations. If necessary, draw a picture and attach it for
better illustration. This is especially helpful if you are troubleshooting a distributed
setup.

We've come around many community questions and compiled this list. Add your own
findings and details please.

* Describe the expected behavior in your own words.
* Describe the actual behavior in one or two sentences.
* Ensure to provide general information such as:
	* How was Icinga 2 installed (and which repository in case) and which distribution are you using
	* `icinga2 --version`
	* `icinga2 feature list`

This file has been truncated. show original

for detailed instructions.

Please share your before/after time measurements here.

rafi01010 · November 23, 2022, 10:24am

Hi @Al2Klimov,
here are my results in version 2.13.2 and 2.13.6:
Unfortunately somewhat sobering. Bernd Erk has mentioned in this year’s osmc that with the new version 2.13.6 the config reload has become about 30% faster.
Which is unfortunately not the case with me.

time icinga2 daemon -C
[2022-11-22 11:26:27 +0100] information/cli: Icinga application loader (version: r2.13.2-1)
real    0m13.148s
user    0m55.685s
sys     0m9.926s

time env LD_PRELOAD=/lib/x86_64-linux-gnu/libjemalloc.so.2 icinga2 daemon -C
[2022-11-22 11:28:23 +0100] information/cli: Icinga application loader (version: r2.13.2-1)
real    0m11.556s
user    0m49.510s
sys     0m9.848s

time icinga2 daemon -C
[2022-11-23 11:15:07 +0100] information/cli: Icinga application loader (version: r2.13.6-1)
real    0m13.642s
user    0m53.440s
sys     0m10.781s

time env LD_PRELOAD=/lib/x86_64-linux-gnu/libjemalloc.so.2 icinga2 daemon -C
[2022-11-23 11:15:25 +0100] information/cli: Icinga application loader (version: r2.13.6-1)
real    0m12.006s
user    0m49.974s
sys     0m10.586s

My Setup:

62824 Notifications
1 IcingaApplication
26740 Dependencies
69 HostGroups
1819 Hosts
1 FileLogger
21092 Downtimes
1 Influxdb2Writer
194 Comments
1 IdoMysqlConnection
1 CheckerComponent
333 Zones
333 Endpoints
5 ApiUsers
1 ApiListener
1 NotificationComponent
311 CheckCommands
22 UserGroups
19 ServiceGroups
11 TimePeriods
55 Users
19935 ScheduledDowntimes
29385 Services
4 NotificationCommands

log1c · November 23, 2022, 2:32pm

Tried it on RHEL7.7 system with a quite outdated icinga2 (update incoming ):

Icinga application loader (version: 2.11.2-1)

 # icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql influxdb livestatus mainlog notification

40509 ScheduledDowntimes.
30012 Dependencies.
23182 Notifications.
17390 Services.
2576 Hosts.
1053 Downtimes.
341 CheckCommands.
229 Users.
129 ServiceGroups.
88 UserGroups.
81 Comments.
42 HostGroups.
10 ApiUsers.
9 NotificationCommands.
6 TimePeriods.
4 Zones.
3 Endpoints.
1 NotificationComponent.
1 LivestatusListener.
1 InfluxdbWriter.
1 IdoMysqlConnection.
1 IcingaApplication.
1 FileLogger.
1 ExternalCommandListener.
1 CheckerComponent.
1 ApiListener.

without:

real    0m22.024s
user    4m25.635s
sys     2m47.647s

with jemalloc

# time env LD_PRELOAD=/lib64/libjemalloc.so.1 icinga2 daemon -C

real    0m19.754s
user    4m13.474s
sys     3m15.749s

seems to not make any difference in this case.
will do another test once the system is updated

Al2Klimov · November 23, 2022, 2:54pm

@berk has also clearly mentioned the non-reliability of benchmarks as such. Just to be clear (as he spoke based on my info and I’ve even made the performance improvement in the first place): The more apply rules like assign where host.name == "h" && service.name == "s" you have, the faster 2.13.6 is compared to 2.13.5.

Also your load time isn’t even 1/2 minute, why you’re reading the Icinga starts/restarts/reloads very slowly section?

rafi01010 · November 23, 2022, 3:19pm

Yes I know, in his eyes its about 90% faster
I have about 200 of your described assign where conditions. Yes our systemd reload is not that slow. But the wohle relad process is slow, see: Remote Icinga instance is not connectet to master - for a few minutes after reload
So I thought it should be at least a little bit faster and I’ll write it in here.

Erik · March 7, 2023, 3:54pm

Instantiated 1 NotificationComponent.
Instantiated 1 LivestatusListener.
Instantiated 1 ExternalCommandListener.
Instantiated 1 CheckerComponent.
Instantiated 4 TimePeriods.
Instantiated 2 Users.
Instantiated 84430 Services.
Instantiated 75474 ScheduledDowntimes.
Instantiated 2 Zones.
Instantiated 4 NotificationCommands.
Instantiated 162678 Notifications.
Instantiated 2977 Hosts.
Instantiated 1 IcingaApplication.
Instantiated 18 HostGroups.
Instantiated 84430 Dependencies.
Instantiated 300 Comments.
Instantiated 10 EventCommands.
Instantiated 421799 Downtimes.
Instantiated 2 Endpoints.
Instantiated 1 FileLogger.
Instantiated 12 ApiUsers.
Instantiated 253 CheckCommands.
Instantiated 1 ApiListener.
Instantiated 1 IcingaDB.

Without:

real	4m7.549s
user	14m33.763s
sys	1m42.542s

With jemalloc:

real	2m15.408s
user	7m11.452s
sys	2m55.066s

Around 50% shaved off the startup time. Nice