Icinga working slow on vmware esxi

For some time it works slow, it has dedicated 8 CPUs on Vmware esxi (which is maximum), 64 GB/ram. It’s true that it runs 570 hosts with around 8500 services. (active checks). What is your opinion possibilities to speed it up? For example it takes around 10-20 seconds sometimes to load service or host. Also noticed that it works even slower when snapshot is present on Virtual machine. Recently I did database clean for entries older than 24 months, so it seems to work faster (with history atleast).

One thing which is although seperated topic is that, acknowledge gives error and Process check result works with great delay.

Topic thread is here:

Give as much information as you can, e.g.
Icinga Web 2 version - 2.11.3
Used modules and their versions (System - About)
Web browser used - Chrome
Icinga 2 version used (icinga2 --version) - r2.14.3-1
PHP version used (php --version) - PHP 8.0.30
Server operating system and version - AlmaLinux release 8.10 (Cerulean Leopard)

Hi @samo.goljat,
Just the number of Hosts and Services should not cause such a load as to slow down this machine (also dependens on the processors there and so).

You probably should diagnose the problem on the machine with tools like htop and so on and look for the cause of the problem. Which processes use all the ressources?

Looking at htop it isn’t burning more then 10-20% CPU and 20% memory. Top processes are icinga2, check_cisco_nex (2%) other processes aren’t even relevant. It turned out that snapshot on vmware esxic did slow down alot.

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND

1973313 icinga 20 0 4626724 1.4g 20404 S 8.6 2.2 120:34.48 icinga2
202596 root 25 5 338176 38400 11908 S 6.3 0.1 0:00.19 check_esxi_hard
1 root 20 0 249452 12224 8172 S 1.3 0.0 40:01.79 systemd
998 dbus 20 0 86536 11780 5312 S 1.3 0.0 47:35.11 dbus-daemon
1973330 icinga 20 0 438644 7032 4672 S 1.0 0.0 14:37.54 icinga2

So reloading icinga takes a while?

One way I managed to get faster reloads was to refactor icinga2 config.
Instead of creating services associated to hostgroups (and so on to hosts).
I created global services and assigned them to hosts via variables etc.

So might be a worthwhile effort to refactor if your configs have increased a lot.