NAME=“Red Hat Enterprise Linux Server”
VERSION=“7.9 (Maipo)”
ID=“rhel”
ID_LIKE=“fedora”
VARIANT=“Server”
VARIANT_ID=“server”
VERSION_ID=“7.9”
PRETTY_NAME=JBoss
That would be great, thank you 
Here you are: https://transfer.netways.de/index.php/s/moCHdoW49dFGaRm
You can yum install *.rpm and, once you’re done, yum history undo last (if you didn’t perform additional yum operations meanwhile).
Thank you so much @Al2Klimov!
I was not able to test this earlier, but i got some interesting results now.
Since i cannot use the provided packages in our production environment i setup a dedicated test VM running icinga2 in the same version as our production environment.
The only difference was, that my test setup is a single master without satellites, while the prod setup is a dual master with 4 satellites.
I created and used the following quick python skript, to spawn 1000 threads, each running a while loop which continuously makes post requests to send passive check results against localhost (endpoint process-check-results):
#!/usr/bin/env python3
import requests
import threading
from datetime import datetime
import random
def start_request_loop(service_number):
while True:
try:
if random.randint(0,1) == 1:
state = 1
else:
state = 0
headers = {'Accept': 'application/json'}
data = {"type": "Service", "filter": f"host.name==\"Host 10\" && service.name==\"Service {service_number}\"", "exit_status": state, "plugin_output": f"TEST OUTPUT -> {datetime.now()}"}
response = requests.post(f'https://127.0.0.1:5665/v1/actions/process-check-result', headers=headers, verify=False, json=data, auth=('usr', 'pwd'))
print(f"Service {service_number} -> {response.status_code}")
except Exception as e:
print(e)
continue
counter = 1
for i in range(1000):
if counter > 10:
counter = 1
print(f"STARTED LOOP #{i}")
t = threading.Thread(target=start_request_loop, args=(counter,))
t.daemon = False
t.start()
counter += 1
As you can see, to add some variety and make icinga do some work, the exit status is random and the target services are 10 different ones.
Here are my results
BEFORE using the custom icinga packages provided →
Idle VM, no api requests running:
1000 Threads of continuous requests are running:
Threads are stopped and i have waited for 5 minutes:
→ As you can see, even after waiting 5 minutes, the used memory is still increased by 12%, which is quite a lot.
AFTER using the custom icinga packages provided →
Note: I wated roughly the same amount of time until i started the tests, as you can see by the uptime.
Idle VM, no api requests running:
1000 Threads of continuous requests are running:
→ This actually used more memory than before
Threads are stopped and i have waited for 5 minutes:
→ As you can see, this time it was around a 9% increase in ram usage after the test was finished.
Of course this is only a small scale test, but i would say there is definetely a improvement in the custom versions.
But still, there is still room for more optimisations.
What is your impression of the results @Al2Klimov?
The api requests still worked without any issues, do you think a release of the current optimisations is possible already?
Especially as those packages “still worked without any issues”, I’d really like to see them in your prod env actually fixing the memory leak first. Or, at least, in a closer replication of your prod which includes the memory leak.
I was trying to undo the operation on my test setup, as you told me, with:
yum history undo last
but this unfortunately led to errors:
It was not able to rollback most of the icinga packages.
Therefore i am not able to test this in our prod environment.
Do you have a dual master test setup at netways, where you could test the packages?
Where did you get 2.14.0-1 from so that it’s not available now?
Hello @Al2Klimov,
sorry for the late response, i was on vacation and sick leave.
I rolled back my test vm today and tested the install und rollback again.
This time it worked without any issues, i am not sure what i did wrong last time.
We are discussing and planning internally, if we can test the packages in our prod environment.
Apart from this, do you have a dual master test setup at netways, where you could test the packages as well?
I am asking this, because we are still unsure if the prod environment can be used for testing this and i would think that you would have a dual master setup for testing purposes?
Testing should be easy, since i provided a working script here:
Understandable. You can also wait for v2.14.1 which should Include:
That is great news
Thank you very much, we will wait for v2.14.1 then and check back again after updating.
This already looks very promising:
Great work @Al2Klimov 
It’s out now, go for it!
@davekempe Btw. you can also try the new version. If that doesn’t help, please create a core dump just before OOM:
https://icinga.com/docs/icinga-2/latest/doc/21-development/#gdb-core-dump