Icinga2 Comments Getting Recreated

Icinga 2
1

I am trying to troubleshoot a high load average on machines in one of our zones that is causing some checks not to execute in a timely manner.

When investigating /var/log/icinga2/icinga2.log on either of the zone endpoints, I see thousands of entries similar to the follow (largely for the same service on the same host):

[2020-03-16 15:29:48 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!c51e4753-9756-4647-816c-1ac110e1c4cd' of type 'Comment'.
[2020-03-16 15:29:48 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!a3c6f92d-bc31-4fc4-90c4-3209dc79c08e' of type 'Comment'.
[2020-03-16 15:29:49 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!d7ec0747-1fa6-4cd7-993e-5e7b03cd9c25' of type 'Comment'.
[2020-03-16 15:29:49 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!2a9fde5a-82a2-40db-b443-ad81a83fca8b' of type 'Comment'.
[2020-03-16 15:29:50 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!04ae6654-1782-433c-bb42-46b3f9bf58ec' of type 'Comment'.
[2020-03-16 15:29:50 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!6bdb4eba-2d8f-4d7c-af33-32e39de1f84e' of type 'Comment'.
[2020-03-16 15:29:50 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!3ecc531d-f3ed-426b-8e10-508930c2a9df' of type 'Comment'.
[2020-03-16 15:29:51 -0500] information/ConfigObjectUtility: Created and activated object 'ar01.gbsntn01!BGP ESP STATUS 172.X.X.X!b1898131-82c7-4be5-b506-8049c5f5a493' of type 'Comment'.

This ballooned the icinga2 ido db, as icinga_comments, icinga_commenthistory, and icinga_acknowledgements all had 50k+ rows.

I cleaned up each of those tables, restarted icinga2 on the satellite agents. I still see the concerning log entries and high load average. Also, icinga_comments is getting filled up with 50K+ entries after the restart again, but acknowledgements and comment history seem to be fine.

What could be causing the comments to be created thousands of times?

2

Found in another forum post that I can clean up comments:
find $(cat active-stage)/conf.d/comments/ -maxdepth 1 -name "*.conf" -print0 | xargs -0 rm

This has helped some, letting Icinga run its course with some jobs to clean out the queue

1 Like
3

Hey there,
thanks for taking the time to post your solution, did that solve the problem or just put a plaster on the wound?

I’ll bump it over to the devs to look at it, maybe it’s something we should keep in mind!

1 Like