Hi,
Hoping someone is able to assist with a major issue we’ve started to have since upgrading to the latest Icinga - I can’t recall what version we jumped to but we had missed a lot of updates, so I am unsure if there has been a config or syntax change between versions.
Our original issue began as downtimes and acknowledgements not getting applied correctly from Icinga Web - after a director config change is pushed, a downtime created by Icinga Web would be removed, despite being still in effect.
Previously the behaviour we had and expect was that downtimes/acks applied via Icinga Web would remain, even post a config change via Director; however it appears that once a config change is pushed from Director, Icinga will reload remove all of the downtime/acks.
This led me to this issue on Github, which describes some of the behaviour I’ve seen. I’ve also commented on there with my findings.
This morning we found that our Icinga master had run low on disk space, and after running du
to find where our space went, it appears our scheduled downtime configuration is generating 8.8GB of config:
This is our config:
apply ScheduledDowntime "Patching Schedule" to Host {
author = "0xliam"
comment = "Monthly Patching Schedule"
fixed = true
assign where "Windows Server" in host.templates
ranges = {
"saturday -1 april" = "22:00-24:00"
"saturday -1 august" = "22:00-24:00"
"saturday -1 december" = "22:00-24:00"
"saturday -1 february" = "22:00-24:00"
"saturday -1 january" = "22:00-24:00"
"saturday -1 july" = "22:00-24:00"
"saturday -1 june" = "22:00-24:00"
"saturday -1 march" = "22:00-24:00"
"saturday -1 may" = "22:00-24:00"
"saturday -1 november" = "22:00-24:00"
"saturday -1 october" = "22:00-24:00"
"saturday -1 september" = "22:00-24:00"
"sunday -1 april" = "00:00-12:00"
"sunday -1 august" = "00:00-12:00"
"sunday -1 december" = "00:00-12:00"
"sunday -1 february" = "00:00-12:00"
"sunday -1 january" = "00:00-12:00"
"sunday -1 july" = "00:00-12:00"
"sunday -1 june" = "00:00-12:00"
"sunday -1 march" = "00:00-12:00"
"sunday -1 may" = "00:00-12:00"
"sunday -1 november" = "00:00-12:00"
"sunday -1 october" = "00:00-12:00"
"sunday -1 september" = "00:00-12:00"
}
}
apply ScheduledDowntime "Patching Schedule" to Service {
author = "0xliam"
comment = "Monthly Patching Schedule"
fixed = true
assign where "Windows Server" in host.templates
ranges = {
"saturday -1 april" = "22:00-24:00"
"saturday -1 august" = "22:00-24:00"
"saturday -1 december" = "22:00-24:00"
"saturday -1 february" = "22:00-24:00"
"saturday -1 january" = "22:00-24:00"
"saturday -1 july" = "22:00-24:00"
"saturday -1 june" = "22:00-24:00"
"saturday -1 march" = "22:00-24:00"
"saturday -1 may" = "22:00-24:00"
"saturday -1 november" = "22:00-24:00"
"saturday -1 october" = "22:00-24:00"
"saturday -1 september" = "22:00-24:00"
"sunday -1 april" = "00:00-12:00"
"sunday -1 august" = "00:00-12:00"
"sunday -1 december" = "00:00-12:00"
"sunday -1 february" = "00:00-12:00"
"sunday -1 january" = "00:00-12:00"
"sunday -1 july" = "00:00-12:00"
"sunday -1 june" = "00:00-12:00"
"sunday -1 march" = "00:00-12:00"
"sunday -1 may" = "00:00-12:00"
"sunday -1 november" = "00:00-12:00"
"sunday -1 october" = "00:00-12:00"
"sunday -1 september" = "00:00-12:00"
}
}
We have been using this exact config since we configured Icinga and it has not created any problems for us.
After digging into the config package, I grep’d for one of our Windows Server hosts and found that it is creating ~1900 config files per host.
[root@master ~]# cat /var/lib/icinga2/api/packages/_api/eda9115e-85c9-40b6-afca-c4e8f8419308/conf.d/downtimes/SERVER.ZONE\!Network\ Utilisation* | grep 'start_time'
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1685188800.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
start_time = 1680346800.000000
start_time = 1685188800.000000
start_time = 1682769600.000000
start_time = 1682769600.000000
As you can see, there are 326 files getting created for this single service, with a huge amount of them sharing the same start time, but there should only be at most 12 * 2 * per service (for the Saturday/Sunday downtime for every month of the year).
Is there anything obviously wrong with our config?