Scheduled Downtimes applied by Director results in huge config package

Hi,

Hoping someone is able to assist with a major issue we’ve started to have since upgrading to the latest Icinga - I can’t recall what version we jumped to but we had missed a lot of updates, so I am unsure if there has been a config or syntax change between versions.

Our original issue began as downtimes and acknowledgements not getting applied correctly from Icinga Web - after a director config change is pushed, a downtime created by Icinga Web would be removed, despite being still in effect.

Previously the behaviour we had and expect was that downtimes/acks applied via Icinga Web would remain, even post a config change via Director; however it appears that once a config change is pushed from Director, Icinga will reload remove all of the downtime/acks.

This led me to this issue on Github, which describes some of the behaviour I’ve seen. I’ve also commented on there with my findings.

This morning we found that our Icinga master had run low on disk space, and after running du to find where our space went, it appears our scheduled downtime configuration is generating 8.8GB of config:

This is our config:

apply ScheduledDowntime "Patching Schedule" to Host {
    author = "0xliam"
    comment = "Monthly Patching Schedule"
    fixed = true
    assign where "Windows Server" in host.templates
    ranges = {
        "saturday -1 april"	= "22:00-24:00"
        "saturday -1 august"	= "22:00-24:00"
        "saturday -1 december"	= "22:00-24:00"
        "saturday -1 february"	= "22:00-24:00"
        "saturday -1 january"	= "22:00-24:00"
        "saturday -1 july"	= "22:00-24:00"
        "saturday -1 june"	= "22:00-24:00"
        "saturday -1 march"	= "22:00-24:00"
        "saturday -1 may"	= "22:00-24:00"
        "saturday -1 november"	= "22:00-24:00"
        "saturday -1 october"	= "22:00-24:00"
        "saturday -1 september"	= "22:00-24:00"
        "sunday -1 april"	= "00:00-12:00"
        "sunday -1 august"	= "00:00-12:00"
        "sunday -1 december"	= "00:00-12:00"
        "sunday -1 february"	= "00:00-12:00"
        "sunday -1 january"	= "00:00-12:00"
        "sunday -1 july"	= "00:00-12:00"
        "sunday -1 june"	= "00:00-12:00"
        "sunday -1 march"	= "00:00-12:00"
        "sunday -1 may"	= "00:00-12:00"
        "sunday -1 november"	= "00:00-12:00"
        "sunday -1 october"	= "00:00-12:00"
        "sunday -1 september"	= "00:00-12:00"
    }
}

apply ScheduledDowntime "Patching Schedule" to Service {
    author = "0xliam"
    comment = "Monthly Patching Schedule"
    fixed = true
    assign where "Windows Server" in host.templates
    ranges = {
        "saturday -1 april"	= "22:00-24:00"
        "saturday -1 august"	= "22:00-24:00"
        "saturday -1 december"	= "22:00-24:00"
        "saturday -1 february"	= "22:00-24:00"
        "saturday -1 january"	= "22:00-24:00"
        "saturday -1 july"	= "22:00-24:00"
        "saturday -1 june"	= "22:00-24:00"
        "saturday -1 march"	= "22:00-24:00"
        "saturday -1 may"	= "22:00-24:00"
        "saturday -1 november"	= "22:00-24:00"
        "saturday -1 october"	= "22:00-24:00"
        "saturday -1 september"	= "22:00-24:00"
        "sunday -1 april"	= "00:00-12:00"
        "sunday -1 august"	= "00:00-12:00"
        "sunday -1 december"	= "00:00-12:00"
        "sunday -1 february"	= "00:00-12:00"
        "sunday -1 january"	= "00:00-12:00"
        "sunday -1 july"	= "00:00-12:00"
        "sunday -1 june"	= "00:00-12:00"
        "sunday -1 march"	= "00:00-12:00"
        "sunday -1 may"	= "00:00-12:00"
        "sunday -1 november"	= "00:00-12:00"
        "sunday -1 october"	= "00:00-12:00"
        "sunday -1 september"	= "00:00-12:00"
    }
}

We have been using this exact config since we configured Icinga and it has not created any problems for us.

After digging into the config package, I grep’d for one of our Windows Server hosts and found that it is creating ~1900 config files per host.

[root@master ~]# cat /var/lib/icinga2/api/packages/_api/eda9115e-85c9-40b6-afca-c4e8f8419308/conf.d/downtimes/SERVER.ZONE\!Network\ Utilisation* | grep 'start_time'
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000

As you can see, there are 326 files getting created for this single service, with a huge amount of them sharing the same start time, but there should only be at most 12 * 2 * per service (for the Saturday/Sunday downtime for every month of the year).

Is there anything obviously wrong with our config?

Wrong? Definitely. Obvious? Unlikely. Maybe a broken active/stage/include.

Can you dispose that whole 8.8G directory and reload?

I deleted the directores under ./packages/_api/ and then re-applied the config from Director and it generated the same data! :frowning:

First of all, if I read correctly, you seem to schedule “a downtime” across Saturday/Sunday (22:00-12:00). Have you tried to deduplicate the ranges per month like this?

- "saturday -1 april"	= "22:00-24:00"
- "sunday -1 april"	= "00:00-12:00"
+ "saturday -1 april"	= "22:00-36:00"
1 Like

I will try this now - thank you :slight_smile: I was never happy with that config but it seemed to work when we were running previous versions… perhaps that is the cause… will report back.