Scheduled Downtimes applied by Director results in huge config package

0xliam · May 17, 2023, 12:40am

Hi,

Hoping someone is able to assist with a major issue we’ve started to have since upgrading to the latest Icinga - I can’t recall what version we jumped to but we had missed a lot of updates, so I am unsure if there has been a config or syntax change between versions.

Our original issue began as downtimes and acknowledgements not getting applied correctly from Icinga Web - after a director config change is pushed, a downtime created by Icinga Web would be removed, despite being still in effect.

Previously the behaviour we had and expect was that downtimes/acks applied via Icinga Web would remain, even post a config change via Director; however it appears that once a config change is pushed from Director, Icinga will reload remove all of the downtime/acks.

This led me to this issue on Github, which describes some of the behaviour I’ve seen. I’ve also commented on there with my findings.

github.com/Icinga/icinga2

Downtimes not reapplied after a reload/restart of Icinga2

opened 02:19PM - 10 Aug 21 UTC

steaksauce-

bug area/runtime

## Describe the bug I am on 2.13.0-1 and notifications are sent for all hosts…/services during scheduled downtime. This problem existed in 2.11.3-1, which prompted our team to do an emergency upgrade to 2.13.0-1. Downtime is applied via Icingaweb2 (not an apply rule) for maintenance windows by our Network Engineers. Downtime is fixed, not “flexible”, and downtime was sent. Here is an example: ![image](https://user-images.githubusercontent.com/8814619/128881886-b3aa9481-db2d-4431-abb5-0c1ad5ed3ef0.png) ![image](https://user-images.githubusercontent.com/8814619/128881908-499789e3-9f75-4afa-b711-b5d0a5a3fb62.png) The examples that I have are check results processed via the API, not via a plugin. In the example above, the notification is sent out after the 2nd run of the API call. (Script runs on a schedule every 20 minutes, takes a few more to actually do a full run) ## To Reproduce Provide a link to a live example, or an unambiguous set of steps to reproduce this bug. Include configuration, logs, etc. to reproduce, if relevant. 1. Schedule downtime for a service that has passive checks enabled and Active checks disabled (this one has check_dummy assigned to it) 2. Run an API `process_checkresult` API call once to the service during the scheduled downtime. No alert sent. 3. Run the `process_checkresult` API call again to the service during the scheduled downtime. Alert sent. ## Expected behavior During a scheduled downtime, I would expect that notifications are never sent, regardless of how many times the service is checked and in a non-OK state. Also during downtime, I would expect that notifications are never sent, even if the service recovers and then goes back into a non-OK state. ## Your Environment Include as many relevant details about the environment you experienced the problem in Version used (icinga2 --version) `2.13.0-1` Operating System and version `CentOS 7 3.10.0-1127.8.2.el7.x86_64` Enabled features (icinga2 feature list) `api checker command ido-mysql influxdb mainlog notification statusdata` Icinga Web 2 version and modules (System - About) `2.7.3` Config validation (icinga2 daemon -C) REDACTED, but config loads fine. If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes (SOME NODES May appear to be in 2 zones, or otherwise be in there twice, but they have different domain names that I have redacted) ``` Object 'icinga02' of type 'Endpoint': % declared in '/etc/icinga2/zones.conf', lines 6:1-6:41 * __name = "icinga02" * host = "" * log_duration = 86400 * name = "icinga02" * package = "_etc" * port = "5665" * source_location * first_column = 1 * first_line = 6 * last_column = 41 * last_line = 6 * path = "/etc/icinga2/zones.conf" * templates = [ "icinga02" ] % = modified in '/etc/icinga2/zones.conf', lines 6:1-6:41 * type = "Endpoint" * zone = "" Object 'ica02m02n' of type 'Endpoint': % declared in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 7:1-7:42 * __name = "ica02m02n" * host = "ica02m02n" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 8:5-8:37 * log_duration = 86400 % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 10:5-10:21 * name = "ica02m02n" * package = "director" * port = "5665" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 9:5-9:17 * source_location * first_column = 1 * first_line = 7 * last_column = 42 * last_line = 7 * path = "/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf" * templates = [ "ica02m02n" ] % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 7:1-7:42 * type = "Endpoint" * zone = "master" Object 'ica01m02n' of type 'Endpoint': % declared in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 1:0-1:41 * __name = "ica01m02n" * host = "ica01m02n" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 2:5-2:37 * log_duration = 86400 % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 4:5-4:21 * name = "ica01m02n" * package = "director" * port = "5665" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 3:5-3:17 * source_location * first_column = 0 * first_line = 1 * last_column = 41 * last_line = 1 * path = "/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf" * templates = [ "ica01m02n" ] % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 1:0-1:41 * type = "Endpoint" * zone = "master" Object 'ica03m02n' of type 'Endpoint': % declared in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 25:1-25:42 * __name = "ica03m02n" * host = "ica03m02n" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 26:5-26:37 * log_duration = 86400 % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 28:5-28:21 * name = "ica03m02n.nsvltn" * package = "director" * port = "5665" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 27:5-27:17 * source_location * first_column = 1 * first_line = 25 * last_column = 42 * last_line = 25 * path = "/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf" * templates = [ "ica03m02n" ] % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 25:1-25:42 * type = "Endpoint" * zone = "master" Object 'ica04m02n' of type 'Endpoint': % declared in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 31:1-31:42 * __name = "ica04m02n" * host = "ica04m02n" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 32:5-32:37 * log_duration = 86400 % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 34:5-34:21 * name = "ica04m02n" * package = "director" * port = "5665" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 33:5-33:17 * source_location * first_column = 1 * first_line = 31 * last_column = 42 * last_line = 31 * path = "/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf" * templates = [ "ica04m02n" ] % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 31:1-31:42 * type = "Endpoint" * zone = "master" Object 'ica01m02n' of type 'Endpoint': % declared in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 13:1-13:42 * __name = "ica01m02n" * host = "ica01m02n" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 14:5-14:37 * log_duration = 86400 % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 16:5-16:21 * name = "ica01m02n" * package = "director" * port = "5665" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 15:5-15:17 * source_location * first_column = 1 * first_line = 13 * last_column = 42 * last_line = 13 * path = "/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf" * templates = [ "ica01m02n" ] % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 13:1-13:42 * type = "Endpoint" * zone = "master" Object 'ica02m02n' of type 'Endpoint': % declared in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 19:1-19:42 * __name = "ica02m02n" * host = "ica02m02n" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 20:5-20:37 * log_duration = 86400 % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 22:5-22:21 * name = "ica02m02n" * package = "director" * port = "5665" % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 21:5-21:17 * source_location * first_column = 1 * first_line = 19 * last_column = 42 * last_line = 19 * path = "/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf" * templates = [ "ica02m02n" ] % = modified in '/var/lib/icinga2/api/packages/director/46b75cce-5f66-49dc-9ea0-3f5dfd5daaa7/zones.d/master/endpoints.conf', lines 19:1-19:42 * type = "Endpoint" * zone = "master" ``` ## Additional context Again, the only examples I have are for services passive-checks via the API. I currently do not know if this affects services with active-checks enabled. Our customer TAC team only recently started using scheduled-downtimes, and that is when we noticed an issue and upgraded from 2.11.x to 2.13.0. I did open a community forum issue about this, but nothing has come from that yet: https://community.icinga.com/t/notifications-are-still-sent-during-scheduled-downtime/7962?u=steaksauce Edit: This is a bug for Downtimes not reapplying after a restart/reload of Icinga 2. Steps to reproduce (as noted by @Yonas-net below): > > is this director specific, or just a reload specific? > > It is not director specific. I was also able to reproduce it manually. > > 1. Schedule a Downtime > 2. `pkill -HUP icinga2` > 3. Wait for a couple of seconds > 4. `Service/Host` is no longer in Downtime > Sorry for being so delayed, but have now taken the time to set up the director and have indeed reproduced the issue 😞. > > **To Reproduce:** > First, schedule a downtime for the `swap` service. > > <img alt="Bildschirmfoto 2021-08-26 um 12 04 18" width="844" src="https://user-images.githubusercontent.com/57616252/130944027-6a3abb29-c55f-49da-a3e8-41d79358b7cc.png"> > > Will also be displayed immediately afterwards in Icinga Web 2. > > <img alt="Bildschirmfoto 2021-08-26 um 12 06 08" width="430" src="https://user-images.githubusercontent.com/57616252/130944458-da1120f0-5cfd-4661-a115-acb51c50d4b1.png"> > > Now deploy any changes with the Director and wait for a couple of seconds till Icinga2 is reloading. > > <img alt="Bildschirmfoto 2021-08-26 um 12 10 27" width="633" src="https://user-images.githubusercontent.com/57616252/130944860-319c5728-5d58-4da2-8072-3259c3f0ed9d.png"> > > <img alt="Bildschirmfoto 2021-08-26 um 12 11 10" width="772" src="https://user-images.githubusercontent.com/57616252/130944949-bbf3b831-f562-441f-9737-c17d2f8182e3.png"> > > Service `swap` is now no longer in Downtime. > > <img alt="Bildschirmfoto 2021-08-26 um 12 14 37" width="428" src="https://user-images.githubusercontent.com/57616252/130945386-93c2d2f3-c01d-4c5c-917b-8cfcd4741774.png">

This morning we found that our Icinga master had run low on disk space, and after running du to find where our space went, it appears our scheduled downtime configuration is generating 8.8GB of config:

This is our config:

apply ScheduledDowntime "Patching Schedule" to Host {
    author = "0xliam"
    comment = "Monthly Patching Schedule"
    fixed = true
    assign where "Windows Server" in host.templates
    ranges = {
        "saturday -1 april"	= "22:00-24:00"
        "saturday -1 august"	= "22:00-24:00"
        "saturday -1 december"	= "22:00-24:00"
        "saturday -1 february"	= "22:00-24:00"
        "saturday -1 january"	= "22:00-24:00"
        "saturday -1 july"	= "22:00-24:00"
        "saturday -1 june"	= "22:00-24:00"
        "saturday -1 march"	= "22:00-24:00"
        "saturday -1 may"	= "22:00-24:00"
        "saturday -1 november"	= "22:00-24:00"
        "saturday -1 october"	= "22:00-24:00"
        "saturday -1 september"	= "22:00-24:00"
        "sunday -1 april"	= "00:00-12:00"
        "sunday -1 august"	= "00:00-12:00"
        "sunday -1 december"	= "00:00-12:00"
        "sunday -1 february"	= "00:00-12:00"
        "sunday -1 january"	= "00:00-12:00"
        "sunday -1 july"	= "00:00-12:00"
        "sunday -1 june"	= "00:00-12:00"
        "sunday -1 march"	= "00:00-12:00"
        "sunday -1 may"	= "00:00-12:00"
        "sunday -1 november"	= "00:00-12:00"
        "sunday -1 october"	= "00:00-12:00"
        "sunday -1 september"	= "00:00-12:00"
    }
}

apply ScheduledDowntime "Patching Schedule" to Service {
    author = "0xliam"
    comment = "Monthly Patching Schedule"
    fixed = true
    assign where "Windows Server" in host.templates
    ranges = {
        "saturday -1 april"	= "22:00-24:00"
        "saturday -1 august"	= "22:00-24:00"
        "saturday -1 december"	= "22:00-24:00"
        "saturday -1 february"	= "22:00-24:00"
        "saturday -1 january"	= "22:00-24:00"
        "saturday -1 july"	= "22:00-24:00"
        "saturday -1 june"	= "22:00-24:00"
        "saturday -1 march"	= "22:00-24:00"
        "saturday -1 may"	= "22:00-24:00"
        "saturday -1 november"	= "22:00-24:00"
        "saturday -1 october"	= "22:00-24:00"
        "saturday -1 september"	= "22:00-24:00"
        "sunday -1 april"	= "00:00-12:00"
        "sunday -1 august"	= "00:00-12:00"
        "sunday -1 december"	= "00:00-12:00"
        "sunday -1 february"	= "00:00-12:00"
        "sunday -1 january"	= "00:00-12:00"
        "sunday -1 july"	= "00:00-12:00"
        "sunday -1 june"	= "00:00-12:00"
        "sunday -1 march"	= "00:00-12:00"
        "sunday -1 may"	= "00:00-12:00"
        "sunday -1 november"	= "00:00-12:00"
        "sunday -1 october"	= "00:00-12:00"
        "sunday -1 september"	= "00:00-12:00"
    }
}

We have been using this exact config since we configured Icinga and it has not created any problems for us.

After digging into the config package, I grep’d for one of our Windows Server hosts and found that it is creating ~1900 config files per host.

[root@master ~]# cat /var/lib/icinga2/api/packages/_api/eda9115e-85c9-40b6-afca-c4e8f8419308/conf.d/downtimes/SERVER.ZONE\!Network\ Utilisation* | grep 'start_time'
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1685188800.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000
	start_time = 1680346800.000000
	start_time = 1685188800.000000
	start_time = 1682769600.000000
	start_time = 1682769600.000000

As you can see, there are 326 files getting created for this single service, with a huge amount of them sharing the same start time, but there should only be at most 12 * 2 * per service (for the Saturday/Sunday downtime for every month of the year).

Is there anything obviously wrong with our config?

Al2Klimov · May 17, 2023, 8:59am

Wrong? Definitely. Obvious? Unlikely. Maybe a broken active/stage/include.

Can you dispose that whole 8.8G directory and reload?

0xliam · May 18, 2023, 12:53am

I deleted the directores under ./packages/_api/ and then re-applied the config from Director and it generated the same data!

Al2Klimov · May 22, 2023, 9:13am

First of all, if I read correctly, you seem to schedule “a downtime” across Saturday/Sunday (22:00-12:00). Have you tried to deduplicate the ranges per month like this?

- "saturday -1 april"	= "22:00-24:00"
- "sunday -1 april"	= "00:00-12:00"
+ "saturday -1 april"	= "22:00-36:00"

0xliam · May 25, 2023, 5:16am

I will try this now - thank you I was never happy with that config but it seemed to work when we were running previous versions… perhaps that is the cause… will report back.