Recuring Downtime wrong

Basti · March 27, 2019, 2:55pm

Hey,

i have a problem with recurring downtimes in my Icinga. i have a host that is always shut down on the last Monday of the month for maintenance.
I have planned a downtime for that in downtime.conf

apply ScheduledDowntime "reboot-system-vsmo" to Host {
        author ="icingaadmin"
        comment = "Monatliche Wartung VSM-CP2"
        ranges = {
                 "monday -1" = "05:45-07:15"
                 }
        assign where host.name == "vsm_sapcp2"
}

For a while this worked quite well. but since last month, this does not work properly.
In IcingaWeb2 the downtime is planed for 01.04.2019 but the last monday in april is the 29th.

In the documentation https://icinga.com/docs/icinga2/latest/doc/08-advanced-topics/#time-periods is the right syntax.

"monday -1 may" = "00:00-24:00" //memorial day (last monday in may)

do not understand what’s going wrong here. even if I write the month in the range changes nothing in the result.

maybe someone can help. Many Thanks

Basti

My System:
Ubuntu 16.04.6 LTS
Icinga2 r2.10.3-1
Icingaweb2: 2.6.2

log1c · March 28, 2019, 8:33am

I tried the same for notifications, but to be honest I forgot about the problem: https://monitoring-portal.org/t/time-period-last-wednesday-of-month/4808

Just having a look through some of the systems I think this works now. At least one system had a warning status on monday but waited until wednesday to send the notification. Time period is set to "wednesday -1" = "00:00-24:00"

Only thing I changed between the posting above and the working notification was updating to 2.10.4. Not sure if any of the changes in 2.10.4 have made the difference or if it was in the versions before (the system was still on 2.9): https://github.com/Icinga/icinga2/releases/tag/v2.10.4

dnsmichi · March 28, 2019, 9:34am

There were a couple of fixes applied for SDs in 2.10.3 (customer request), highly likely this also affects the scheduling of future downtimes. Honestly I lost track on those changes.

Basti · March 28, 2019, 10:16am

have just checked for updates and seen there are new ones. I will import these in the following days and check if the error still exists.

Thank you

Basti · March 29, 2019, 8:49am

I update my Icinga2 to version r2.10.4-1. after update i deleted the downtime from DB table “icinga_scheduleddowntime”. then the files in “/var/lib/icinga2/api/packages/_api/xxxx-1513244872-1/conf.d/downtimes”

after restart the downtimes were created but the date is wrong. the downtime planed for monday 1 april.

my config:

"monday -1 april" = "05:45-07:15"

any ideas?

dnsmichi · March 29, 2019, 8:50am

What does the debug log say here?

Basti · March 29, 2019, 9:41am

found the following:

[2019-03-29 10:29:34 +0100] notice/Downtime: Added downtime 'vsm_sapcp2_db!ping4!70f54537-4cf6-4846-aad5-5a18b9a710b8' between '2019-04-01 05:45:00' and '2019-04-01 07:15:00'.
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Creating new Downtime for ScheduledDowntime "vsm_sapcp2!check_sap_availability_by_saprouter!reboot-system-vsmo"
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Finding running scheduled downtime segment for time 1553851774 (minEnd -)
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 april: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 august: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 december: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 february: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 january: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 july: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 june: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 march: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 may: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 november: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 october: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating (running?) segment: monday -1 september: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Finding next scheduled downtime segment for time 1553851774
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 april: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Apr 29 05:45:00 2019 -> Mon Apr 29 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: (best match yet)
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 august: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Aug 26 05:45:00 2019 -> Mon Aug 26 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 december: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Dec 30 05:45:00 2019 -> Mon Dec 30 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 february: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 january: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 july: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Jul 29 05:45:00 2019 -> Mon Jul 29 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 june: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Jul  1 05:45:00 2019 -> Mon Jul  1 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 march: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Apr  1 05:45:00 2019 -> Mon Apr  1 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: (best match yet)
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 may: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon May 27 05:45:00 2019 -> Mon May 27 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 november: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Nov 25 05:45:00 2019 -> Mon Nov 25 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 october: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Oct 28 05:45:00 2019 -> Mon Oct 28 07:15:00 2019
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Evaluating segment: monday -1 september: 05:45-07:15
[2019-03-29 10:29:34 +0100] debug/ScheduledDowntime: Considering segment: Mon Sep 30 05:45:00 2019 -> Mon Sep 30 07:15:00 2019
[2019-03-29 10:29:34 +0100] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/aspbazinga04-1513244872-1/conf.d/downtimes/vsm_sapcp2!check_sap_availability_by_saprouter!d70e284f-6d54-49d1-a5cb-6295325547d9.conf


[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'ping4',  check_source = 'aspbazinga.takeasp.de',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '0',  endpoint_object_id = 493,  event_handler_enabled = '1',  execution_time = '4.033288',  flap_detection_enabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable = '1',  last_check = FROM_UNIXTIME(1553851774),  last_hard_state = '0',  last_hard_state_change = FROM_UNIXTIME(1553626658),  last_notification = FROM_UNIXTIME(1552473522),  last_state_change = FROM_UNIXTIME(1553626658),  last_time_critical = FROM_UNIXTIME(1553626590),  last_time_ok = FROM_UNIXTIME(1553851774),  last_time_warning = FROM_UNIXTIME(1546284567),  latency = '0.000352',  long_output = '',  max_check_attempts = '5',  next_check = FROM_UNIXTIME(1553852005),  next_notification = FROM_UNIXTIME(1553853512),  normal_check_interval = '4',  notifications_enabled = '1',  original_attributes = 'null',  output = 'PING OK - Packet loss = 0%, RTA = 9.81 ms',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = 'rta=9.806000ms;500.000000;1000.000000;0.000000 pl=0%;70;80;0',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '1',  scheduled_downtime_depth = '0',  service_object_id = 1990,  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1553851774) WHERE service_object_id = 1990
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_scheduleddowntime SET actual_start_time = FROM_UNIXTIME(1554090300),  actual_start_time_usec = '0',  author_name = 'icingaadmin',  comment_data = 'Monatliche Wartung VSM-CP2',  downtime_type = '1',  duration = '0',  endpoint_object_id = 493,  entry_time = FROM_UNIXTIME(1553851774),  instance_id = 1,  internal_downtime_id = '34',  is_fixed = '1',  is_in_effect = '0',  name = 'vsm_sapcp2!ping4!b485d3ce-f610-406a-be05-0c8121d1f681',  object_id = 2239,  scheduled_end_time = FROM_UNIXTIME(1554095700),  scheduled_start_time = FROM_UNIXTIME(1554090300),  session_token = 1553851712,  was_started = '0' WHERE entry_time = FROM_UNIXTIME(1553851774) AND name = 'vsm_sapcp2!ping4!b485d3ce-f610-406a-be05-0c8121d1f681' AND object_id = 2239
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET scheduled_downtime_depth = '0' WHERE instance_id = 1 AND service_object_id = 2239
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_downtimehistory (actual_start_time, actual_start_time_usec, author_name, comment_data, downtime_type, duration, endpoint_object_id, entry_time, instance_id, internal_downtime_id, is_fixed, is_in_effect, name, object_id, scheduled_end_time, scheduled_start_time, was_started) VALUES (FROM_UNIXTIME(1554090300), '0', 'icingaadmin', 'Monatliche Wartung VSM-CP2', '1', '0', 493, FROM_UNIXTIME(1553851774), 1, '34', '1', '0', 'vsm_sapcp2!ping4!b485d3ce-f610-406a-be05-0c8121d1f681', 2239, FROM_UNIXTIME(1554095700), FROM_UNIXTIME(1554090300), '0')
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_scheduleddowntime SET actual_start_time = FROM_UNIXTIME(1554090300),  actual_start_time_usec = '0',  author_name = 'icingaadmin',  comment_data = 'Monatliche Wartung VSM-CP2',  downtime_type = '1',  duration = '0',  endpoint_object_id = 493,  entry_time = FROM_UNIXTIME(1553851774),  instance_id = 1,  internal_downtime_id = '35',  is_fixed = '1',  is_in_effect = '0',  name = 'vsm_sapcp2_appl01!Check TCP Port!7c9dc60f-92e8-4931-85fd-1172784e3d9d',  object_id = 3021,  scheduled_end_time = FROM_UNIXTIME(1554095700),  scheduled_start_time = FROM_UNIXTIME(1554090300),  session_token = 1553851712,  was_started = '0' WHERE entry_time = FROM_UNIXTIME(1553851774) AND name = 'vsm_sapcp2_appl01!Check TCP Port!7c9dc60f-92e8-4931-85fd-1172784e3d9d' AND object_id = 3021
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET scheduled_downtime_depth = '0' WHERE instance_id = 1 AND service_object_id = 3021
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_downtimehistory (actual_start_time, actual_start_time_usec, author_name, comment_data, downtime_type, duration, endpoint_object_id, entry_time, instance_id, internal_downtime_id, is_fixed, is_in_effect, name, object_id, scheduled_end_time, scheduled_start_time, was_started) VALUES (FROM_UNIXTIME(1554090300), '0', 'icingaadmin', 'Monatliche Wartung VSM-CP2', '1', '0', 493, FROM_UNIXTIME(1553851774), 1, '35', '1', '0', 'vsm_sapcp2_appl01!Check TCP Port!7c9dc60f-92e8-4931-85fd-1172784e3d9d', 3021, FROM_UNIXTIME(1554095700), FROM_UNIXTIME(1554090300), '0')
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_scheduleddowntime SET actual_start_time = FROM_UNIXTIME(1554090300),  actual_start_time_usec = '0',  author_name = 'icingaadmin',  comment_data = 'Monatliche Wartung VSM-CP2',  downtime_type = '1',  duration = '0',  endpoint_object_id = 493,  entry_time = FROM_UNIXTIME(1553851774),  instance_id = 1,  internal_downtime_id = '36',  is_fixed = '1',  is_in_effect = '0',  name = 'vsm_sapcp2_db!ping4!70f54537-4cf6-4846-aad5-5a18b9a710b8',  object_id = 3132,  scheduled_end_time = FROM_UNIXTIME(1554095700),  scheduled_start_time = FROM_UNIXTIME(1554090300),  session_token = 1553851712,  was_started = '0' WHERE entry_time = FROM_UNIXTIME(1553851774) AND name = 'vsm_sapcp2_db!ping4!70f54537-4cf6-4846-aad5-5a18b9a710b8' AND object_id = 3132
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET scheduled_downtime_depth = '0' WHERE instance_id = 1 AND service_object_id = 3132
[2019-03-29 10:29:35 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_downtimehistory (actual_start_time, actual_start_time_usec, author_name, comment_data, downtime_type, duration, endpoint_object_id, entry_time, instance_id, internal_downtime_id, is_fixed, is_in_effect, name, object_id, scheduled_end_time, scheduled_start_time, was_started) VALUES (FROM_UNIXTIME(1554090300), '0', 'icingaadmin', 'Monatliche Wartung VSM-CP2', '1', '0', 493, FROM_UNIXTIME(1553851774), 1, '36', '1', '0', 'vsm_sapcp2_db!ping4!70f54537-4cf6-4846-aad5-5a18b9a710b8', 3132, FROM_UNIXTIME(1554095700), FROM_UNIXTIME(1554090300), '0')

the files in /var/lib/icinga2/api/packages/_api/aspbazinga04-1513244872-1/conf.d/downtimes

object Downtime "b485d3ce-f610-406a-be05-0c8121d1f681" ignore_on_error {
        author = "icingaadmin"
        authoritative_zone = "wuerzburg"
        comment = "Monatliche Wartung VSM-CP2"
        config_owner = "vsm_sapcp2!ping4!reboot-system-vsmo"
        duration = 0.000000
        end_time = 1554095700.000000
        entry_time = 1553851774.661778
        fixed = true
        host_name = "vsm_sapcp2"
        scheduled_by = "vsm_sapcp2!ping4!reboot-system-vsmo"
        service_name = "ping4"
        start_time = 1554090300.000000
        triggered_by = ""
        version = 1553851774.661806
        zone = "wuerzburg"
}

hope you can find anything, thank you

log1c · March 29, 2019, 10:20am

Only thing I find interesting is that it looks like that if a month begins with a Monday(or probably the day you chose in the downtime/time period config), this day is considered to be of the previous month

Evaluating segment: monday -1 april: 05:45-07:15
Considering segment: Mon Apr 29 05:45:00 2019 -> Mon Apr 29 07:15:00 2019
(best match yet)
Evaluating segment: monday -1 august: 05:45-07:15
Considering segment: Mon Aug 26 05:45:00 2019 -> Mon Aug 26 07:15:00 2019
Evaluating segment: monday -1 december: 05:45-07:15
Considering segment: Mon Dec 30 05:45:00 2019 -> Mon Dec 30 07:15:00 2019
Evaluating segment: monday -1 february: 05:45-07:15
Evaluating segment: monday -1 january: 05:45-07:15
Evaluating segment: monday -1 july: 05:45-07:15
Considering segment: Mon Jul 29 05:45:00 2019 -> Mon Jul 29 07:15:00 2019
Evaluating segment: monday -1 june: 05:45-07:15
Considering segment: Mon Jul  1 05:45:00 2019 -> Mon Jul  1 07:15:00 2019
Evaluating segment: monday -1 march: 05:45-07:15
Considering segment: Mon Apr  1 05:45:00 2019 -> Mon Apr  1 07:15:00 2019
(best match yet)
Evaluating segment: monday -1 may: 05:45-07:15
Considering segment: Mon May 27 05:45:00 2019 -> Mon May 27 07:15:00 2019
Evaluating segment: monday -1 november: 05:45-07:15
Considering segment: Mon Nov 25 05:45:00 2019 -> Mon Nov 25 07:15:00 2019
Evaluating segment: monday -1 october: 05:45-07:15
Considering segment: Mon Oct 28 05:45:00 2019 -> Mon Oct 28 07:15:00 2019
Evaluating segment: monday -1 september: 05:45-07:15
Considering segment: Mon Sep 30 05:45:00 2019 -> Mon Sep 30 07:15:00 2019

The algorithm (or whatever) is going through the months alphabetically.
For July and April the date is set to the last Monday, but gets “marked” with best match yet, as both start with a Monday as well.
For the months before those two (March and June) the downtime doesn’t get set to the “real” last Monday, but the first Monday of the following month, as they start with it. And as the day is in the same month as the best match yet day, this gets replaced by it.

Could this have something to do with the 30 or 31 days per month, or the shifting number of days in February in leap years?

edit: Here is a github issue, with a similar problem (with last saturday): https://github.com/Icinga/icinga2/issues/5660
No feedback from the creator and thus closed, though

Basti · March 29, 2019, 10:44am

interesting , but why is the 1.04 the best result for march -1? that would actually be the 25.03

If I look at it that way, the result for January, February, march and june is not correct. However, all other months fit again.

that would confirm the statements of my colleagues. that meant the last 2-3 months, the downtime would no longer fit.

dnsmichi · March 29, 2019, 10:56am

Interesting. I don’t like those code regions, the timeperiod range format inherited from the old world is horrible.

Sounds like an off by one error, or something else. Can you collect all the details found here, and open a new issue on GitHiub?

SDs only schedule the first match for a downtime, the rest comes later. The ranges are not sorted in a specific way, since it is a dictionary in the background, you’ll likely just get alphanumeric sorts by default.

Cheers,
Michael

Basti · March 29, 2019, 11:27am

Issue created.

Thank you
Basti