Renotifications in timeperiods; First one skipped

Icinga 2
1

Hello,

i have a problem, and i hope you can help me.

What do i want to do:
I do have a check, which shall be checking the whole day, but only sends out notifications between 23:00 and 23:30. This check is always in critical state (i know this is dumb, but this is out of my hands).

Service:

apply Service "EC printout times check" {
                check_command = "ec_printout_times_check"
                check_interval = 15m
                retry_interval = 1m
                groups=["single-mail-evening"]
                assign where "DB" in host.groups
}

Timeperiod

object TimePeriod "evening" {
  display_name = "Icinga 2 evening period"
  ranges = {
    "monday"    = "23:00-23:30"
    "tuesday"   = "23:00-23:30"
    "wednesday" = "23:00-23:30"
    "thursday"  = "23:00-23:30"
    "friday"    = "23:00-23:30"
    "saturday"  = "23:00-23:30"
    "sunday"    = "23:00-23:30"
  }
}

Notification:

apply Notification "single-mail-evening-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = ["icingaadmins"]
  vars.notification_from = "<mailaddress>"
  interval = 15m
  period="evening"


assign where "single-mail-evening" in service.groups
}

My problem is, that the first check, which will be executed during the time of the “evening” period, will not trigger a (re)notification sending.

Example:
Check runs an 23:05 no notification generated
Check runs again at 23:20, i will get a notification.

I also checked icinga debug log, and there is no entry about a notification (excerpt from a test 1 hour later)

[2021-06-08 00:12:06 +0000] debug/CheckerComponent: Scheduling info for checkable 'PROD - DB!EC printout times check' (2021-06-08 00:12:06 +0000): Object 'PROD - DB!EC printout times check', Next Check: 2021-06-08 00:12:06 +0000(1.62311e+09).
[2021-06-08 00:12:06 +0000] debug/CheckerComponent: Executing check for 'PROD - DB!EC printout times check'
[2021-06-08 00:12:06 +0000] debug/Checkable: Update checkable 'PROD - DB!EC printout times check' with check interval '900' from last check time at 2021-06-07 23:57:50 +0000 (1.62311e+09) to next check time at 2021-06-08 00:26:22 +0000 (1.62311e+09).
[2021-06-08 00:12:06 +0000] notice/ApiListener: Relaying 'event::SetLastCheckStarted' message
[2021-06-08 00:12:06 +0000] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-06-08 00:12:06 +0000] notice/Process: Running command '/usr/lib//nagios/plugins/ec_printout_times_check.rb' '--critical_value' '60' '--dbtype' 'anon' '--hostname' 'anon' '--password' 'anon' '--username' 'anon': PID 4511
[2021-06-08 00:12:06 +0000] debug/CheckerComponent: Check finished for object 'PROD - DB!EC printout times check'
[2021-06-08 00:12:06 +0000] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 155; Checks/s: 0.716667
...

[2021-06-08 00:12:06 +0000] notice/Process: PID 4511 ('/usr/lib//nagios/plugins/ec_printout_times_check.rb' '--critical_value' '60' '--dbtype' 'anon' '--hostname' 'anon' '--password' 'anon' '--username' 'anon') terminated with exit code 2
[2021-06-08 00:12:06 +0000] debug/Checkable: Update checkable 'PROD - DB!EC printout times check' with check interval '900' from last check time at 2021-06-08 00:12:06 +0000 (1.62311e+09) to next check time at 2021-06-08 00:26:22 +0000 (1.62311e+09).
[2021-06-08 00:12:06 +0000] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-06-08 00:12:06 +0000] notice/ApiListener: Relaying 'event::CheckResult' message

Do i have some error in my configuration?

Thanks in advance for your help

2

I would say yes there is some error in the logic, but let me explain why.

As you say the check is always in an error state, there will be no notification on state change it will only be the re-notification defined by the interval at the notification. This will be suppressed by the timeperiod if it is not between 23:00 and 23:30, so based on the initial state change it will hit this timeperiod twice and send a re-notification, no relation to the service checks anymore.

I assume the output does not change, too, so also making the service volatile will not change the behavior.

If who ever insists on this check behaviour but wants a notification based on the first check in the timeperiod, I would implement a script that changes the state back to OK as passive event about one check interval before the timeperiod starts. So the check and state change will happen in the timeperiod and result in a notification.

3

Hello,

thank you! With your explanation i was able to fix this.