How do send notification from a problem occuring outside the support timeperiod when the period becomes active

Hi there,

the subject already explains what I am trying to achieve but I’ll try to extend here based on what I am trying to achieve. My support period basically covers 8x5 (so a basic workday support schedule). Now if something happens at midnight and continues to be a problem till the workday support schedule starts I want to immediately notify. As far as I understand notifications only trigger on state changes, so I have no good idea on how to notify those users.

Any pointers?

Thanks,
Florian

Hi @apollo13,

welcome to the community!

Afaik, it’s not possible to get this notification immediately after the start of the notification period but you could use the interval property for notifications to re-send notifications every hour - unless the problem was acknowledged.

Best,
Marcel

HI Marcel,

Thank you for your response and the warm welcome.

I thought about using interval, but it doesn’t do exactly what I want. What I am going with for now is that I’ll apply the notification twice (one notification during support hours and one outside of it) and pass a custom variable down to the notification command. The extra variable will determine the severity of the alert when it is passed on to pagerduty. During night those events will be routed with low priority by pagerduty and once the support hours start it will re escalate existing open alerts with high priority immediately triggering on-call agents. During day, they will be delivered as high priority to pagerduty and also immediately trigger on-call.

The only downside is that I’ll have to keep support hours in Icinga and pagerduty in sync, but that’s just a minor annoyance.

Either way, I’ll be interested to hear if there are other ideas to solve this problem :slight_smile:

thanks,
Florian

Have you tried it with a dummy check? I am not 100% sure, but I think the notification should be sent immediately when the time period starts, if the problem still persists.

1 Like

The idea from log1c sounds good, too!

I only used opsgenie and do not have any experience with pagerduty but why do you have to maintain the support hours twice? Isn’t it sufficient if pagerduty knows who to ping? Or do you bypass pagerduty during the support hours?

I think I tried that, but I guess I’ll have to double-check

Doing it in pagerduty would be sufficient if I could change the urgency of the notification during support-hours. Sadly pagerduty doesn’t allow me to do this so I use icinga to send as high urgency during support hours and low urgency otherwise.