Configuring different escalations based on downtime

Icinga 2 Core
#1

Hi, I have a machine with the following details

OS : CentOS Linux 8
Icinga2 version : 2.12.1-1

I am curious to know if different escalations can be notified based on the difference between the current time and incident time. The notified user should get a comment / summary about how many hours the machine has been down.
I went through the docs, it talks about this escalation method, but uses


times = {
	begin = xh
	end = xh
	}

I want to know if something like this can be done

	downhours = currenttime - downtime
	
	if (downhours >= 2) {
		apply Notification "level2-escation"
			...
			...
			...
		assign  ... notification.telegram
	}	
	
	else {
		apply Notification "level1-escation"
			...
			...
			...
		assign  ... notification.mail 
	}

Basically the point is, the person receiving the escalation, one hour from the incident should get a statement saying “Host” ABC has been down for the past 1 hour, when he gets the next notification and subsequently, the statement should summarise the number of hours it has been down, saying the “Host” ABC has been down for the past x hours. I am sure Icinga2 has a way to doing this.

Thanks, Best

#2

Hi @blueaquan,

I think something simple will do like this:

apply Notification "level1" to Service {
  import "generic-notification"
  command = "mail-notification"
  times.begin = 15m // delay notification window
}
apply Notification "level2" to Service {
  import "generic-notification"
  command = "mail-notification-escalation"
  times.begin = 2h // delay notification window
}

From here you can adapt the escalation command to include a different alert text. Knowing the second alert will trigger after 2 hours so you can put that in the body but it will make it static then you create a new alert script for every escalation.

or you could make this more dynamic with some variables from icinga to the script, and use bash or python to do the calculation. Either way the configuration will happen more on the escalation script side of things rather than on the main icinga side as that will only trigger the script and not do the actual alerting.

Hope it helps

1 Like
#3

Thank you @belastingvormulier for your quick reply. The only problem with times.begin = as you mentioned is that it’s static. Yes, the calculation part could be done on a script, but I was wondering if there is an out of the box way that Icinga2 provides which can be used to achieve this…?

The escalation part does not worry me much, I want to be able to send a statement to the Team that receives these escalation on how long a particular Service has been down. For better clarity, on my testing machine I set the notification interval of 10 minutes for a particular IP that was unavailable. I noticed each of those notifications that came to me at every 10 min, carried the timestamp of when that particular notification was generated, but it does not say, how long the service has been down. My goal, is just that, to give some sort of information on when the service first become unavailable or the time difference between when the service first became unavailable to the present moment".

Thanks, Best

#4

Have you seen this:

There is a:

host.last_state_change - The last state change’s timestamp

You could put this in your alert from icinga to your script It would be the time since your host / service is in hard state.
The icinga slack notification script uses this, so perhaps that can be a sneak peak on how to use it.

1 Like
#5

Thank you @belastingvormulier, yes, I had gone through that documentation, but I couldn’t recall it, as often it happens with theory. But, now since it’s an experiential knowledge, it will remain in my memory.

Thank you for pointing this out, it solved the problem. Here’s what I did

commands.conf

	env = {
		...
		...
		...
		EPOCHTIME = "$host.last_state_change$"
	}

telegram-host-notification.sh

EVENT_TIME=$(date -d @$EPOCHTIME)
	template=$(
	...
	...
	...
	*Escalation: $HOSTALIAS IS DOWN SINCE $EVENT_TIME*
	)

Thank you once again, it was wonderful learning.

1 Like