Aligning Icinga2 notifications with external alert policy

Icinga 2
1

Hello,

I am using Icinga2 v2.16 on Debian 13.

I am currently doing an overhaul of all my services and notifications to match my company’s policy on alarms. Basically we have 3 levels of alarm, say yellow, orange and red in order of severity. These levels mean that different actions will be taken (for example email different people or send notifications to different channels). However, this does not necessarily map with the Warning/Critical states of Icinga. For example, Service A should send a yellow alert only when critical. Or Service B should send an orange alert when warning, but red when critical. Or Service C should send a yellow warning when warning, but orange when critical.

My question is, how could I best structure this? I could of course make separate warning/critical notifications for each service but this is going to be very redundant and messy. I could also make notifications like “yellow-warning”, “yellow-critical”, “orange-warning”, etc. but I wonder if there’s a better way of handling this.

Thank you

2

Hi Gaëtan,

I would consider describing the criticality of services and hosts at Service and Host level, respectively, with custom variables, and use the custom variables in the Notification Apply Rules.

Hope this helps,

Jean

3

Thanks. I see, do you mean something like this? It makes sense.

apply Service "MyService" to Host {
  //import...
  
  vars.warningLevel = "orange"
  vars.criticalLevel = "red"

  //rest of the service
  //...
}

function SendAlarmNotifications(colour) {
	if(colour == "yellow") {
	  //send yellow notifications
	}
	else if(colour == "orange") {
	  //send orange notifications	
	}
	else if(colour == "red") {
	  //send red notifications		
	}
}

apply Notification "warning-service" to Service {
    //import...
	
	states = [ Warning ]
    
	SendAlarmNotifications(vars.warningLevel)

    //assign where ...
}

apply Notification "critical-service" to Service {
    //import...
	
	states = [ Warning ]
    
	SendAlarmNotifications(vars.criticalLevel)

    //assign where ...
}
4

I must admit I was too lazy and too short on time to envision the solution. I was just pointing to a potential direction, i.e., describe the Service and Host objects with custom variables, provide the values of the variables at the correct level (Host/Service template, Host/Service object, or Service Apply Rule), and then use the variables where it makes sense (in this case the Notification Apply Rule).

What I see above goes along those lines, and I am very glad you tried the suggestion.

NB: We use the Director, and I am not familiar at all with the “function” object type. I thought the custom variables would be used in the where clause of Notification Apply rules

5

Hi, you might want to handle this with an external alerting service like SIGNL4: Icinga » SIGNL4 .

In that setup, Icinga sends all events to SIGNL4, and from there you can filter and route alerts however you like. For example, critical alerts can wake someone up at night, while less critical ones can politely let the on-call team keep dreaming. You can also enrich alerts with colors, icons, and other useful context.

Full disclosure: I’m with SIGNL4, so my opinion may come with a tiny bit of branded seasoning. That said, SIGNL4 integrates nicely with Icinga, including two-way status updates, alerting via app push, SMS, and voice calls, plus escalation and on-call scheduling.

You can find the documentation here: Icinga | SIGNL4 Docs .

6

For inspiration, this is what we do for notifications:

Services have these notification-specific variables:

image
image708×503 8.63 KB

(Hosts have similiar variables for host-specific states)

Based on these we have a `apply for` rule in our master zone (configured via config files)

apply Notification "notification-jira-service_" for (state in [ "critical", "warning" ]) to Service {
        import "jira-service-notification-template"

        if (host.vars.ticketsystem == "this") {
                if (state == "warning") {
                        users = [ "this-ticketsystem_warning" ]
                } else {
                        users = [ "this-ticketsystem" ]
                }
        } else {
                if (state == "warning") {
                        users = [ "that-ticketsystem_warning" ]
                } else {
                        users = [ "that-ticketsystem" ]
                }
        }


        # 24x7
        if (service.vars["notification_period_" + state] == "24x7") {
                period = "24x7"
                assign where service.vars["notification_period_" + state] == "24x7" && service.vars.notification_contact == ""

        # 8x5
        } else if (service.vars["notification_period_" + state] == "8x5") {
               period = "8x5"
               assign where service.vars["notification_period_" + state] == "8x5" && service.vars.notification_contact == ""

        # Default
        # Fallback 
        } else {
                if (state == "critical") {
                        period = "24x7"

                        assign where true && service.vars.notification_contact == ""
                }
        }
        var notification_delay = service.vars["notification_delay_" + state]
        if (service.vars["notification_delay_" + state]) {
                times.begin = number(service.vars["notification_delay_" +  state])
        } else if (number(DateTime().format("%H")) < 8 || number(DateTime().format("%H")) > 18 ) {
                times.begin = 900
        }

		# ignore if host should not notify
        ignore where host.vars.notification_alerting == false

        # ignore if service should not notify
        ignore where service.vars["notification_period_" + state] == "none"
}
7

Thanks, it’s actually pretty similar to what I’ve written above. I’ve done a poc and it works fine