Notifications for passive services

Hi together,
i’m currently fighting with the notifications for passive services. The most of them are traps.
I’d created a template for this services:

template Service "Passive Service Volatile Template" {
check_command = "trap-passive"
max_check_attempts = "1"
check_period = "timeperiod-24x7"
check_interval = 1d
enable_notifications = true
enable_active_checks = true
enable_passive_checks = true
enable_event_handler = false
enable_flapping = false
enable_perfdata = false
volatile = true
vars.enable_trap_notification = true

}

For the notification i’d created this template:

template Notification "contact-24x7-trap-service-template" {
command = "mail-service-notification-html"
interval = 0s
period = "timeperiod-24x7"
states = [ Critical, Warning ]
types = [ Problem ]

}

My great problem that every trap will send out a notification, also if the service was critical before.

I know that “volatile = true” is a param for this, but if i set this to false i don’t see all received trap anymore. Only the state changes were available in the history. The support team will see all traps which are arrived but they only wont to get a notification on the first state change.

Does anyone have an idea to get this working like i described?

Regards
Sascha

PS: I’d checked the old configuration of ICINGA 1. In ICINGA 1 it works like charm. We’d seen every trap in the history but the support only gets one notification after the state change. We’d done this with escalation rules:

define serviceescalation {
hostgroup_name PRODUCTION
service_description *
contact_groups +CCO-SUPPORT-CALL
first_notification 1
last_notification 1
notification_interval 0
last_warning_notification 1
last_critical_notification 1
escalation_period ZEITPERIODE-24x7-TEMPLATE
escalation_options c,w
}

Have you tried playing with

times = {
        end = 1m
    }

and put some time value there?
Not sure if this would help, I have no experience with this myself

Thanks for reply.
Yes - i’d played with this values:

apply Notification “cco1-contact-24x7-trap-service” to Service {
times = {
begin = 1s
end = 5s
}
command = “mail-service-notification-html”
interval = 0s
period = “timeperiod-24x7”
assign where (“production” in host.groups || “cco-server” in host.groups) && service.vars.enable_trap_notification && service.enable_notifications && service.vars.passive_service_type == “TRAP”
states = [ Critical, Warning ]
types = [ Problem ]
user_groups = [ “cco1-traps-email” ]
}

If i set the service to critical sometime i got an email and sometimes not :frowning:
in the logfile i will see this:

[2020-01-30 14:46:27 +0100] information/Checkable: Checkable ‘fw3-dmz.tld.com!CRITICAL_ALARM_TRAP’ has 2 notification(s). Checking filters for type ‘Problem’, sends will be logged.

And this is teh contact

object User “Sascha” {
import “Email-Template”

display_name = "Sascha"
email = "******@*****"
enable_notifications = true
states = [ Critical, Down, Up, Warning ]
types = [ Acknowledgement, Custom, DowntimeStart, Problem ]
groups = [
    "cco-email",
    "cco-traps-email",
]
vars.SM9_ASSIGNMENT_GROUP = ""
vars.can_create_tickets = true
vars.icinga_ldap_username = ""
vars.notification_handy_phone_number = ""
vars.notification_phone_number = ""

}

In my opinion the email have to be sent out, but not every time.

If i remove the times i got for every trap an email. It doesn’t matter whether there is a state change or not. And this comes from the “volatile = true”.

From the docs

It may be reasonable to have a volatile service which stays in a HARD state if the service stays in a NOT-OK state. That way each service recheck will automatically trigger a notification unless the service is acknowledged or in a scheduled downtime.

Might setting an automatic acknowledgment after the first notification be something worth persuing?

Good morning. I understand but this isn’t a way we wont to go.
We’d changed last year from ICINGA1 to ICINGA2 and our support team ackknowledge the NON-OK services by themself, because we are using this to create a ticket in our ticket system.
I can change the “volatile = true” to “volatile = true” and the notifications will work like we want but then the following traps wont be logged in the history. And that is an importent thing for the support.

Under ICINGA1 we’d stop the emails on volatile services with this 2 params:

last_warning_notification 1
last_critical_notification 1

But they do not exist anymore.

Is it possible to disbled to email notifications after the first STATE change with a dependencies?

apply Dependency “Disabled Notification wheter NON-OK” to Service {
disable_checks = false
disable_notifications = true
ignore_soft_states = true
period = “timeperiod-24x7”
assign where service.volatile && service.vars.passive_service_type == “TRAP”
states = [ Critical, Warning ]
}

If this works then it is ok for us.

If your problem is that not all traps are logged in history, maybe you should take a look here: History for passive check results

Thanks Stevie. That’s the same problem we have… :frowning:
But a new system like loki is no solution in our enviroment.

Our workarround is to send passive check results from logstash into icinga is following:
First we send a “unknown”-state with the plugin output “to provoke a change of status” and then the real message. Because we use logstash to parse traps it was the eaisiest way
At the moment we also don’t think about to install a complete ELK stack only for this.

We are parsing the hole traps with an own script and make them ICINGA compatible services. So a logstash server is not needed. Ok. A workarround will be send an UNKNOW before sending the real STATUS. But the status change will trigger a notification and in our enviroment is it often a phone call.
They will kill me if a service sends every second a phone call… :wink:

Not really! Only, if you send for every state a notification. If you send notifications only for up/down, ok/critical then you could send a unknown state to icinga.

Do you know trapdirector? I’m not using it but it sounds promising.

1 Like

We only send on WARNING and CRITICAL a notification. And the state change from UNKNOWN to CRITCIAL will trigger a new notification. But we want only have a notification if the service changes from OK to CRITICAL, but we want have all sendet trap informations in the history

very interesting. Thanks for this. But i need now a fast solution. I will test it the next time

hmm… ok the truth is you have two problems.

  • for the history you could try to provoke a state change like I have proposed.
  • for the notification you have to find a way to get only such messages you want to have. but for this i have no idea. Only to filter them like written in the docs

I’d tried a lot of settings. But nothing works like under ICINGA 1.
At the moment i think about to extend my email script to create a flag file which includes th last service state and before sending an email or creating a call the status will be checked. But i must take some thoughts about this.

What about my idea with the dependencies?
Is this not a way to stop the email and call notifications?

For the dependencies:
So far there has been no requirement where we have to worry about such a situation. Sorry

Thanks for all your ideas and your help. I hope i will find a solution for this problem.
Maybe i will open a feature request for the old param from ICINGA 1

1 Like