template Notification "contact-24x7-trap-service-template" {
command = "mail-service-notification-html"
interval = 0s
period = "timeperiod-24x7"
states = [ Critical, Warning ]
types = [ Problem ]
}
My great problem that every trap will send out a notification, also if the service was critical before.
I know that “volatile = true” is a param for this, but if i set this to false i don’t see all received trap anymore. Only the state changes were available in the history. The support team will see all traps which are arrived but they only wont to get a notification on the first state change.
Does anyone have an idea to get this working like i described?
Regards
Sascha
PS: I’d checked the old configuration of ICINGA 1. In ICINGA 1 it works like charm. We’d seen every trap in the history but the support only gets one notification after the state change. We’d done this with escalation rules:
Thanks for reply.
Yes - i’d played with this values:
apply Notification “cco1-contact-24x7-trap-service” to Service {
times = {
begin = 1s
end = 5s
}
command = “mail-service-notification-html”
interval = 0s
period = “timeperiod-24x7”
assign where (“production” in host.groups || “cco-server” in host.groups) && service.vars.enable_trap_notification && service.enable_notifications && service.vars.passive_service_type == “TRAP”
states = [ Critical, Warning ]
types = [ Problem ]
user_groups = [ “cco1-traps-email” ]
}
If i set the service to critical sometime i got an email and sometimes not
in the logfile i will see this:
[2020-01-30 14:46:27 +0100] information/Checkable: Checkable ‘fw3-dmz.tld.com!CRITICAL_ALARM_TRAP’ has 2 notification(s). Checking filters for type ‘Problem’, sends will be logged.
In my opinion the email have to be sent out, but not every time.
If i remove the times i got for every trap an email. It doesn’t matter whether there is a state change or not. And this comes from the “volatile = true”.
It may be reasonable to have a volatile service which stays in a HARD state if the service stays in a NOT-OK state. That way each service recheck will automatically trigger a notification unless the service is acknowledged or in a scheduled downtime.
Might setting an automatic acknowledgment after the first notification be something worth persuing?
Good morning. I understand but this isn’t a way we wont to go.
We’d changed last year from ICINGA1 to ICINGA2 and our support team ackknowledge the NON-OK services by themself, because we are using this to create a ticket in our ticket system.
I can change the “volatile = true” to “volatile = true” and the notifications will work like we want but then the following traps wont be logged in the history. And that is an importent thing for the support.
Under ICINGA1 we’d stop the emails on volatile services with this 2 params:
last_warning_notification
1
last_critical_notification
1
But they do not exist anymore.
Is it possible to disbled to email notifications after the first STATE change with a dependencies?
apply Dependency “Disabled Notification wheter NON-OK” to Service {
disable_checks = false
disable_notifications = true
ignore_soft_states = true
period = “timeperiod-24x7”
assign where service.volatile && service.vars.passive_service_type == “TRAP”
states = [ Critical, Warning ]
}
Our workarround is to send passive check results from logstash into icinga is following:
First we send a “unknown”-state with the plugin output “to provoke a change of status” and then the real message. Because we use logstash to parse traps it was the eaisiest way
At the moment we also don’t think about to install a complete ELK stack only for this.
We are parsing the hole traps with an own script and make them ICINGA compatible services. So a logstash server is not needed. Ok. A workarround will be send an UNKNOW before sending the real STATUS. But the status change will trigger a notification and in our enviroment is it often a phone call.
They will kill me if a service sends every second a phone call…
Not really! Only, if you send for every state a notification. If you send notifications only for up/down, ok/critical then you could send a unknown state to icinga.
We only send on WARNING and CRITICAL a notification. And the state change from UNKNOWN to CRITCIAL will trigger a new notification. But we want only have a notification if the service changes from OK to CRITICAL, but we want have all sendet trap informations in the history
very interesting. Thanks for this. But i need now a fast solution. I will test it the next time
for the history you could try to provoke a state change like I have proposed.
for the notification you have to find a way to get only such messages you want to have. but for this i have no idea. Only to filter them like written in the docs
I’d tried a lot of settings. But nothing works like under ICINGA 1.
At the moment i think about to extend my email script to create a flag file which includes th last service state and before sending an email or creating a call the status will be checked. But i must take some thoughts about this.
What about my idea with the dependencies?
Is this not a way to stop the email and call notifications?
Thanks for all your ideas and your help. I hope i will find a solution for this problem.
Maybe i will open a feature request for the old param from ICINGA 1