Notifications

I know there is a traditional way of enabling mail notifications by using the following format:

object Host “server1” {
check_command = “hostalive”
address = “10.10.10.10”
vars.notification[“mail”] = {
groups = [ “sysadmins” ]
}
}

Does this mean I have to declare the mail notification in every object host or is there a way to make it global?

Hi,

this isn’t really the traditional way of doing so, it just mimics the 1.x habits with defining contacts on the host level. Typically you would define the users and user groups inside the notification apply rule, that’s what I do normally.

apply Notification "..." to Service {

   users = [ "admin1", "oncall2" ]
   user_groups = [ "oncall" ]

  ...
}

Having such, you don’t need the host’s data to create a notification, even global with assign where true.

Cheers,
Michael

Is this correct?

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = host.vars.notification.mail.groups
  users = host.vars.notification.mail.users

  //interval = 2h

  //vars.notification_logtosyslog = true

  assign where host.vars.notification.mail
}

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = host.vars.notification.mail.groups
  users = host.vars.notification.mail.users

  //interval = 2h

  //vars.notification_logtosyslog = true

  assign where host.vars.notification.mail
}

That’s from the example docs which takes host as reference into account and reads the nested dictionary values.

Instead, I would just define users and user_groups with an array of names to actual User and UserGroup objects.

Cheers,
Michael

Please forgive me I am trying to understand how this works. So I must be doing something wrong, this is what I have now and it doesn’t send any notifications

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = [ "sysadmins" ]
  assign where host.vars.notification.mail
}

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = [ "sysadmins" ]
  assign where host.vars.notification.mail
}

I would do the usual things:

  • Verify that the Notification object is created from apply rules.
    • The config validation will log warnings for non-matching apply rules.
    • icinga2 object list --type Notification --name ...
  • Verify that the default state and types filters match
  • Verify that the User object doesn’t filter away the notification event
  • Enable the debug log and trace the notification being triggered up until the script is fired, check whether specific conditions hinder that.

More on that in the troubleshooting docs.

Cheers,
Michael

I have searched high and low. When I add the

vars.notification[“mail”] = {
groups = [ “sysadmins” ]
}

To the host object the mail notification works. Without this I get nothing.

Here is my configuration for transparency.

Templates.conf

template Host "generic-host" {
  max_check_attempts = 3
  check_interval = 1m
  retry_interval = 30s
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
  vars.notification["telegram"] = {
    users = [ "telegram-bot" ]
  }

  check_command = "hostalive"
}

template Service "generic-service" {
  max_check_attempts = 5
  check_interval = 1m
  retry_interval = 30s
  vars.notification["telegram"] = {
   users = [ "telegram-bot" ]
 }

}

template User "generic-user" {

}

##############################
########  EMAIL ##############
##############################

##### HOST #####

template Notification "mail-host-notification" {
  command = "mail-host-notification"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_logtosyslog = false
    notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
    notification_from = "infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
  }

  period = "24x7"
}

##### SERVICE #####

template Notification "mail-service-notification" {
  command = "mail-service-notification"

  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
    notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
    notification_logtosyslog = false
  }

  period = "24x7"
}

#################################
########### TELEGRAM ############
#################################


##### HOST #####

template Notification "telegram-host-notification" {
  command = "telegram-host-notification"
 
  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  vars += {
    notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
    notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
    notification_logtosyslog = false
  }
  period = "24x7"
}

##### SERVICE #####
 
template Notification "telegram-service-notification" {
  command = "telegram-service-notification"
 
  states = [ Critical ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  vars += {
    notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
    notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
    notification_logtosyslog = false
  }
 
  period = "24x7"
}

Users.conf

##################################
#### default icinga admin ########
##################################


object User "icingaadmin" {
  import "generic-user"

  display_name = "Icinga 2 Admin"
  groups = [ "icingaadmins" ]

  email = "user@mydomain.com"
}

object UserGroup "icingaadmins" {
  display_name = "Icinga 2 Admin Group"
}

##################################
##########   USERS  ##############
##################################

object User "mainuser" {
  import "generic-user"

  display_name = "Users Name"
  groups = [ "sysadmins" ]

  email = "user@mydomain.com"
}

####### GROUPS #######

object UserGroup "sysadmins" {
  display_name = "IT SysAdmins"
}

##################################
##########  telegram  ############
##################################


object User "telegram-bot" {
  import "generic-user"

  display_name = "telegram-bot"

}

object User "it-telegram-bot" {
  import "generic-user"

  display_name = "it-telegram-bot"
}

Notifications.conf

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = [ "sysadmins" ]
  assign where host.vars.notification.mail
}

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = [ "sysadmins" ]
  assign where host.vars.notification.mail
}

apply Notification "telegram-icingaadmin" to Host {
  import "telegram-host-notification"
  user_groups = host.vars.notification.telegram.groups
  users = host.vars.notification.telegram.users
 
  assign where host.vars.notification.telegram
}
 
apply Notification "telegram-icingaadmin" to Service {
  import "telegram-service-notification"
  user_groups = host.vars.notification.telegram.groups
  users = host.vars.notification.telegram.users
 
  assign where host.vars.notification.telegram
 
}

The notification feature has been enabled.

I hope you can help.

You did not check whether a notification object was created from the apply rule by using object list. I really doubt there is one - hint: The filter doesn’t match. Follow my advice and steps please, I’m not writing them for fun :stuck_out_tongue:

-> here

I apologise I am not trying to mess you around. I guess I dont really understand “filiter doesnt match”

Get to know how apply rules work, and how such assign where expressions are modeled.

The above for example only returns true, if the Host object has vars.notification set, with the key mail inside the nested dictionary. If none of your hosts has this, or the specific one you’re expecting a notification for, this expression always returns false. Meaning to say, no notification object can be created.

That is why I’m pointing you towards object list and modifying the filter, with a gentle poke to try and learn. I could also post the solution for copy paste, but that’s boring and won’t help you with further questions.

Cheers,
Michael

okay you have beat me, I have been reading and I am still clueless.

I don’t want to beat you, I’m wondering why you don’t understand how apply rules generate objects.

Write an apply rule like this:

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = [ "sysadmins" ]
  assign where host.address != ""
}

Now run icinga2 object list --type Notification --name *mail-icingaadmin. Does it return a notification config object?

If yes, try to trigger a notification via Icinga Web.

Next up, modify the assign where expression in the notification apply rule.

Set it to match your host name, by using the match() function for example. Try things out, not just read them.

Cheers,
Michael

It does return an object but when I validate the config I get this

[2019-04-26 15:38:56 +0200] critical/config: Error: Validation failed for object 'mydomain.com!icinga!mail-icingaadmin' of type 'Notification'; Attribute 'states': State filter is invalid.
Location: in /etc/icinga2/conf.d/templates.conf: 38:3-38:23
/etc/icinga2/conf.d/templates.conf(36):   command = "mail-host-notification"
/etc/icinga2/conf.d/templates.conf(37): 
/etc/icinga2/conf.d/templates.conf(38):   states = [ Up, Down ]
                                          ^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/conf.d/templates.conf(39):   types = [ Problem, Acknowledgement, Recovery, Custom,
/etc/icinga2/conf.d/templates.conf(40):             FlappingStart, FlappingEnd,

Uhm, never seen this before. Please show me the output of icinga2 --version.

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.2-1)

System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64

Build information:
Compiler: GNU 6.3.0
Build host: 486e413fb159

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

Please show the content from line 36 with some lines around it - or attach the entire file.

When such a validation error occurs, the host states filter is put into a Service notification, where it is not valid.

HOST

template Notification “mail-host-notification” {
command = “mail-host-notification”

states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]

vars += {
notification_logtosyslog = false
notification_icingaweb2url = “https://inframon.mydomain.com/icingaweb2
notification_from = “Infrastructure Icinga2 Monitoring Server inframon@mydomain.com
}

period = “24x7”
}

SERVICE

template Notification “mail-service-notification” {
command = “mail-service-notification”

states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]

vars += {
notification_icingaweb2url = “https://inframon.mydomain.com/icingaweb2
notification_from = “Infrastructure Icinga2 Monitoring Server inframon@mydomain.com
notification_logtosyslog = false
}

period = “24x7”
}

That’s not possible …

Ok, then do a recursive grep on here the host notification template is used. I bet that it is included in a notification apply to Service rule.

grep -A 10 -B 10 -r mail-host-notification /etc/icinga2/

/etc/icinga2/conf.d/templates.conf-template User "generic-user" {
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-}
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-##############################
/etc/icinga2/conf.d/templates.conf-########  EMAIL ##############
/etc/icinga2/conf.d/templates.conf-##############################
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-##### HOST #####
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf:template Notification "mail-host-notification" {
/etc/icinga2/conf.d/templates.conf:  command = "mail-host-notification"
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-  states = [ Up, Down ]
/etc/icinga2/conf.d/templates.conf-  types = [ Problem, Acknowledgement, Recovery, Custom,
/etc/icinga2/conf.d/templates.conf-            FlappingStart, FlappingEnd,
/etc/icinga2/conf.d/templates.conf-            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-  vars += {
/etc/icinga2/conf.d/templates.conf-    notification_logtosyslog = false
/etc/icinga2/conf.d/templates.conf-    notification_icingaweb2url = "https://inframon.mydomain.com/icingaweb2"
/etc/icinga2/conf.d/templates.conf-    notification_from = "infrastructure Icinga2 Monitoring Server <inframon@cognitec.com>"
--
/etc/icinga2/conf.d/notifications.conf-apply Notification "mail-icingaadmin" to Host {
/etc/icinga2/conf.d/notifications.conf:  import "mail-host-notification"
/etc/icinga2/conf.d/notifications.conf-  user_groups = [ "sysadmins" ]
/etc/icinga2/conf.d/notifications.conf-  assign where host.address !=" "
/etc/icinga2/conf.d/notifications.conf-}
/etc/icinga2/conf.d/notifications.conf-
/etc/icinga2/conf.d/notifications.conf-apply Notification "mail-icingaadmin" to Service {
/etc/icinga2/conf.d/notifications.conf:  import "mail-host-notification"
/etc/icinga2/conf.d/notifications.conf-  user_groups = [ "sysadmins" ]
/etc/icinga2/conf.d/notifications.conf-  assign where host.address !=" "
/etc/icinga2/conf.d/notifications.conf-}
/etc/icinga2/conf.d/notifications.conf-
/etc/icinga2/conf.d/notifications.conf-apply Notification "telegram-icingaadmin" to Host {
/etc/icinga2/conf.d/notifications.conf-  import "telegram-host-notification"
/etc/icinga2/conf.d/notifications.conf-  user_groups = host.vars.notification.telegram.groups
/etc/icinga2/conf.d/notifications.conf-  users = host.vars.notification.telegram.users
/etc/icinga2/conf.d/notifications.conf-  assign where host.vars.notification.telegram
/etc/icinga2/conf.d/notifications.conf-}
--
/etc/icinga2/conf.d/commands.conf-        "-H" = "$host$"
/etc/icinga2/conf.d/commands.conf-        "-c" = "$check$"
/etc/icinga2/conf.d/commands.conf-        "-t" = "20:3"
/etc/icinga2/conf.d/commands.conf-        }
/etc/icinga2/conf.d/commands.conf-}
/etc/icinga2/conf.d/commands.conf-
/etc/icinga2/conf.d/commands.conf-################################
/etc/icinga2/conf.d/commands.conf-####   MAIL NOTIFICATIONS   ####
/etc/icinga2/conf.d/commands.conf-################################
/etc/icinga2/conf.d/commands.conf-
/etc/icinga2/conf.d/commands.conf:object NotificationCommand "mail-host-notification" {
/etc/icinga2/conf.d/commands.conf:  command = [ SysconfDir + "/icinga2/scripts/mail-host-notification.sh" ]
/etc/icinga2/conf.d/commands.conf-
/etc/icinga2/conf.d/commands.conf-  env = {
/etc/icinga2/conf.d/commands.conf-    NOTIFICATIONTYPE = "$notification.type$"
/etc/icinga2/conf.d/commands.conf-    HOSTALIAS = "$host.display_name$"
/etc/icinga2/conf.d/commands.conf-    HOSTADDRESS = "$address$"
/etc/icinga2/conf.d/commands.conf-    HOSTSTATE = "$host.state$"
/etc/icinga2/conf.d/commands.conf-    LONGDATETIME = "$icinga.long_date_time$"
/etc/icinga2/conf.d/commands.conf-    HOSTOUTPUT = "$host.output$"
/etc/icinga2/conf.d/commands.conf-    NOTIFICATIONAUTHORNAME = "$notification.author$"
/etc/icinga2/conf.d/commands.conf-    NOTIFICATIONCOMMENT = "$notification.comment$"
--
/etc/icinga2/scripts/mail-service-notification.sh-       Usage ;;
/etc/icinga2/scripts/mail-service-notification.sh-    :) echo "Missing option argument for -$OPTARG" >&2
/etc/icinga2/scripts/mail-service-notification.sh-       Usage ;;
/etc/icinga2/scripts/mail-service-notification.sh-    *) echo "Unimplemented option: -$OPTARG" >&2
/etc/icinga2/scripts/mail-service-notification.sh-       Usage ;;
/etc/icinga2/scripts/mail-service-notification.sh-  esac
/etc/icinga2/scripts/mail-service-notification.sh-done
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh-shift $((OPTIND - 1))
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh:## Keep formatting in sync with mail-host-notification.sh
/etc/icinga2/scripts/mail-service-notification.sh-for P in LONGDATETIME HOSTNAME HOSTDISPLAYNAME SERVICENAME SERVICEDISPLAYNAME SERVICEOUTPUT SERVICESTATE USEREMAIL NOTIFICATIONTYPE ; do
/etc/icinga2/scripts/mail-service-notification.sh-        eval "PAR=\$${P}"
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh-        if [ ! "$PAR" ] ; then
/etc/icinga2/scripts/mail-service-notification.sh-                Error "Required parameter '$P' is missing."
/etc/icinga2/scripts/mail-service-notification.sh-        fi
/etc/icinga2/scripts/mail-service-notification.sh-done
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh-## Build the message's subject
/etc/icinga2/scripts/mail-service-notification.sh-SUBJECT="[$NOTIFICATIONTYPE] $SERVICEDISPLAYNAME on $HOSTDISPLAYNAME is $SERVICESTATE!"
--
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-template User "generic-user" {
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-}
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-/**
/etc/icinga2/templates.conf.bak- * Provides default settings for host notifications.
/etc/icinga2/templates.conf.bak- * By convention all host notifications should import
/etc/icinga2/templates.conf.bak- * this template.
/etc/icinga2/templates.conf.bak- */
/etc/icinga2/templates.conf.bak:template Notification "mail-host-notification" {
/etc/icinga2/templates.conf.bak:  command = "mail-host-notification"
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-  states = [ Up, Down ]
/etc/icinga2/templates.conf.bak-  types = [ Problem, Acknowledgement, Recovery, Custom,
/etc/icinga2/templates.conf.bak-            FlappingStart, FlappingEnd,
/etc/icinga2/templates.conf.bak-            DowntimeStart, DowntimeEnd, DowntimeRemoved ]
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-  vars += {
/etc/icinga2/templates.conf.bak-    // notification_icingaweb2url = "https://www.example.com/icingaweb2"
/etc/icinga2/templates.conf.bak-    // notification_from = "Icinga 2 Host Monitoring <icinga@example.com>"
/etc/icinga2/templates.conf.bak-    notification_logtosyslog = false

Ok, here we go.

/etc/icinga2/conf.d/notifications.conf-apply Notification "mail-icingaadmin" to Service {
/etc/icinga2/conf.d/notifications.conf:  import "mail-host-notification"
/etc/icinga2/conf.d/notifications.conf-  user_groups = [ "sysadmins" ]
/etc/icinga2/conf.d/notifications.conf-  assign where host.address !=" "
/etc/icinga2/conf.d/notifications.conf-}

A notification apply rule has a target. If you say to Service, you want to create notification objects for service objects. Within a service notification, the specific service states are valid filters. Meaning to say, when you want to receive a notification for a service turning Critical, you’ll set that into the states filters for the service notification.

On the other hand, to Host creates host notification objects, which serve the purpose of alerting you when a host goes Down for example. If you want to receive this state per notification, add it to the states filter.

Now comes the problem: You need to import the correct template which sets specific attributes in this scope, there isn’t a generic one. That is why the example configuration uses two Notification templates:

  • mail-host-notification for Host apply rules
  • mail-service-notification for Service apply rules

In your configuration, you’re importing the host notification template into the service notification apply rule. The config compiler is correct about this, the states attribute is not valid in this scope.

The fully constructed object, without the template looks like this in memory:

apply Notification "mail-icingaadmin" to Service {
//  import "mail-host-notification" //template source
  user_groups = [ "sysadmins" ]

//from the imported template
  command = “mail-host-notification”

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
  FlappingStart, FlappingEnd,
  DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_logtosyslog = false
    notification_icingaweb2url = “https://inframon.mydomain.com/icingaweb2”
    notification_from = “Infrastructure Icinga2 Monitoring Server inframon@mydomain.com”
  }

  period = “24x7”
//from imported template, end

  assign where host.address != ""
}

For learning purposes, if you add this somewhere and let the config validation run, it will correctly tell you again, that in this context, the state filter is invalid.

The fix is easy - import the correct template, you’ve made a copy paste error. Understanding why, that is key here.

Cheers,
Michael

1 Like