I know there is a traditional way of enabling mail notifications by using the following format:
object Host “server1” {
check_command = “hostalive”
address = “10.10.10.10”
vars.notification[“mail”] = {
groups = [ “sysadmins” ]
}
}
Does this mean I have to declare the mail notification in every object host or is there a way to make it global?
Hi,
this isn’t really the traditional way of doing so, it just mimics the 1.x habits with defining contacts on the host level. Typically you would define the users and user groups inside the notification apply rule, that’s what I do normally.
apply Notification "..." to Service {
users = [ "admin1", "oncall2" ]
user_groups = [ "oncall" ]
...
}
Having such, you don’t need the host’s data to create a notification, even global with assign where true.
Cheers,
Michael
Is this correct?
apply Notification "mail-icingaadmin" to Host {
import "mail-host-notification"
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users
//interval = 2h
//vars.notification_logtosyslog = true
assign where host.vars.notification.mail
}
apply Notification "mail-icingaadmin" to Service {
import "mail-service-notification"
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users
//interval = 2h
//vars.notification_logtosyslog = true
assign where host.vars.notification.mail
}
That’s from the example docs which takes host as reference into account and reads the nested dictionary values.
Instead, I would just define users and user_groups with an array of names to actual User and UserGroup objects.
Cheers,
Michael
Please forgive me I am trying to understand how this works. So I must be doing something wrong, this is what I have now and it doesn’t send any notifications
apply Notification "mail-icingaadmin" to Host {
import "mail-host-notification"
user_groups = [ "sysadmins" ]
assign where host.vars.notification.mail
}
apply Notification "mail-icingaadmin" to Service {
import "mail-service-notification"
user_groups = [ "sysadmins" ]
assign where host.vars.notification.mail
}
I would do the usual things:
icinga2 object list --type Notification --name ...More on that in the troubleshooting docs.
Cheers,
Michael
I have searched high and low. When I add the
vars.notification[“mail”] = {
groups = [ “sysadmins” ]
}To the host object the mail notification works. Without this I get nothing.
Here is my configuration for transparency.
Templates.conf
template Host "generic-host" {
max_check_attempts = 3
check_interval = 1m
retry_interval = 30s
vars.notification["mail"] = {
groups = [ "icingaadmins" ]
}
vars.notification["telegram"] = {
users = [ "telegram-bot" ]
}
check_command = "hostalive"
}
template Service "generic-service" {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s
vars.notification["telegram"] = {
users = [ "telegram-bot" ]
}
}
template User "generic-user" {
}
##############################
######## EMAIL ##############
##############################
##### HOST #####
template Notification "mail-host-notification" {
command = "mail-host-notification"
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_logtosyslog = false
notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
notification_from = "infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
}
period = "24x7"
}
##### SERVICE #####
template Notification "mail-service-notification" {
command = "mail-service-notification"
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
notification_logtosyslog = false
}
period = "24x7"
}
#################################
########### TELEGRAM ############
#################################
##### HOST #####
template Notification "telegram-host-notification" {
command = "telegram-host-notification"
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
notification_logtosyslog = false
}
period = "24x7"
}
##### SERVICE #####
template Notification "telegram-service-notification" {
command = "telegram-service-notification"
states = [ Critical ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_icingaweb2url = "https://mon.mydomain.com/icingaweb2"
notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
notification_logtosyslog = false
}
period = "24x7"
}
Users.conf
##################################
#### default icinga admin ########
##################################
object User "icingaadmin" {
import "generic-user"
display_name = "Icinga 2 Admin"
groups = [ "icingaadmins" ]
email = "user@mydomain.com"
}
object UserGroup "icingaadmins" {
display_name = "Icinga 2 Admin Group"
}
##################################
########## USERS ##############
##################################
object User "mainuser" {
import "generic-user"
display_name = "Users Name"
groups = [ "sysadmins" ]
email = "user@mydomain.com"
}
####### GROUPS #######
object UserGroup "sysadmins" {
display_name = "IT SysAdmins"
}
##################################
########## telegram ############
##################################
object User "telegram-bot" {
import "generic-user"
display_name = "telegram-bot"
}
object User "it-telegram-bot" {
import "generic-user"
display_name = "it-telegram-bot"
}
Notifications.conf
apply Notification "mail-icingaadmin" to Host {
import "mail-host-notification"
user_groups = [ "sysadmins" ]
assign where host.vars.notification.mail
}
apply Notification "mail-icingaadmin" to Service {
import "mail-service-notification"
user_groups = [ "sysadmins" ]
assign where host.vars.notification.mail
}
apply Notification "telegram-icingaadmin" to Host {
import "telegram-host-notification"
user_groups = host.vars.notification.telegram.groups
users = host.vars.notification.telegram.users
assign where host.vars.notification.telegram
}
apply Notification "telegram-icingaadmin" to Service {
import "telegram-service-notification"
user_groups = host.vars.notification.telegram.groups
users = host.vars.notification.telegram.users
assign where host.vars.notification.telegram
}
The notification feature has been enabled.
I hope you can help.
You did not check whether a notification object was created from the apply rule by using object list. I really doubt there is one - hint: The filter doesn’t match. Follow my advice and steps please, I’m not writing them for fun 
-> here
I apologise I am not trying to mess you around. I guess I dont really understand “filiter doesnt match”
Get to know how apply rules work, and how such assign where expressions are modeled.
The above for example only returns true, if the Host object has vars.notification set, with the key mail inside the nested dictionary. If none of your hosts has this, or the specific one you’re expecting a notification for, this expression always returns false. Meaning to say, no notification object can be created.
That is why I’m pointing you towards object list and modifying the filter, with a gentle poke to try and learn. I could also post the solution for copy paste, but that’s boring and won’t help you with further questions.
Cheers,
Michael
okay you have beat me, I have been reading and I am still clueless.
I don’t want to beat you, I’m wondering why you don’t understand how apply rules generate objects.
Write an apply rule like this:
apply Notification "mail-icingaadmin" to Host {
import "mail-host-notification"
user_groups = [ "sysadmins" ]
assign where host.address != ""
}
Now run icinga2 object list --type Notification --name *mail-icingaadmin. Does it return a notification config object?
If yes, try to trigger a notification via Icinga Web.
Next up, modify the assign where expression in the notification apply rule.
Set it to match your host name, by using the match() function for example. Try things out, not just read them.
Cheers,
Michael
It does return an object but when I validate the config I get this
[2019-04-26 15:38:56 +0200] critical/config: Error: Validation failed for object 'mydomain.com!icinga!mail-icingaadmin' of type 'Notification'; Attribute 'states': State filter is invalid.
Location: in /etc/icinga2/conf.d/templates.conf: 38:3-38:23
/etc/icinga2/conf.d/templates.conf(36): command = "mail-host-notification"
/etc/icinga2/conf.d/templates.conf(37):
/etc/icinga2/conf.d/templates.conf(38): states = [ Up, Down ]
^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/conf.d/templates.conf(39): types = [ Problem, Acknowledgement, Recovery, Custom,
/etc/icinga2/conf.d/templates.conf(40): FlappingStart, FlappingEnd,Uhm, never seen this before. Please show me the output of icinga2 --version.
icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.2-1)
System information:
Platform: Debian GNU/Linux
Platform version: 9 (stretch)
Kernel: Linux
Kernel version: 4.9.0-7-amd64
Architecture: x86_64
Build information:
Compiler: GNU 6.3.0
Build host: 486e413fb159
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Please show the content from line 36 with some lines around it - or attach the entire file.
When such a validation error occurs, the host states filter is put into a Service notification, where it is not valid.
template Notification “mail-host-notification” {
command = “mail-host-notification”
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_logtosyslog = false
notification_icingaweb2url = “https://inframon.mydomain.com/icingaweb2”
notification_from = “Infrastructure Icinga2 Monitoring Server inframon@mydomain.com”
}
period = “24x7”
}
template Notification “mail-service-notification” {
command = “mail-service-notification”
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_icingaweb2url = “https://inframon.mydomain.com/icingaweb2”
notification_from = “Infrastructure Icinga2 Monitoring Server inframon@mydomain.com”
notification_logtosyslog = false
}
period = “24x7”
}
That’s not possible …
Ok, then do a recursive grep on here the host notification template is used. I bet that it is included in a notification apply to Service rule.
grep -A 10 -B 10 -r mail-host-notification /etc/icinga2/
/etc/icinga2/conf.d/templates.conf-template User "generic-user" {
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-}
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-##############################
/etc/icinga2/conf.d/templates.conf-######## EMAIL ##############
/etc/icinga2/conf.d/templates.conf-##############################
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf-##### HOST #####
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf:template Notification "mail-host-notification" {
/etc/icinga2/conf.d/templates.conf: command = "mail-host-notification"
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf- states = [ Up, Down ]
/etc/icinga2/conf.d/templates.conf- types = [ Problem, Acknowledgement, Recovery, Custom,
/etc/icinga2/conf.d/templates.conf- FlappingStart, FlappingEnd,
/etc/icinga2/conf.d/templates.conf- DowntimeStart, DowntimeEnd, DowntimeRemoved ]
/etc/icinga2/conf.d/templates.conf-
/etc/icinga2/conf.d/templates.conf- vars += {
/etc/icinga2/conf.d/templates.conf- notification_logtosyslog = false
/etc/icinga2/conf.d/templates.conf- notification_icingaweb2url = "https://inframon.mydomain.com/icingaweb2"
/etc/icinga2/conf.d/templates.conf- notification_from = "infrastructure Icinga2 Monitoring Server <inframon@cognitec.com>"
--
/etc/icinga2/conf.d/notifications.conf-apply Notification "mail-icingaadmin" to Host {
/etc/icinga2/conf.d/notifications.conf: import "mail-host-notification"
/etc/icinga2/conf.d/notifications.conf- user_groups = [ "sysadmins" ]
/etc/icinga2/conf.d/notifications.conf- assign where host.address !=" "
/etc/icinga2/conf.d/notifications.conf-}
/etc/icinga2/conf.d/notifications.conf-
/etc/icinga2/conf.d/notifications.conf-apply Notification "mail-icingaadmin" to Service {
/etc/icinga2/conf.d/notifications.conf: import "mail-host-notification"
/etc/icinga2/conf.d/notifications.conf- user_groups = [ "sysadmins" ]
/etc/icinga2/conf.d/notifications.conf- assign where host.address !=" "
/etc/icinga2/conf.d/notifications.conf-}
/etc/icinga2/conf.d/notifications.conf-
/etc/icinga2/conf.d/notifications.conf-apply Notification "telegram-icingaadmin" to Host {
/etc/icinga2/conf.d/notifications.conf- import "telegram-host-notification"
/etc/icinga2/conf.d/notifications.conf- user_groups = host.vars.notification.telegram.groups
/etc/icinga2/conf.d/notifications.conf- users = host.vars.notification.telegram.users
/etc/icinga2/conf.d/notifications.conf- assign where host.vars.notification.telegram
/etc/icinga2/conf.d/notifications.conf-}
--
/etc/icinga2/conf.d/commands.conf- "-H" = "$host$"
/etc/icinga2/conf.d/commands.conf- "-c" = "$check$"
/etc/icinga2/conf.d/commands.conf- "-t" = "20:3"
/etc/icinga2/conf.d/commands.conf- }
/etc/icinga2/conf.d/commands.conf-}
/etc/icinga2/conf.d/commands.conf-
/etc/icinga2/conf.d/commands.conf-################################
/etc/icinga2/conf.d/commands.conf-#### MAIL NOTIFICATIONS ####
/etc/icinga2/conf.d/commands.conf-################################
/etc/icinga2/conf.d/commands.conf-
/etc/icinga2/conf.d/commands.conf:object NotificationCommand "mail-host-notification" {
/etc/icinga2/conf.d/commands.conf: command = [ SysconfDir + "/icinga2/scripts/mail-host-notification.sh" ]
/etc/icinga2/conf.d/commands.conf-
/etc/icinga2/conf.d/commands.conf- env = {
/etc/icinga2/conf.d/commands.conf- NOTIFICATIONTYPE = "$notification.type$"
/etc/icinga2/conf.d/commands.conf- HOSTALIAS = "$host.display_name$"
/etc/icinga2/conf.d/commands.conf- HOSTADDRESS = "$address$"
/etc/icinga2/conf.d/commands.conf- HOSTSTATE = "$host.state$"
/etc/icinga2/conf.d/commands.conf- LONGDATETIME = "$icinga.long_date_time$"
/etc/icinga2/conf.d/commands.conf- HOSTOUTPUT = "$host.output$"
/etc/icinga2/conf.d/commands.conf- NOTIFICATIONAUTHORNAME = "$notification.author$"
/etc/icinga2/conf.d/commands.conf- NOTIFICATIONCOMMENT = "$notification.comment$"
--
/etc/icinga2/scripts/mail-service-notification.sh- Usage ;;
/etc/icinga2/scripts/mail-service-notification.sh- :) echo "Missing option argument for -$OPTARG" >&2
/etc/icinga2/scripts/mail-service-notification.sh- Usage ;;
/etc/icinga2/scripts/mail-service-notification.sh- *) echo "Unimplemented option: -$OPTARG" >&2
/etc/icinga2/scripts/mail-service-notification.sh- Usage ;;
/etc/icinga2/scripts/mail-service-notification.sh- esac
/etc/icinga2/scripts/mail-service-notification.sh-done
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh-shift $((OPTIND - 1))
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh:## Keep formatting in sync with mail-host-notification.sh
/etc/icinga2/scripts/mail-service-notification.sh-for P in LONGDATETIME HOSTNAME HOSTDISPLAYNAME SERVICENAME SERVICEDISPLAYNAME SERVICEOUTPUT SERVICESTATE USEREMAIL NOTIFICATIONTYPE ; do
/etc/icinga2/scripts/mail-service-notification.sh- eval "PAR=\$${P}"
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh- if [ ! "$PAR" ] ; then
/etc/icinga2/scripts/mail-service-notification.sh- Error "Required parameter '$P' is missing."
/etc/icinga2/scripts/mail-service-notification.sh- fi
/etc/icinga2/scripts/mail-service-notification.sh-done
/etc/icinga2/scripts/mail-service-notification.sh-
/etc/icinga2/scripts/mail-service-notification.sh-## Build the message's subject
/etc/icinga2/scripts/mail-service-notification.sh-SUBJECT="[$NOTIFICATIONTYPE] $SERVICEDISPLAYNAME on $HOSTDISPLAYNAME is $SERVICESTATE!"
--
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-template User "generic-user" {
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-}
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak-/**
/etc/icinga2/templates.conf.bak- * Provides default settings for host notifications.
/etc/icinga2/templates.conf.bak- * By convention all host notifications should import
/etc/icinga2/templates.conf.bak- * this template.
/etc/icinga2/templates.conf.bak- */
/etc/icinga2/templates.conf.bak:template Notification "mail-host-notification" {
/etc/icinga2/templates.conf.bak: command = "mail-host-notification"
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak- states = [ Up, Down ]
/etc/icinga2/templates.conf.bak- types = [ Problem, Acknowledgement, Recovery, Custom,
/etc/icinga2/templates.conf.bak- FlappingStart, FlappingEnd,
/etc/icinga2/templates.conf.bak- DowntimeStart, DowntimeEnd, DowntimeRemoved ]
/etc/icinga2/templates.conf.bak-
/etc/icinga2/templates.conf.bak- vars += {
/etc/icinga2/templates.conf.bak- // notification_icingaweb2url = "https://www.example.com/icingaweb2"
/etc/icinga2/templates.conf.bak- // notification_from = "Icinga 2 Host Monitoring <icinga@example.com>"
/etc/icinga2/templates.conf.bak- notification_logtosyslog = false
Ok, here we go.
/etc/icinga2/conf.d/notifications.conf-apply Notification "mail-icingaadmin" to Service {
/etc/icinga2/conf.d/notifications.conf: import "mail-host-notification"
/etc/icinga2/conf.d/notifications.conf- user_groups = [ "sysadmins" ]
/etc/icinga2/conf.d/notifications.conf- assign where host.address !=" "
/etc/icinga2/conf.d/notifications.conf-}
A notification apply rule has a target. If you say to Service, you want to create notification objects for service objects. Within a service notification, the specific service states are valid filters. Meaning to say, when you want to receive a notification for a service turning Critical, you’ll set that into the states filters for the service notification.
On the other hand, to Host creates host notification objects, which serve the purpose of alerting you when a host goes Down for example. If you want to receive this state per notification, add it to the states filter.
Now comes the problem: You need to import the correct template which sets specific attributes in this scope, there isn’t a generic one. That is why the example configuration uses two Notification templates:
mail-host-notification for Host apply rulesmail-service-notification for Service apply rulesIn your configuration, you’re importing the host notification template into the service notification apply rule. The config compiler is correct about this, the states attribute is not valid in this scope.
The fully constructed object, without the template looks like this in memory:
apply Notification "mail-icingaadmin" to Service {
// import "mail-host-notification" //template source
user_groups = [ "sysadmins" ]
//from the imported template
command = “mail-host-notification”
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
notification_logtosyslog = false
notification_icingaweb2url = “https://inframon.mydomain.com/icingaweb2”
notification_from = “Infrastructure Icinga2 Monitoring Server inframon@mydomain.com”
}
period = “24x7”
//from imported template, end
assign where host.address != ""
}
For learning purposes, if you add this somewhere and let the config validation run, it will correctly tell you again, that in this context, the state filter is invalid.
The fix is easy - import the correct template, you’ve made a copy paste error. Understanding why, that is key here.
Cheers,
Michael