Notifications not sending correctly

Hello,

Currently I am stuck and not sure why I am not getting emails sent to me correctly. I was asked to add some email notifications for some alerts and hosts and logs say they are being sent but I don’t see any emails coming through. Currently my test is limited to a single host notification. I am able to bring the host up and down as needed and have seen logs that messages were being sent with a completed message in the log.

Please see details of my icinga instance, I am running on centos7.5

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.5-1)

I did enable debug and received the following output, I am manually trigger the notification through the web GUI.

[2019-10-15 02:16:44 +0000] notice/Notification: Attempting to send notifications for notification object ‘testhost!mail-icingaadmin’.
[2019-10-15 02:16:44 +0000] notice/Notification: Not checking notification filters for notification object ‘testhost!mail-icingaadmin’: Notification was forced.
[2019-10-15 02:16:44 +0000] notice/Notification: Not checking notification filters for notification object ‘testhost!mail-icingaadmin’ and user ‘icingaadmin’: Notification was forced.
[2019-10-15 02:16:44 +0000] information/Notification: Sending ‘Custom’ notification ‘testhost!mail-icingaadmin’ for user ‘icingaadmin’
[2019-10-15 02:16:44 +0000] debug/DbEvents: add notification history for ‘testhost’
[2019-10-15 02:16:44 +0000] debug/DbEvents: add contact notification history for service ‘testhost’ and user ‘icingaadmin’.
[2019-10-15 02:16:44 +0000] notice/Process: Running command ‘/etc/icinga2/scripts/mail-host-notification.sh’ ‘-4’ ‘testhost’ ‘-6’ ‘’ ‘-b’ ‘zareh’ ‘-c’ ‘testsetst’ ‘-d’ ‘2019-10-15 02:16:44 +0000’ ‘-l’ ‘testhost’ ‘-n’ ‘testhost’ ‘-o’ ‘PING OK - Packet loss = 0%, RTA = 0.26 ms’ ‘-r’ ‘icinga@localhost’ ‘-s’ ‘UP’ ‘-t’ ‘CUSTOM’ ‘-v’ ‘false’: PID 21002
[2019-10-15 02:16:44 +0000] information/Notification: Completed sending ‘Custom’ notification ‘testhost!mail-icingaadmin’ for checkable ‘testhost’ and user ‘icingaadmin’.
[2019-10-15 02:16:44 +0000] notice/Process: PID 21002 (’/etc/icinga2/scripts/mail-host-notification.sh’ ‘-4’ ‘testhost’ ‘-6’ ‘’ ‘-b’ ‘zareeh’ ‘-c’ ‘testsetst’ ‘-d’ ‘2019-10-15 02:16:44 +0000’ ‘-l’ ‘testhost’ ‘-n’ ‘testhost’ ‘-o’ ‘PING OK - Packet loss = 0%, RTA = 0.26 ms’ ‘-r’ ‘icinga@localhost’ ‘-s’ ‘UP’ ‘-t’ ‘CUSTOM’ ‘-v’ ‘false’) terminated with exit code 0

I am able to receive emails when I run the script manually as below

/etc/icinga2/scripts/mail-host-notification.sh -d 10.10.10 -l test -n test -o test -r change@me.com -s FAIL -t problem

I am using the below apply statement with the following host configuration

apply Notification “mail-icingaadmin” to Host {
import “mail-host-notification”
user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users
//interval = 2h
vars.notification_logtosyslog = true
assign where host.vars.notification.mail
}

object Host “testhost” {
import “generic-host”
address = “testhost”
vars.notification[“mail”] = {
groups = [ “icingaadmins” ]
}
}

My group configuration looks like

object User “icingaadmin” {
import “generic-user”
enable_notifications = true
display_name = “Icinga 2 Admin”
groups = [ “icingaadmins” ]
email = “change@me.com
}
object UserGroup “icingaadmins” {
display_name = “Icinga 2 Admin Group”
}

I feel like I am missing something very obvious, please let me know if any other information would be helpful. I am using postfix to send mail and echo "Test body" | mail -s "Subject" change@me.com also works without issue. I checked our mailserver and do not see any messages coming from the server that are not manual test messages either and nothing getting sent to junk.

Thank you.

What state and type filters are configured for the notification and the user?

1 Like

My user configuration is above along with my group. I am trying to use as many defaults as possible. My host notification has the following configuration

template Notification “mail-host-notification” {
command = “mail-host-notification”
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
vars += {
// notification_icingaweb2url = “https://www.example.com/icingaweb2
// notification_from = “Icinga 2 Host Monitoring icinga@example.com
notification_logtosyslog = true
}
period = “24x7”
}

And my notficiation object looks like

object 'testhost!mail-icingaadmin' of type 'Notification':
  % declared in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 11:1-11:45
  * __name = "testhost!mail-icingaadmin"
  * command = "mail-host-notification"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 47:3-47:36
  * command_endpoint = ""
  * host_name = "testhost"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 11:1-11:45
  * interval = 1800
  * name = "mail-icingaadmin"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 11:1-11:45
  * period = "24x7"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 60:3-60:17
  * service_name = ""
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 45
    * last_line = 11
    * path = "/etc/icinga2/zones.d/global-templates/notifications.conf"
  * states = [ "Up", "Down" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 49:3-49:23
  * templates = [ "mail-icingaadmin", "mail-host-notification" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 11:1-11:45
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 46:1-46:46
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd", "DowntimeStart", "DowntimeEnd", "DowntimeRemoved" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 50:3-52:57
  * user_groups = [ "icingaadmins" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 13:3-13:50
  * users = null
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 14:3-14:43
  * vars
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 54:3-58:3
    * notification_logtosyslog = false
  * zone = "master"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 11:1-11:45

And it looks like I have been editing files in the wrong place, above shows that all my files are at global-tempaltes and i have been editing stuff in conf.d. Will update the correct files and update here.

Edit

As stated above the error was I was adding my user details to /etc/icinga2/conf.d/user.conf instead of /etc/icinga2/global-templates/user.conf. Sadly it was a really simple thing I totally missed when I was looking at my notification objects.