Notifications not working with director

Hi Guys,

I’m new on this and I was able to install icinga2 with director and host been monitoring OK.

I configured everything based on documentation using Director.
I was able to send test emails from shell using mail command (mailutils + postfix)
I tried to send a test from custom notifications from director force it to send the email and same… not working.

I ran debug/notice level and found this:

[2020-05-04 23:10:01 +0000] notice/ExternalCommandProcessor: Sending custom notification for host fileserver
[2020-05-04 23:10:01 +0000] notice/Checkable: Skipping checkable ‘fileserver’ which doesn’t have any notification objects configured.

hostname: fileserver

I want to receive notifications at host level when host is down.
Can you please help me to resolve this?

icinga2 --version output:
https://pastebin.com/JVexytHy

notification_templates.conf
template Notification “notif_templ” {
command = “host notification”
interval = 2m
period = “time_period”
states = [ Down, Up ]
types = [ Acknowledgement, Problem, Recovery ]
users = [ “Admins” ]
}

#####################################

notification_apply.conf
apply Notification “HostDown” to Host {
import “notif_templ”

assign where host.check_timeout == "3"
users = [ "Admins" ]

}

############################

/users.conf
object User “Admins” {
import “user_templ”

display_name = "Admin"
email = "my_email@gmail.com"
states = [ Down ]

}

###############################

Any help would be appreciated.

Thanks,

Please share the output of

icinga2 --version
‘icinga2 feature list’

as well as attach a full config example (host, notification, users, notification command objects)

To check whether your apply rules are good

icinga2 object list --type=notification

It should return your list of objects where the notification is applied. Mostly the problem lies with apply rules

Take a look at this as well

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.11.3-1)

Copyright © 2012-2020 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Ubuntu
Platform version: 18.04.3 LTS (Bionic Beaver)
Kernel: Linux
Kernel version: 5.3.0-1017-aws
Architecture: x86_64

Build information:
Compiler: GNU 8.3.0
Build host: runner-LTrJQZ9N-project-298-concurrent-0

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

####################################

Disabled features: compatlog elasticsearch gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command debuglog ido-mysql mainlog notification

###################################

all this is under: /var/lib/icinga2/api/zones

conf.d is excluded.

###########################
hosts.conf

object Host “fileserver” {
import “Win_Servers_Templ”

display_name = "fileserver"
address = "172.31.10.100"

}

##############################

host_templates.conf
template Host “Win_Servers_Templ” {
check_command = “hostalive”
check_period = “time_period”
enable_notifications = true
}

##############################

/agent_endpoints.conf
object Endpoint “fileserver” {
host = “172.31.10.100”
log_duration = 0s
}

##############################

user_templates.conf
template User “user_templ” {
enable_notifications = true
period = “time_period”
states = [ Down ]
}

##############################

timeperiods.conf
object TimePeriod “time_period” {
import “legacy-timeperiod”
import “Always_timeperiod”

display_name = "time_period"
ranges = {
}

}

##############################

/timeperiod_templates.conf
template TimePeriod “Always_timeperiod” {
import “legacy-timeperiod”
display_name = “All days monitoring”
ranges = {
“friday” = “00:00-24:00”
“monday” = “00:00-24:00”
“thursday” = “00:00-24:00”
“tuesday” = “00:00-24:00”
“wednesday” = “00:00-24:00”
}
}

#######################################

/commands.conf
object NotificationCommand “host notification” {
import “plugin-notification-command”
import “mail-host-notification_template”
}

#######################################

/notification_apply.conf
apply Notification “HostDown” to Host {
import “notif_templ”

assign where host.check_timeout == "3"
users = [ "Admins" ]

}

#############################

notification_templates.conf
template Notification “notif_templ” {
command = “host notification”
interval = 2m
period = “time_period”
states = [ Down, Up ]
types = [ Acknowledgement, Problem, Recovery ]
users = [ “Admins” ]
}

I didn’t get nothing.

Your Apply Rule is probably incorrect.
Start Simple and use the above command to see

Your Apply Rule is probably incorrect.
Start Simple and use the above command to see

you mean to this?

Assign where rule “host.check_timeout = 3”
Probably start with something simple

I did, and I started with ip check and when I try to test the notification (manual) it says it was sent on history notifications but I never got the email.

I installed mailutils package and I was able to send a test email from shell using mail command.
I set the mail command path on the mail-host-notification.sh and same… is not working.

I dont know what else to do.
This is very frustrating

In the script write a line to log locally whether it is being called
Please post output of

icinga2 object list --type=notification

Well I had moved forward a little bit on this:

Now I’m having issues with the final step on the email sent.
############################################

is like is getting a wrong $variable email address variable


[2020-05-13 01:33:26 +0000] warning/PluginNotificationTask: Notification command for object 'fileserver' (PID: 8659, arguments: '/etc/icinga2/scripts/mail-host-notification.sh' '-b' '' '-d' '2020-05-13 01:33:25 +0000' '-l' 'fileserver' '-n' 'fileserver' '-o' 'PING CRITICAL - Packet loss = 100%' '-r' 'yyyyyyyy@xxxxxxx.com' '-s' 'DOWN' '-t' 'PROBLEM') terminated with exit code 1, output: ssmtp: RCPT TO:<[PROBLEM] Host fileserver is DOWN!@gmail.com> (553 5.1.3 is not a valid RFC-5321 address. 14sm13098553pfy.38 - gsmtp)



root@icinga2:/home/ubuntu# cat /etc/ssmtp/ssmtp.conf
#
# Config file for sSMTP sendmail
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=xxxxxxxx@gmail.com

# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
mailhub=smtp.gmail.com:587
AuthUser=xxxxxxxx@gmail.com
AuthPass=xxxxxxxxxxxxxxxx
UseTLS=YES
UseSTARTTLS=YES

# Where will the mail seem to come from?
rewriteDomain=gmail.com

# The full hostname
hostname=xxxxxxxx45.us-west-1.compute.amazonaws.com

# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: address
FromLineOverride=YES
Debug=YES


root@icinga2:/home/ubuntu# cat /etc/ssmtp/revaliases
# sSMTP aliases
#
# Format:       local_account:outgoing_address:mailhub
#
# Example: root:your_login@your.domain:mailhub.your.domain[:port]
# where [:port] is an optional port number that defaults to 25.
root:xxxxxxxxx@gmail.com:smtp.gmail.com:587



SSMTP LOG:


May 13 01:53:27 icinga2 sSMTP[9012]: Creating SSL connection to host
May 13 01:53:27 icinga2 sSMTP[9012]: 220 smtp.gmail.com ESMTP d18sm13139547pfq.177 - gsmtp
May 13 01:53:27 icinga2 sSMTP[9012]: EHLO ecXXXXXXX-west-1.compute.amazonaws.com
May 13 01:53:27 icinga2 sSMTP[9012]: 250 SMTPUTF8
May 13 01:53:27 icinga2 sSMTP[9012]: STARTTLS
May 13 01:53:27 icinga2 sSMTP[9012]: 220 2.0.0 Ready to start TLS
May 13 01:53:28 icinga2 sSMTP[9012]: SSL connection using ECDHE_ECDSA_CHACHA20_POLY1305
May 13 01:53:28 icinga2 sSMTP[9012]: EHLO eXXXXXXXX45.us-west-1.compute.amazonaws.com
May 13 01:53:28 icinga2 sSMTP[9012]: 250 SMTPUTF8
May 13 01:53:28 icinga2 sSMTP[9012]: AUTH LOGIN
May 13 01:53:28 icinga2 sSMTP[9012]: 334 XXXXX
May 13 01:53:28 icinga2 sSMTP[9012]: XXXXXXXXXX=
May 13 01:53:28 icinga2 sSMTP[9012]: 334 XXXXXXXX
May 13 01:53:28 icinga2 sSMTP[9012]: XXXXXXXXXX==
May 13 01:53:28 icinga2 sSMTP[9012]: 235 2.7.0 Accepted
May 13 01:53:28 icinga2 sSMTP[9012]: MAIL FROM:<nagios@XXXXXXXX.us-west-1.compute.amazonaws.com>
May 13 01:53:28 icinga2 sSMTP[9012]: 250 2.1.0 OK d18sm13139547pfq.177 - gsmtp
May 13 01:53:28 icinga2 sSMTP[9012]: RCPT TO:<[PROBLEM] Host fileserver is DOWN!@eXXXXXX136-45.us-west-1.compute.amazonaws.com>
May 13 01:53:28 icinga2 sSMTP[9012]: 553 5.1.3 address. d18sm13139547pfq.177 - gsmtp
May 13 01:53:28 icinga2 sSMTP[9012]: RCPT TO:<[PROBLEM] Host fileserver is DOWN!@ecXXXXXXXX136-45.us-west-1.compute.amazonaws.com> (553 5.1.3 address. d18sm13139547pfq.177 - gsmtp)

ssmtp works fine I tested it with the same .conf and it was able to send emails out.

Thanks in advance guys.

Not sure - but have you defined the user who is suppose to receive the email. You can do it using director

image

And in the Apply Rules

This is already created. As you can see here there was an attempt to sent the email

[2020-05-13 01:33:26 +0000] warning/PluginNotificationTask: Notification command for object ‘fileserver’ (PID: 8659, arguments: ‘/etc/icinga2/scripts/mail-host-notification.sh’ ‘-b’ ‘’ ‘-d’ ‘2020-05-13 01:33:25 +0000’ ‘-l’ ‘fileserver’ ‘-n’ ‘fileserver’ ‘-o’ ‘PING CRITICAL - Packet loss = 100%’ ‘-r’ ‘yyyyyyyy@xxxxxxx.com’ ‘-s’ ‘DOWN’ ‘-t’ ‘PROBLEM’) terminated with exit code 1, output: ssmtp: RCPT TO:<[PROBLEM] Host fileserver is DOWN!@gmail.com> (553 5.1.3 is not a valid RFC-5321 address. 14sm13098553pfy.38 - gsmtp)

image

Can you see if the above command can be run manually

Strangely ‘-r’ is suppose to be recipient email address is being replaced by the actual problem description. Can you replace the ssmtp to postfix - I think something is screwed in your ssmtp.conf

Resolved.

Using postfix (mail command path instead posftix on .sh) resolved this. Now I’m getting notifications.

Thank you for all amazin support!!

1 Like

Glad to hear. Thank for your patience :slight_smile: