Email Notification

Hi,

I’ve been playing around with IcingaWeb2 in the last couple of days.
I want to insert an email notification system where I get emails when there is something wrong.

I have struggled to do that though.

Here are some snippets from my config files.

/etc/icinga2/conf.d/notifications.conf

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = [ "icingaadmins" ]
  users = host.vars.notification.mail.users
  assign where host.address != " users = host.vars.notification.mail.users"
}

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = [ "icingaadmins" ]
  users = host.vars.notification.mail.users
  assign where host.address != ""
}        

/etc/icinga2/conf.d/hosts.conf

object Host "svr_testblade" {
        import "generic-host"
        address = "123.456.123.456"
        check_command = "hostalive"
        vars.os = "Servers"

        vars.notification["mail"] = {
                groups = ["icingaadmins"]
                users = ["icingaadmin"]

/etc/icinga2/conf.d/users.conf

object User "icingaadmin" {
  import "generic-user"

  display_name = "Icinga2 Admin"
  groups = [ "icingaadmins" ]

  email = "myemail@domain.com"
}
object UserGroup "icingaadmins" {
  display_name = "Icinga2 Admin Group"
}

/etc/icinga2/scripts/mail-host-notification.sh

## Send the mail using the $MAILBIN command.
## If an explicit sender was specified, try to set it.
if [ -n "$MAILFROM" ] ; then

  ## Modify this for your own needs!

  ## Debian/Ubuntu use mailutils which requires `-a` to append the header
  if [ -f /etc/debian_version ]; then
     /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -a "From: $MAILFROM" -s "$SUBJECT" $USEREMAIL

     ## Other distributions (RHEL/SUSE/etc.) prefer mailx which sets a sender address with `-r`
  else

  ##  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "$MAILFROM" -s "$SUBJECT" $USEREMAIL
       /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "icinga2@domain.com" -s "$SUBJECT" $USERMAIL
  fi

else
  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" \                                                                                            | $MAILBIN -s "$SUBJECT" $USEREMAIL
fi

template=`cat <<TEMPLATE
***** Icinga EPAY *****
Notification Type: $NOTIFICATIONTYPE
Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $HOSTSTATE
Date/Time: $LONGDATETIME
Additional Info: $HOSTOUTPUT
Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
`
echo "TEST" | msmtp=myemail@domain.com -s "$NOTIFICATIONTYPE - $HOSTDISPLAYNAME is $HOSTSTATE" $USEREMAIL
#!/bin/sh

Here ist something from the icinga2 log whenever I want to send a forced custom notification.

    [2020-06-04 13:59:44 +0200] information/Notification: Sending 'Custom' notification 'svr_testblade!mail-icingaadmin' for user 'icingaadmin'
    [2020-06-04 13:59:44 +0200] information/Notification: Completed sending 'Custom' notification 'svr_testblade!mail-icingaadmin' for checkable 'svr_testblade' and user 'icingaadmin'.
    [2020-06-04 13:59:44 +0200] information/HttpServerConnection: HTTP client disconnected (from [127.0.0.1]:55712)
    [2020-06-04 14:00:44 +0200] warning/Process: Killing process group 24605 ('/etc/icinga2/scripts/mail-host-notification.sh' '-4' '123.456.123.456' '-6' '' '-b' 'bak0523' '-c' 'Test123' '-d' '2020-06-04 13:59:44 +0200' '-l' 'svr_testblade' '-n' 'svr_testblade' '-o' 'PING OK - Packet loss = 0%, RTA = 0.85 ms' '-r' 'gabriel.laszlo@bk-group.eu' '-s' 'UP' '-t' 'CUSTOM' '-v' 'false') after timeout of 60 seconds
    [2020-06-04 14:00:44 +0200] warning/Process: PID 24605 was terminated by signal 9 (Killed)
    [2020-06-04 14:00:44 +0200] warning/PluginNotificationTask: Notification command for object 'svr_testblade' (PID: 24605, arguments: '/etc/icinga2/scripts/mail-host-notification.sh' '-4' '123.456.123.456' '-6' '' '-b' 'bak0523' '-c' 'Test123' '-d' '2020-06-04 13:59:44 +0200' '-l' 'svr_testblade' '-n' 'svr_testblade' '-o' 'PING OK - Packet loss = 0%, RTA = 0.85 ms' '-r' 'myemail@domain.com' '-s' 'UP' '-t' 'CUSTOM' '-v' 'false') terminated with exit code 128, output: <Timeout exceeded.><Terminated by signal 9 (Killed).>
    [2020-06-04 14:00:52 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 1, rate: 3.45/s (207/min 22972/5min 25138/15min);

I did not recive any email. After I tried this a couple of times, i got a mail but from “nagios@monitoring” which I never defined anywhere and i even searched all the configs for nagios and didn’t find any script that adds nagios@ to my Hostname (monitoring).

I would appriciate if someone could help me with this.

Kind regards,
Gabriel Laszlo

Hi,

in the Script /etc/icinga2/scripts/mail-host-notification.sh is defined a variable $MAILBIN. What is the value of this and is this program installed?
If yes? is icinga allowed to call this program?

1 Like

Hi Stevie,

These are the default configurations that were set from Icinga so therefore i did not change any variable.

I do not recall having installed a MAILBIN neither do I find any MAILBIN package. The closest being Mailman.

I did install a mSMTP. With that I can send emails within the terminal.

Thank you for your reply.
Kind regards,
Gabriel

So in this case you should put the path of your installed mail programm to the variable MAILBIN. You can also use another script. We for example are using a python script not the installed bash script. Or you call the mail program directly. Chosse that way you prefer :wink:

Hi, sending mail is working now, but when sending notifications icinga always throws a timout error in the log:

[2020-06-09 14:39:12 +0200] warning/Process: Killing process group 853 (’/etc/icinga2/scripts/mail-service-notification.sh’ ‘-4’ ‘127.0.0.1’ ‘-6’ ‘::1’ ‘-b’ ‘user’ ‘-c’ ‘55555’ ‘-d’ ‘2020-06-09 14:38:12 +0200’ ‘-e’ ‘apt’ ‘-l’ ‘monitoring’ ‘-n’ ‘monitoring’ ‘-o’ 'APT CRITICAL: 2 packages available for upgrade (1 critical updates). ’ ‘-r’ ‘user@domain.com’ ‘-s’ ‘CRITICAL’ ‘-t’ ‘CUSTOM’ ‘-u’ ‘apt’) after timeout of 60 seconds
[2020-06-09 14:39:12 +0200] warning/Process: PID 853 was terminated by signal 9 (Killed)

I configured the php.ini and sendmail timouts to more than 60s but in the log it still says killed after 60s. Do you know where this 60s timeout value comes from? When i run the command as it says in the log directly from the commmandline

(’/etc/icinga2/scripts/mail-service-notification.sh’ ‘-4’ ‘127.0.0.1’ ‘-6’ ‘::1’ ‘-b’ ‘user’ ‘-c’ ‘55555’ ‘-d’ ‘2020-06-09 14:38:12 +0200’ ‘-e’ ‘apt’ ‘-l’ ‘monitoring’ ‘-n’ ‘monitoring’ ‘-o’ 'APT CRITICAL: 2 packages available for upgrade (1 critical updates). ’ ‘-r’ ‘user@domain.com’ ‘-s’ ‘CRITICAL’ ‘-t’ ‘CUSTOM’ ‘-u’ ‘apt’)

,the mail gets delivered…

To debug this issue you should try to start the script manually with the icinga user.

Hi,
i ran the command with “sudo -u nagios” and it worked, the mail was delivered and no 60s timeout. So i think the problem is within the execution of the webserver or something like that. But i cant find the 60s timeout value in any configuration file…

For the moment I’m out of ideas. Maybe you should take a look into the debug.log from icinga. Maybe you’ll see there the reason of the delay.

The timeout likely comes from Icinga 2 itself, i.e. NotificationCommand#timeout.