Modify contents of Icinga2 Alert Emails

tzwarycz · February 4, 2019, 6:15pm

Hello,

When Icinga2 sends a host or service alert, we get the following garbage text before seeing the information we need:

Usage:
icinga2 <command> [<arguments>]

Supported commands:

api setup (setup for API)

ca list (lists all certificate signing requests)

ca sign (signs an outstanding certificate request)

console (Icinga console)

daemon (starts Icinga 2)

feature disable (disables specified feature)

feature enable (enables specified feature)

feature list (lists all available features)

node setup (set up node)

node wizard (wizard for node setup)

object list (lists all objects)

pki new-ca (sets up a new CA)

pki new-cert (creates a new CSR)

pki request (requests a certificate)

pki save-cert (saves another Icinga 2 instance’s certificate)

pki sign-csr (signs a CSR)

pki ticket (generates a ticket)

troubleshoot (collect information for troubleshooting)

variable get (gets a variable)

variable list (lists all variables)

Global options:
-h [ --help ] show this help message
-V [ --version ] show version information
–color use VT100 color codes even when stdout is not a
terminal
-D [ --define ] arg define a constant
-I [ --include ] arg add include search directory
-x [ --log-level ] arg specify the log level for the console log.
The valid value is either debug, notice,
information (default), warning, or critical
-X [ --script-debugger ] whether to enable the script debugger

Report bugs at LINK
Get support: LINK
Documentation: LINK
Icinga home page: LINK

Is there a way we can remove the unneeded text in our email alerts? I’ve tried to look in all the configs that I know of but cannot see a way. Any help is greatly appreciated.

dnsmichi · February 4, 2019, 6:29pm

Hi,

could you please provide a little more context to your problem?

Something like:

Which Icinga 2 version
Notification script used here
Notification command and notification apply rule configuration from Icinga

From a quick guess, your script calls the “icinga2” binary without any CLI parameters. Might be an oversight with backticks shell execution.

blabla `icinga2` blablatext

Cheers,
Michael

tzwarycz · February 4, 2019, 6:51pm

Hi,

The Icinga 2 network monitoring daemon (version: r2.10.1-1).
We are using this mail-host-notification.sh script:

    > #!/usr/bin/env bash
    > #
    > # Copyright (C) 2012-2018 Icinga Development Team (LINK)
    > # Except of function urlencode which is Copyright (C) by Brian White (brian@aljex.com) used under MIT license
    > 
    > PROG="`basename $0`"
    > ICINGA2HOST="`hostname`"
    > MAILBIN="mail"
    > 
    > if [ -z "`which $MAILBIN`" ] ; then
    >   echo "$MAILBIN not found in \$PATH. Consider installing it."
    >   exit 1
    > fi
    > 
    > ## Function helpers
    > Usage() {
    > cat << EOF
    > 
    > Required parameters:
    >   -d LONGDATETIME (\$icinga.long_date_time\$)
    >   -l HOSTNAME (\$host.name\$)
    >   -n HOSTDISPLAYNAME (\$host.display_name\$)
    >   -o HOSTOUTPUT (\$host.output\$)
    >   -r USEREMAIL (\$user.email\$)
    >   -s HOSTSTATE (\$host.state\$)
    >   -t NOTIFICATIONTYPE (\$notification.type\$)
    > 
    > Optional parameters:
    >   -4 HOSTADDRESS (\$address\$)
    >   -6 HOSTADDRESS6 (\$address6\$)
    >   -b NOTIFICATIONAUTHORNAME (\$notification.author\$)
    >   -c NOTIFICATIONCOMMENT (\$notification.comment\$)
    >   -i ICINGAWEB2URL (\$notification_icingaweb2url\$, Default: unset)
    >   -f MAILFROM (\$notification_mailfrom\$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
    >   -v (\$notification_sendtosyslog\$, Default: false)
    > 
    > EOF
    > }
    > 
    > Help() {
    >   Usage;
    >   exit 0;
    > }
    > 
    > Error() {
    >   if [ "$1" ]; then
    >     echo $1
    >   fi
    >   Usage;
    >   exit 1;
    > }
    > urlencode() {
    >   local LANG=C i c e=''
    >   for ((i=0;i<${#1};i++)); do
    >     c=${1:$i:1}
    >     [[ "$c" =~ [a-zA-Z0-9\.\~\_\-] ]] || printf -v c '%%%02X' "'$c"
    >     e+="$c"
    >   done
    >   echo "$e"
    > }
    > 
    > ## Main
    > while getopts 4:6::b:c:d:f:hi:l:n:o:r:s:t:v: opt
    > do
    >   case "$opt" in
    >     4) HOSTADDRESS=$OPTARG ;;
    >     6) HOSTADDRESS6=$OPTARG ;;
    >     b) NOTIFICATIONAUTHORNAME=$OPTARG ;;
    >     c) NOTIFICATIONCOMMENT=$OPTARG ;;
    >     d) LONGDATETIME=$OPTARG ;; # required
    >     f) MAILFROM=$OPTARG ;;
    >     h) Help ;;
    >     i) ICINGAWEB2URL=$OPTARG ;;
    >     l) HOSTNAME=$OPTARG ;; # required
    >     n) HOSTDISPLAYNAME=$OPTARG ;; # required
    >     o) HOSTOUTPUT=$OPTARG ;; # required
    >     r) USEREMAIL=$OPTARG ;; # required
    >     s) HOSTSTATE=$OPTARG ;; # required
    >     t) NOTIFICATIONTYPE=$OPTARG ;; # required
    >     v) VERBOSE=$OPTARG ;;
    >    \?) echo "ERROR: Invalid option -$OPTARG" >&2
    >        Error ;;
    >     :) echo "Missing option argument for -$OPTARG" >&2
    >        Error ;;
    >     *) echo "Unimplemented option: -$OPTARG" >&2
    >        Error ;;
    >   esac
    > done
    > 
    > shift $((OPTIND - 1))
    > 
    > ## Keep formatting in sync with mail-service-notification.sh
    > for P in LONGDATETIME HOSTNAME HOSTDISPLAYNAME HOSTOUTPUT HOSTSTATE USEREMAIL NOTIFICATIONTYPE ; do
    >         eval "PAR=\$${P}"
    > 
    >         if [ ! "$PAR" ] ; then
    >                 Error "Required parameter '$P' is missing."
    >         fi
    > done
    > ## Build the message's subject
    > SUBJECT="[$NOTIFICATIONTYPE] Host $HOSTDISPLAYNAME is $HOSTSTATE!"
    > 
    > ## Build the notification message
    > NOTIFICATION_MESSAGE=`cat << EOF
    > ***** Host Monitoring on $ICINGA2HOST *****
    > 
    > $HOSTDISPLAYNAME is $HOSTSTATE!
    > 
    > Info:    $HOSTOUTPUT
    > 
    > When:    $LONGDATETIME
    > Host:    $HOSTNAME
    > EOF
    > `
    > 
    > ## Check whether IPv4 was specified.
    > if [ -n "$HOSTADDRESS" ] ; then
    >   NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
    > IPv4:    $HOSTADDRESS"
    > fi
    > 
    > ## Check whether IPv6 was specified.
    > if [ -n "$HOSTADDRESS6" ] ; then
    >   NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
    > IPv6:    $HOSTADDRESS6"
    > fi
    > 
    > ## Check whether author and comment was specified.
    > if [ -n "$NOTIFICATIONCOMMENT" ] ; then
    >   NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
    > 
    > Comment by $NOTIFICATIONAUTHORNAME:
    >   $NOTIFICATIONCOMMENT"
    > fi
    > 
    > ## Check whether Icinga Web 2 URL was specified.
    > if [ -n "$ICINGAWEB2URL" ] ; then
    >   NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
    > 
    > $ICINGAWEB2URL/monitoring/host/show?host=$(urlencode "$HOSTNAME")"
    > fi
    > 
    > ## Check whether verbose mode was enabled and log to syslog.
    > if [ "$VERBOSE" == "true" ] ; then
    >   logger "$PROG sends $SUBJECT => $USEREMAIL"
    > fi
    > ## Send the mail using the $MAILBIN command.
    > ## If an explicit sender was specified, try to set it.
    > if [ -n "$MAILFROM" ] ; then
    > 
    >   ## Modify this for your own needs!
    > 
    >   ## Debian/Ubuntu use mailutils which requires `-a` to append the header
    >   if [ -f /etc/debian_version ]; then
    >     /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -a "From: $MAILFROM" -s "$SUBJECT" $USEREMAIL
    >   ## Other distributions (RHEL/SUSE/etc.) prefer mailx which sets a sender address with `-r`
    >   else
    >     /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "$MAILFROM" -s "$SUBJECT" $USEREMAIL
    >   fi
    > 
    > else
    >   /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" \
    >   | $MAILBIN -s "$SUBJECT" $USEREMAIL
    > fi

- Notification apply rule:

> /**
>  * The example notification apply rules.
>  *
>  * Only applied if host/service objects have
>  * the custom attribute `notification` defined
>  * and containing `mail` as key.
>  *
>  * Check `hosts.conf` for an example.
>  */
> 
> apply Notification "mail-icingaadmin" to Host {
>   import "mail-host-notification"
>   user_groups = host.vars.notification.mail.groups
>   users = host.vars.notification.mail.users
> 
>   interval = 1h
>   times.begin = 5m
> 
>   vars.notification_logtosyslog = true
> 
>   assign where host.vars.notification.mail
> }
> 
> apply Notification "mail-icingaadmin" to Service {
>   import "mail-service-notification"
>   user_groups = host.vars.notification.mail.groups
>   users = host.vars.notification.mail.users
> 
>   interval = 1h
>   times.begin = 5m
> 
>   vars.notification_logtosyslog = true
> 
>   assign where host.vars.notification.mail
> }

Hope this is sufficient information. My apologies I am very new to this system.

Thanks again.

dnsmichi · February 4, 2019, 7:31pm

Please encapsulate code, snippets, logs, config within three backticks for better readability. I’ve edited your post once to make it a bit more readable - please edit it again.

More tips here: Create topics and master Markdown formatting

What exactly did you modify in that script? Also, did you modify the NotificationCommand object too?

Cheers,
Michael

tzwarycz · February 4, 2019, 9:38pm

Nothing modified, no NotificationCommand changes.

dnsmichi · February 5, 2019, 7:48am

Strange. So you’re saying that you did not change anything on this system, except for the notification apply rules?

What happens, if you use a dummy notification script instead, do you still get that output?

object NotificationCommand "dummy" {
  command = [ ConfigDir + "/scripts/notification-dummy.sh" ]
}

vim /etc/icinga2/scripts/notification-dummy.sh

#!/bin/bash

echo "`date +%s` Dummy notification script triggered on `hostname -f`" >> /tmp/i2_notification_test

Which distribution are you on btw?

Cheers,
Michael

tzwarycz · February 6, 2019, 4:54pm

CentOS 7.5. These are the only modifications I am aware of. Just wondering if we can remove the garbage in the initial warning/critical email that is sent as shown in my initial post. It looks like it’s being put there by icinga…

dnsmichi · February 6, 2019, 6:16pm

That’s the output of icinga2 when invoked on the shell, without any parameters. That’s why I was looking for an occurrence of this single command invocation, inside the bash script.

This doesn’t happen if you use the proposed dummy script, right?

Question aside, what happens when you invoke hostname on your command shell?