Including custom variables in notifications

itineed2022 · November 2, 2023, 7:54pm

I stopped trying to dynamically grab the vcenter data each time an alert is sent by Icinga and I mapped the values directly to the servers in Icinga.

How do I go about including variables, like “vars.vcenter_name”, in a notification?

rivad · November 3, 2023, 9:18am

In your notification commands, write values into environment variables and read them out again in the notification script.
https://icinga.com/docs/icinga-2/latest/doc/03-monitoring-basics/#environment-variables

This is better then arguments as there is a limit on argument length

itineed2022 · November 3, 2023, 1:30pm

I put my “vars.vcenter_name” variable in /etc/icinga2/conf.d/commands.conf under the mail-host-notification command. Hosts with this are configured in /etc/icinga2/conf.d/hosts.conf. But when I send a custom notification, I still do not see the value. What did I miss?

itineed2022 · November 3, 2023, 2:31pm

Why is this so difficult? I have the information I need to include in my alerts. I have the documentation, but for the two days I’ve been reading it I can’t figure out how to just include that information in the alert. Surely this has been done before? I can’t be the first person to want to send information other than what is in the defaults Icinga provides.

rivad · November 3, 2023, 3:25pm

The default Icinga provides is a inspiration to write your own notification scripts.
I use the ones from https://github.com/Linuxfabrik/monitoring-plugins/tree/main/notification-plugins and GitHub - seffparker/icinga2-teams-notification: Send Icinga2 notification to Microsoft Teams as Incoming Webhooks.

Did you also add the code to read out the environment variable in the mail script? This isn’t in the documentation as after the env.vcenter_name = host.vars.vcenter_name in the mail-host-notification, it’s just normal shell scripting or what ever language the command you call is written in.

itineed2022 · November 3, 2023, 3:50pm

What is “the mail script”? Are you referring to mail-host-notification.sh? Where do I add the code you’re referring to? In commands.conf? templates.conf? hosts.conf?

So to be clear, there is no standardized way of adding simple text information to a notification?

rivad · November 3, 2023, 4:35pm

So to be clear, there is no standardized way of adding simple text information to a notification?

No, except as a Linux administrator, I regard environment variables or command line arguments and a simple shell script as quite standard.

If I got your intent right:

you add the information to the host into something like a vars.vcenter_name
in the notification use add the value to the env with something like env.VCENTER_NAME = host.vars.vcenter_name or was it env.VCENTER_NAME = $host.vars.vcenter_name$
in the script mail-host-notification.sh and mail-service-notification.sh also defined in the notification as command,
add some code to get the information from the env - copy something already present like:

## Check whether Icinga Web 2 URL was specified.
if [ -n "$ICINGAWEB2URL" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

$ICINGAWEB2URL/icingadb/service?name=$(urlencode "$SERVICENAME")&host.name=$(urlencode "$HOSTNAME")"
fi

just replace $ICINGAWEB2URL with your env variable name and rewite the second last line to fit your style:

## Check whether vCenter name was specified.
if [ -n "$VCENTER_NAME" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

vCenter = $VCENTER_NAME"
fi

I haven’t tested anything of the above!

itineed2022 · November 6, 2023, 3:24pm

This is in my templates.conf file at /etc/icinga2/conf.d/templates.conf:

/**
 * Provides default settings for host notifications.
 * By convention all host notifications should import
 * this template.
 */
template Notification "mail-host-notification" {
  command = "mail-host-notification"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    //notification_icingaweb2url = "https://www.example.com/icingaweb2"
    //notification_from = "Icinga 2 Host Monitoring <icinga@example.com>"
    notification_logtosyslog = false
  }

  period = "24x7"
}

I take it I would add:

host.notes = $host.vars.vcenter_name$

to:

  vars += {
    //notification_icingaweb2url = "https://www.example.com/icingaweb2"
    //notification_from = "Icinga 2 Host Monitoring <icinga@example.com>"
    notification_logtosyslog = false
    host.notes = $host.vars.vcenter_name$
  }

?

rivad · November 6, 2023, 5:06pm

I don’t think so.

More like:

/**
 * Provides default settings for host notifications.
 * By convention all host notifications should import
 * this template.
 */
template Notification "mail-host-notification" {
  command = "mail-host-notification"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    //notification_icingaweb2url = "https://www.example.com/icingaweb2"
    //notification_from = "Icinga 2 Host Monitoring <icinga@example.com>"
    notification_logtosyslog = false
  }

  period = "24x7"

  if (host.vars.vcenter_name) {
     env.VCENTER_NAME = host.vars.vcenter_name
  }
}

itineed2022 · November 6, 2023, 10:11pm

I keep trying to set an environment variable but icinga2 fails to start afterward.

template Notification "my-host-notification" {
  command = "mail-host-notification-me"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_icingaweb2url = "$vcenter_name$"
    vcenter_notes = "$vcenter_name$"
    // notification_from = "Icinga 2 Service Monitoring <icinga@example.com>"
    notification_logtosyslog = false
  }
  period = "24x7"
  if (host.vars.vcenter_name) {
     env.VCENTER_NAME = host.vars.vcenter_name
  }
}

I tried creating my own flag “-j” but I guess Icinga doesn’t support custom flags.

itineed2022 · November 6, 2023, 10:16pm

Is there anyway to just set host.notes to a value? I see the flag isn’t being used but nothing I try makes it work.

rivad · November 7, 2023, 10:43am

What happens if you do a icinga2 daemon -C.

Both environment variables and custom flags should be a valid option.
Did you import the existing command definition in your custom one with the new flag?

itineed2022 · November 8, 2023, 5:32pm

I figured out how the env field works and I even managed to send alerts using them!

however, my lead on the Icinga project says that environment variables are a hack. He won’t say why they’re a hack.

How do I set “host.notes” to a value? I tried with

vars += { host.notes = "test" }
-also-
vars += { $host.notes$ = "test" }
-also-
vars += { notes = "test" }
-and finally-
vars += { $notes$ = "test" }

rivad · November 8, 2023, 6:03pm

I don’t understand why environment variables are supposed to be a hack. For credentials they would be even preferred as they don’t show up in the process list. I guess this is the main difference about arguments and environment variables.

If he can’t answer the question it’s only a opinion and no argument

In the object NotificationCommand you need to add the mapping between Icinga2 variables and arguments for the notification script, if you want to add a new one.

If you want to append to the notes, why not:

if (host.vars.vcenter_name) {
  host.notes += "\n vCenter: $host.vars.vcenter_name$"
}

?

itineed2022 · November 8, 2023, 6:20pm

When I add that to my Notification, nothing is sent in the e-mail:

template Notification "custom-host-notification" {
  command = "mail-host-notification"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    // notification_from = "Icinga 2 Service Monitoring <icinga@example.com>"
    notification_logtosyslog = false
  }

  period = "24x7"
  if (host.vars.vcenter_name) {
        host.notes += "\n Vcenter: $host.vars.vcenter_name$"
  }
}

The Notifcation Command looks like:

!/usr/bin/env bash
# this mod includes the VCENTERNOTES value.
# Icinga 2 | (c) 2012 Icinga GmbH | GPLv2+
# Except of function urlencode which is Copyright (C) by Brian White (brian@aljex.com) used under MIT license

PROG="`basename $0`"
ICINGA2HOST="`hostname`"
MAILBIN="mail"

if [ -z "`which $MAILBIN`" ] ; then
  echo "$MAILBIN not found in \$PATH. Consider installing it."
  exit 1
fi

## Function helpers
Usage() {
cat << EOF

Required parameters:
  -d LONGDATETIME (\$icinga.long_date_time\$)
  -l HOSTNAME (\$host.name\$)
  -n HOSTDISPLAYNAME (\$host.display_name\$)
  -o HOSTOUTPUT (\$host.output\$)
  -r USEREMAIL (\$user.email\$)
  -s HOSTSTATE (\$host.state\$)
  -t NOTIFICATIONTYPE (\$notification.type\$)

Optional parameters:
  -4 HOSTADDRESS (\$address\$)
  -6 HOSTADDRESS6 (\$address6\$)
  -X HOSTNOTES (\$host.notes\$)
  -b NOTIFICATIONAUTHORNAME (\$notification.author\$)
  -c NOTIFICATIONCOMMENT (\$notification.comment\$)
  -i ICINGAWEB2URL (\$notification_icingaweb2url\$, Default: unset)
  -f MAILFROM (\$notification_mailfrom\$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
  -v (\$notification_sendtosyslog\$, Default: false)

EOF
}

Help() {
  Usage;
  exit 0;
}

Error() {
  if [ "$1" ]; then
    echo $1
  fi
  Usage;
  exit 1;
}

urlencode() {
  local LANG=C i=0 c e s="$1"

  while [ $i -lt ${#1} ]; do
    [ "$i" -eq 0 ] || s="${s#?}"
    c=${s%"${s#?}"}
    [ -z "${c#[[:alnum:].~_-]}" ] || c=$(printf '%%%02X' "'$c")
    e="${e}${c}"
    i=$((i + 1))
  done
  echo "$e"
}

## Main
while getopts 4:6::b:c:d:f:hi:l:n:o:r:s:t:v:X: opt
do
  case "$opt" in
    4) HOSTADDRESS=$OPTARG ;;
    6) HOSTADDRESS6=$OPTARG ;;
    b) NOTIFICATIONAUTHORNAME=$OPTARG ;;
    c) NOTIFICATIONCOMMENT=$OPTARG ;;
    d) LONGDATETIME=$OPTARG ;; # required
    f) MAILFROM=$OPTARG ;;
    h) Help ;;
    i) ICINGAWEB2URL=$OPTARG ;;
    l) HOSTNAME=$OPTARG ;; # required
    n) HOSTDISPLAYNAME=$OPTARG ;; # required
    o) HOSTOUTPUT=$OPTARG ;; # required
    X) HOSTNOTES=$OPTARG ;;
    r) USEREMAIL=$OPTARG ;; # required
    s) HOSTSTATE=$OPTARG ;; # required
    t) NOTIFICATIONTYPE=$OPTARG ;; # required
    v) VERBOSE=$OPTARG ;;
   \?) echo "ERROR: Invalid option -$OPTARG" >&2
       Error ;;
    :) echo "Missing option argument for -$OPTARG" >&2
       Error ;;
    *) echo "Unimplemented option: -$OPTARG" >&2
       Error ;;
  esac
done

shift $((OPTIND - 1))

## Keep formatting in sync with mail-service-notification.sh
for P in LONGDATETIME HOSTNAME HOSTDISPLAYNAME HOSTOUTPUT HOSTSTATE USEREMAIL NOTIFICATIONTYPE ; do
	eval "PAR=\$${P}"

	if [ ! "$PAR" ] ; then
		Error "Required parameter '$P' is missing."
	fi
done

## Build the message's subject
SUBJECT="[$NOTIFICATIONTYPE] Host $HOSTDISPLAYNAME is $HOSTSTATE!"

## Build the notification message
NOTIFICATION_MESSAGE=`cat << EOF
***** Host Monitoring on $ICINGA2HOST *****

$HOSTDISPLAYNAME is $HOSTSTATE!

Info:    $HOSTOUTPUT

When:    $LONGDATETIME
Host:    $HOSTNAME
EOF
`

## Check whether IPv4 was specified.
if [ -n "$HOSTADDRESS" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
IPv4:	 $HOSTADDRESS"
fi

## Check whether IPv6 was specified.
if [ -n "$HOSTADDRESS6" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
IPv6:	 $HOSTADDRESS6"
fi

## Check whether host notes was specified.
if [ -n "$HOSTNOTES" ] ; then
        NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
Host notes: $HOSTNOTES"
fi

## Check whether author and comment was specified.
if [ -n "$NOTIFICATIONCOMMENT" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

Comment by $NOTIFICATIONAUTHORNAME:
  $NOTIFICATIONCOMMENT"
fi

## Check whether Icinga Web 2 URL was specified.
if [ -n "$ICINGAWEB2URL" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

$ICINGAWEB2URL/icingadb/host?name=$(urlencode "$HOSTNAME")"
fi

## Check whether vcenter_notes was specified.
if [ -n "$VCENTERNOTES" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

$VCENTERNOTES"
fi

## Check whether verbose mode was enabled and log to syslog.
if [ "$VERBOSE" = "true" ] ; then
  logger "$PROG sends $SUBJECT => $USEREMAIL"
fi

## Send the mail using the $MAILBIN command.
## If an explicit sender was specified, try to set it.
if [ -n "$MAILFROM" ] ; then

  ## Modify this for your own needs!

  ## Debian/Ubuntu use mailutils which requires `-a` to append the header
  if [ -f /etc/debian_version ]; then
    /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -a "From: $MAILFROM" -s "$SUBJECT" $USEREMAIL
  ## Other distributions (RHEL/SUSE/etc.) prefer mailx which sets a sender address with `-r`
  else
    /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "$MAILFROM" -s "$SUBJECT" $USEREMAIL
  fi

else
  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" \
  | $MAILBIN -s "$SUBJECT" $USEREMAIL
fi

so there is a HOSTNOTES, but for some reason, its not getting anything

I removed the if block just to see if I could force the macro. Unfortunately, it didn’t work.

template Notification "custom-host-notification" {
  command = "mail-host-notification-custom"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    // notification_from = "Icinga 2 Service Monitoring <icinga@example.com>"
    notification_logtosyslog = false
  }

  period = "24x7"
  host.notes += "\n Vcenter: $vars.vcenter_name$"
}

rivad · November 8, 2023, 6:39pm

I think it needs to go into mail-host-notification-custom.

itineed2022 · November 8, 2023, 6:55pm

I tried this:

object NotificationCommand "mail-host-notification-custom" {
  command = [ ConfigDir + "/scripts/mail-host-notification.sh" ]

  env = {
    VCENTERNOTES = "Host Information Below:\nvCenter name: $vcenter_name$\nvCenter_cluster: $vcenter_cluster$\nvCenter_esx_host: $vcenter_esx_host$"
  }

  arguments += {
    "-4" = "$notification_address$"
    "-6" = "$notification_address6$"
    "-b" = "$notification_author$"
    "-c" = "$notification_comment$"
    "-d" = {
      required = true
      value = "$notification_date$"
    }
    "-f" = {
      value = "$notification_from$"
      description = "Set from address. Requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE)"
    }
    "-i" = "$notification_icingaweb2url$"
    "-l" = {
      required = true
      value = "$notification_hostname$"
    }
    "-n" = {
      required = true
      value = "$notification_hostdisplayname$"
    }
    "-o" = {
      required = true
      value = "$notification_hostoutput$"
    }
    "-r" = {
      required = true
      value = "$notification_useremail$"
    }
    "-s" = {
      required = true
      value = "$notification_hoststate$"
    }
    "-t" = {
      required = true
      value = "$notification_type$"
    }
    "-v" = "$notification_logtosyslog$"
    "-X" = "$notification_notes$"
  }
  if (host.vars.vcenter_name) {
    host.notes += "\n vCenter: $host.vars.vcenter_name$"
  }
  vars += {
    notification_address = "$address$"
    notification_address6 = "$address6$"
    notification_author = "$notification.author$"
    notification_comment = "$notification.comment$"
    notification_type = "$notification.type$"
    notification_date = "$icinga.long_date_time$"
    notification_hostname = "$host.name$"
    notification_hostdisplayname = "$host.display_name$"
    notification_hostoutput = "$host.output$"
    notification_hoststate = "$host.state$"
    notification_useremail = "$user.email$"
    notification_notes = "$host.notes$"
  }
}

It doesn’t seem to want to assign it there:

[root@devapp01 icinga2]# icinga2 daemon -C
[2023-11-08 13:52:04 -0500] information/cli: Icinga application loader (version: r2.14.0-1)
[2023-11-08 13:52:04 -0500] information/cli: Loading configuration file(s).
[2023-11-08 13:52:04 -0500] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/devapp01.mycompany.com' for unknown zone 'devapp01.mycompany.com'.
[2023-11-08 13:52:04 -0500] information/ConfigItem: Committing config item(s).
[2023-11-08 13:52:04 -0500] critical/config: Error: Error while evaluating expression: Tried to access undefined script variable 'host'
Location: in /etc/icinga2/conf.d/custom.conf: 89:7-89:10
/etc/icinga2/conf.d/custom.conf(87):     "-X" = "$notification_notes$"
/etc/icinga2/conf.d/custom.conf(88):   }
/etc/icinga2/conf.d/custom.conf(89):   if (host.vars.vcenter_name) {
                                                  ^^^^
/etc/icinga2/conf.d/custom.conf(90):     host.notes += "\n vCenter: $host.vars.vcenter_name$"
/etc/icinga2/conf.d/custom.conf(91):   }
[2023-11-08 13:52:04 -0500] critical/config: 1 error
[2023-11-08 13:52:04 -0500] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

rivad · November 9, 2023, 8:15am

Maybe you need a macro or a lamda to make sure it gets evaluated at runtime not at config time:

 notification_notes = {{
    notes = host.notes
    if (host.vars.vcenter_name) {
      notes += "\n vCenter: $host.vars.vcenter_name$"
    }
    return notes
  }}

When do you fill host.vars.vcenter_name for the hosts?

itineed2022 · November 9, 2023, 2:15pm

Thanks for all your help on this Dominik. I finally got it working. For anyone in the future, below I’ve added the three ways I found to make this work:

template Notification "custom-host-notification" {
  command = "mail-host-notification-custom"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    // notification_from = "Icinga 2 Service Monitoring <icinga@example.com>"
    notification_logtosyslog = false
    vcenter_info = "\nHost Information Below:\nvCenter name: $vcenter_name$\nvCenter_cluster: $vcenter_cluster$\nvCenter_esx_host: $vcenter_esx_host$"
  }

  period = "24x7"
}

The NotificationCommand Looks like:

object NotificationCommand "mail-host-notification-custom" {
  command = [ ConfigDir + "/scripts/mail-host-notification-custom.sh" ]
  // The environment variable. Using an env block you can define as many as you want.
  env = {
    VCENTERNOTES = "Host Information Below:\nvCenter name: $vcenter_name$\nvCenter_cluster: $vcenter_cluster$\nvCenter_esx_host: $vcenter_esx_host$"
  }

  arguments += {
    "-4" = "$notification_address$"
    "-6" = "$notification_address6$"
    "-b" = "$notification_author$"
    "-c" = "$notification_comment$"
    "-d" = {
      required = true
      value = "$notification_date$"
    }
    "-f" = {
      value = "$notification_from$"
      description = "Set from address. Requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE)"
    }
    "-i" = "$notification_icingaweb2url$"
    "-l" = {
      required = true
      value = "$notification_hostname$"
    }
    "-n" = {
      required = true
      value = "$notification_hostdisplayname$"
    }
    "-o" = {
      required = true
      value = "$notification_hostoutput$"
    }
    "-r" = {
      required = true
      value = "$notification_useremail$"
    }
    "-s" = {
      required = true
      value = "$notification_hoststate$"
    }
    "-t" = {
      required = true
      value = "$notification_type$"
    }
    "-v" = "$notification_logtosyslog$"
    // Passing notification notes to -X overrides $host.notes$ which seems to be reset on run-time. It is a run-time macro so this might explain the behavior.
    "-X" = "$notification_notes$"
    // Custom flag "j", this WILL need to be included in three locations in the mail script to work
    "-j" = "$notification_notes$"
  }
  vars += {
    notification_address = "$address$"
    notification_address6 = "$address6$"
    notification_author = "$notification.author$"
    notification_comment = "$notification.comment$"
    notification_type = "$notification.type$"
    notification_date = "$icinga.long_date_time$"
    notification_hostname = "$host.name$"
    notification_hostdisplayname = "$host.display_name$"
    notification_hostoutput = "$host.output$"
    notification_hoststate = "$host.state$"
    notification_useremail = "$user.email$"
    // Notice here that the variable from Notification is being passed. For reasons I do not understand, directly calling vcenter_info with the flag throws an error.
    notification_notes = "$vcenter_info$"
  }
}

Finally, the actual mailing script:

#!/usr/bin/env bash
# This is based on the example provided by Icinga.
# Icinga 2 | (c) 2012 Icinga GmbH | GPLv2+
# Except of function urlencode which is Copyright (C) by Brian White (brian@aljex.com) used under MIT license

PROG="`basename $0`"
ICINGA2HOST="`hostname`"
MAILBIN="mail"

if [ -z "`which $MAILBIN`" ] ; then
  echo "$MAILBIN not found in \$PATH. Consider installing it."
  exit 1
fi

## Function helpers
Usage() {
cat << EOF

Required parameters:
  -d LONGDATETIME (\$icinga.long_date_time\$)
  -l HOSTNAME (\$host.name\$)
  -n HOSTDISPLAYNAME (\$host.display_name\$)
  -o HOSTOUTPUT (\$host.output\$)
  -r USEREMAIL (\$user.email\$)
  -s HOSTSTATE (\$host.state\$)
  -t NOTIFICATIONTYPE (\$notification.type\$)

Optional parameters:
  -4 HOSTADDRESS (\$address\$)
  -6 HOSTADDRESS6 (\$address6\$)
  -X HOSTNOTES (\$host.notes\$)
  -b NOTIFICATIONAUTHORNAME (\$notification.author\$)
  -c NOTIFICATIONCOMMENT (\$notification.comment\$)
  -i ICINGAWEB2URL (\$notification_icingaweb2url\$, Default: unset)
  -f MAILFROM (\$notification_mailfrom\$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
  -v (\$notification_sendtosyslog\$, Default: false)
  -j (\$notification_notes\$, Default: unset)
EOF
}
## Notice the -j flag and its position. Positioning seems to matter when adding custom flags.

Help() {
  Usage;
  exit 0;
}

Error() {
  if [ "$1" ]; then
    echo $1
  fi
  Usage;
  exit 1;
}

urlencode() {
  local LANG=C i=0 c e s="$1"

  while [ $i -lt ${#1} ]; do
    [ "$i" -eq 0 ] || s="${s#?}"
    c=${s%"${s#?}"}
    [ -z "${c#[[:alnum:].~_-]}" ] || c=$(printf '%%%02X' "'$c")
    e="${e}${c}"
    i=$((i + 1))
  done
  echo "$e"
}

## Main
## The "j" option at the end of the getopts arg is the custom flag. The order seems to matter. Again I don't know why but I'd recommend always adding the custom flags to the end of the arguments.
while getopts 4:6::b:c:d:f:hi:l:n:o:r:s:t:v:X:j: opt
do
  case "$opt" in
    4) HOSTADDRESS=$OPTARG ;;
    6) HOSTADDRESS6=$OPTARG ;;
    b) NOTIFICATIONAUTHORNAME=$OPTARG ;;
    c) NOTIFICATIONCOMMENT=$OPTARG ;;
    d) LONGDATETIME=$OPTARG ;; # required
    f) MAILFROM=$OPTARG ;;
    h) Help ;;
    i) ICINGAWEB2URL=$OPTARG ;;
    l) HOSTNAME=$OPTARG ;; # required
    n) HOSTDISPLAYNAME=$OPTARG ;; # required
    o) HOSTOUTPUT=$OPTARG ;; # required
    X) HOSTNOTES=$OPTARG ;;
    r) USEREMAIL=$OPTARG ;; # required
    s) HOSTSTATE=$OPTARG ;; # required
    t) NOTIFICATIONTYPE=$OPTARG ;; # required
    v) VERBOSE=$OPTARG ;;
    j) NOTIFICATIONNOTES=$OPTARG;;
   \?) echo "ERROR: Invalid option -$OPTARG" >&2
       Error ;;
    :) echo "Missing option argument for -$OPTARG" >&2
       Error ;;
    *) echo "Unimplemented option: -$OPTARG" >&2
       Error ;;
  esac
done

shift $((OPTIND - 1))

## Keep formatting in sync with mail-service-notification.sh
for P in LONGDATETIME HOSTNAME HOSTDISPLAYNAME HOSTOUTPUT HOSTSTATE USEREMAIL NOTIFICATIONTYPE ; do
    eval "PAR=\$${P}"

    if [ ! "$PAR" ] ; then
        Error "Required parameter '$P' is missing."
    fi
done

## Build the message's subject
SUBJECT="[$NOTIFICATIONTYPE] Host $HOSTDISPLAYNAME is $HOSTSTATE!"

## Build the notification message
NOTIFICATION_MESSAGE=`cat << EOF
***** Host Monitoring on $ICINGA2HOST *****

$HOSTDISPLAYNAME is $HOSTSTATE!

Info:    $HOSTOUTPUT

When:    $LONGDATETIME
Host:    $HOSTNAME
EOF
`

## Check whether IPv4 was specified.
if [ -n "$HOSTADDRESS" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
IPv4:     $HOSTADDRESS"
fi

## Check whether IPv6 was specified.
if [ -n "$HOSTADDRESS6" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
IPv6:     $HOSTADDRESS6"
fi

## Check whether host notes was specified.
if [ -n "$HOSTNOTES" ] ; then
        NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
Host notes: $HOSTNOTES"
fi

## This is the vCenter data, but passed using a custom flag.
if [ -n "$NOTIFICATIONNOTES" ] ; then
        NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
Custom Notes: $NOTIFICATIONNOTES"
fi

## Check whether author and comment was specified.
if [ -n "$NOTIFICATIONCOMMENT" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

Comment by $NOTIFICATIONAUTHORNAME:
  $NOTIFICATIONCOMMENT"
fi

## Check whether Icinga Web 2 URL was specified.
if [ -n "$ICINGAWEB2URL" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

$ICINGAWEB2URL/icingadb/host?name=$(urlencode "$HOSTNAME")"
fi

## Check whether vcenter_notes was specified. This is the environment variable.
if [ -n "$VCENTERNOTES" ] ; then
  NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE

$VCENTERNOTES"
fi

## Check whether verbose mode was enabled and log to syslog.
if [ "$VERBOSE" = "true" ] ; then
  logger "$PROG sends $SUBJECT => $USEREMAIL"
fi

## Send the mail using the $MAILBIN command.
## If an explicit sender was specified, try to set it.
if [ -n "$MAILFROM" ] ; then

  ## Modify this for your own needs!

  ## Debian/Ubuntu use mailutils which requires `-a` to append the header
  if [ -f /etc/debian_version ]; then
    /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -a "From: $MAILFROM" -s "$SUBJECT" $USEREMAIL
  ## Other distributions (RHEL/SUSE/etc.) prefer mailx which sets a sender address with `-r`
  else
    /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | $MAILBIN -r "$MAILFROM" -s "$SUBJECT" $USEREMAIL
  fi

else
  /usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" \
  | $MAILBIN -s "$SUBJECT" $USEREMAIL
fi

The only thing I’m concerned about is overriding the -X host.notes option. I could get the values in the Icinga web console but every time I tried to send them to -X, the value was blank. It’s like something is replacing it at runtime. Reusing the -X option seems okay but if in the future someone actually wanted to use the feature in Icinga director overriding -X will prevent those notes from being included in alerts.