Thanks for responding so quickly and I will try to answer everything the best I can.
I checked the icinga2.log file and below is what it said when I forced a notification from the director.
I replaced some of the identifiable information.
[2025-01-16 09:46:14 -0500] information/ApiListener: New client connection from [::ffff:127.0.0.1]:41222 (no client certificate)
[2025-01-16 09:46:14 -0500] information/Checkable: Checkable 'ServerName' has 1 notification(s). Checking filters for type 'Custom', sends will be logged.
[2025-01-16 09:46:14 -0500] information/Notification: Sending 'Custom' notification 'ServerName!wrapper-template-mail-host-notification' for user 'id-prusk'
[2025-01-16 09:46:14 -0500] information/Notification: Sending 'Custom' notification 'ServerName!wrapper-template-mail-host-notification' for user 'id-rmosher'
[2025-01-16 09:46:14 -0500] information/Notification: Sending 'Custom' notification 'ServerName!wrapper-template-mail-host-notification' for user 'id-helpdesk'
[2025-01-16 09:46:14 -0500] information/HttpServerConnection: Request POST /v1/actions/send-custom-notification (from [::ffff:127.0.0.1]:41222), user: icingaweb2, agent: , status: OK) took 0ms.
[2025-01-16 09:46:14 -0500] information/Notification: Completed sending 'Custom' notification 'ServerName!wrapper-template-mail-host-notification' for checkable 'ServerName' and user 'id-prusk' using command 'mail-host-notification'.
[2025-01-16 09:46:14 -0500] information/Notification: Completed sending 'Custom' notification 'ServerName!wrapper-template-mail-host-notification' for checkable 'ServerName' and user 'id-rmosher' using command 'mail-host-notification'.
[2025-01-16 09:46:14 -0500] information/Notification: Completed sending 'Custom' notification 'ServerName!wrapper-template-mail-host-notification' for checkable 'ServerName' and user 'id-helpdesk' using command 'mail-host-notification'.
[2025-01-16 09:46:14 -0500] warning/PluginNotificationTask: Notification command for object 'ServerName' (PID: 37308, arguments: '/etc/icinga2/scripts/mail-host-notification.sh' '-4' '' '-6' '' '-X' '' '-b' 'EmailAddress' '-c' 'test' '-d' '2025-01-16 09:46:14 -0500' '-l' 'ServerName' '-n' 'ServerName' '-o' 'Zone 'ServerName' is connected. Log lag: less than 1 millisecond' '-r' 'EmailAddress' '-s' 'UP' '-t' 'CUSTOM') terminated with exit code 1, output: Illegal option -X
ERROR: Invalid option -
Required parameters:
-d LONGDATETIME ($icinga.long_date_time$)
-l HOSTNAME ($host.name$)
-n HOSTDISPLAYNAME ($host.display_name$)
-o HOSTOUTPUT ($host.output$)
-r USEREMAIL ($user.email$)
-s HOSTSTATE ($host.state$)
-t NOTIFICATIONTYPE ($notification.type$)
Optional parameters:
-4 HOSTADDRESS ($address$)
-6 HOSTADDRESS6 ($address6$)
-b NOTIFICATIONAUTHORNAME ($notification.author$)
-c NOTIFICATIONCOMMENT ($notification.comment$)
-i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
-f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
-v ($notification_sendtosyslog$, Default: false)
[2025-01-16 09:46:14 -0500] warning/PluginNotificationTask: Notification command for object 'ServerName' (PID: 37310, arguments: '/etc/icinga2/scripts/mail-host-notification.sh' '-4' '' '-6' '' '-X' '' '-b' 'EmailAddress' '-c' 'test' '-d' '2025-01-16 09:46:14 -0500' '-l' 'ServerName' '-n' 'ServerName' '-o' 'Zone 'ServerName' is connected. Log lag: less than 1 millisecond' '-r' 'EmailAddress' '-s' 'UP' '-t' 'CUSTOM') terminated with exit code 1, output: Illegal option -X
ERROR: Invalid option -
Required parameters:
-d LONGDATETIME ($icinga.long_date_time$)
-l HOSTNAME ($host.name$)
-n HOSTDISPLAYNAME ($host.display_name$)
-o HOSTOUTPUT ($host.output$)
-r USEREMAIL ($user.email$)
-s HOSTSTATE ($host.state$)
-t NOTIFICATIONTYPE ($notification.type$)
Optional parameters:
-4 HOSTADDRESS ($address$)
-6 HOSTADDRESS6 ($address6$)
-b NOTIFICATIONAUTHORNAME ($notification.author$)
-c NOTIFICATIONCOMMENT ($notification.comment$)
-i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
-f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
-v ($notification_sendtosyslog$, Default: false)
[2025-01-16 09:46:14 -0500] warning/PluginNotificationTask: Notification command for object 'ServerName' (PID: 37309, arguments: '/etc/icinga2/scripts/mail-host-notification.sh' '-4' '' '-6' '' '-X' '' '-b' 'EmailAddress' '-c' 'test' '-d' '2025-01-16 09:46:14 -0500' '-l' 'ServerName' '-n' 'ServerName' '-o' 'Zone 'ServerName' is connected. Log lag: less than 1 millisecond' '-r' 'EmailAddress' '-s' 'UP' '-t' 'CUSTOM') terminated with exit code 1, output: Illegal option -X
ERROR: Invalid option -
Required parameters:
-d LONGDATETIME ($icinga.long_date_time$)
-l HOSTNAME ($host.name$)
-n HOSTDISPLAYNAME ($host.display_name$)
-o HOSTOUTPUT ($host.output$)
-r USEREMAIL ($user.email$)
-s HOSTSTATE ($host.state$)
-t NOTIFICATIONTYPE ($notification.type$)
Optional parameters:
-4 HOSTADDRESS ($address$)
-6 HOSTADDRESS6 ($address6$)
-b NOTIFICATIONAUTHORNAME ($notification.author$)
-c NOTIFICATIONCOMMENT ($notification.comment$)
-i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
-f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
-v ($notification_sendtosyslog$, Default: false)
[2025-01-16 09:46:14 -0500] information/HttpServerConnection: HTTP client disconnected (from [::ffff:127.0.0.1]:41222)
Ran the command sudo -u nagios icinga2 daemon -C --dump-objects and got the result below.
sudo -u nagios icinga2 daemon -C --dump-objects
[2025-01-16 09:55:31 -0500] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
[2025-01-16 09:55:31 -0500] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
[2025-01-16 09:55:31 -0500] information/cli: Icinga application loader (version: r2.14.2-1)
[2025-01-16 09:55:31 -0500] information/cli: Loading configuration file(s).
[2025-01-16 09:55:32 -0500] information/ConfigItem: Committing config item(s).
[2025-01-16 09:55:32 -0500] information/ApiListener: My API identity: ServerName
[2025-01-16 09:55:32 -0500] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
[2025-01-16 09:55:32 -0500] warning/ApplyRule: Apply rule 'disk-windows' (in /var/lib/icinga2/api/packages/director/77c36342-73d1-49a4-9a44-4eb7776a63ad/zones.d/ServerName/service_apply.conf: 146:1-146:28) for type 'Service' does not match anywhere!
[2025-01-16 09:55:32 -0500] warning/ApplyRule: Apply rule 'swap-windows' (in /var/lib/icinga2/api/packages/director/77c36342-73d1-49a4-9a44-4eb7776a63ad/zones.d/ServerName/service_apply.conf: 156:1-156:28) for type 'Service' does not match anywhere!
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 2 NotificationCommands.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 198 Notifications.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 5 HostGroups.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 19 Hosts.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 10 Comments.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 IcingaDB.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 FileLogger.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 21 Zones.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 4 Users.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 19 Endpoints.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 2 ApiUsers.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 ApiListener.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 247 CheckCommands.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 2 UserGroups.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 4 ServiceGroups.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-01-16 09:55:32 -0500] information/ConfigItem: Instantiated 179 Services.
[2025-01-16 09:55:32 -0500] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-01-16 09:55:32 -0500] information/cli: Finished validating the configuration file(s).
Ran the command sudo -u nagios icinga2 object list -t Notification and below is information I got.
sudo -u nagios icinga2 oject list -t Notification
[2025-01-16 09:58:08 -0500] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
[2025-01-16 09:58:08 -0500] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.2-1)
Usage:
icinga2 <command> [<arguments>]
Supported commands:
* api setup (setup for API)
* ca list (lists pending certificate signing requests)
* ca remove (removes an outstanding certificate request)
* ca restore (restores a removed certificate request)
* ca sign (signs an outstanding certificate request)
* console (Icinga console)
* daemon (starts Icinga 2)
* feature disable (disables specified feature)
* feature enable (enables specified feature)
* feature list (lists all available features)
* node setup (set up node)
* node wizard (wizard for node setup)
* object list (lists all objects)
* pki new-ca (sets up a new CA)
* pki new-cert (creates a new CSR)
* pki request (requests a certificate)
* pki save-cert (saves another Icinga 2 instance's certificate)
* pki sign-csr (signs a CSR)
* pki ticket (generates a ticket)
* pki verify (verify TLS certificates: CN, signed by CA, is CA; Print certificate)
* variable get (gets a variable)
* variable list (lists all variables)
Global options:
-h [ --help ] show this help message
-V [ --version ] show version information
--color use VT100 color codes even when stdout is not a
terminal
-D [ --define ] arg define a constant
-I [ --include ] arg add include search directory
-x [ --log-level ] arg specify the log level for the console log.
The valid value is either debug, notice,
information (default), warning, or critical
-X [ --script-debugger ] whether to enable the script debugger
And lastly, how does your mail-host-notifications.sh script look like? Does it match the upstream one? No it does not. Below is our version.
#!/bin/sh
# Icinga 2 | (c) 2012 Icinga GmbH | GPLv2+
# Except of function urlencode which is Copyright (C) by Brian White used under MIT license
PROG="`basename $0`"
ICINGA2HOST="`hostname`"
MAILBIN="sendEmail"
if [ -z "`which $MAILBIN`" ] ; then
echo "$MAILBIN not found in \$PATH. Consider installing it."
exit 1
fi
## Function helpers
Usage() {
cat << EOF
Required parameters:
-d LONGDATETIME (\$icinga.long_date_time\$)
-l HOSTNAME (\$host.name\$)
-n HOSTDISPLAYNAME (\$host.display_name\$)
-o HOSTOUTPUT (\$host.output\$)
-r USEREMAIL (\$user.email\$)
-s HOSTSTATE (\$host.state\$)
-t NOTIFICATIONTYPE (\$notification.type\$)
Optional parameters:
-4 HOSTADDRESS (\$address\$)
-6 HOSTADDRESS6 (\$address6\$)
-b NOTIFICATIONAUTHORNAME (\$notification.author\$)
-c NOTIFICATIONCOMMENT (\$notification.comment\$)
-i ICINGAWEB2URL (\$notification_icingaweb2url\$, Default: unset)
-f MAILFROM (\$notification_mailfrom\$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
-v (\$notification_sendtosyslog\$, Default: false)
EOF
}
Help() {
Usage;
exit 0;
}
Error() {
if [ "$1" ]; then
echo $1
fi
Usage;
exit 1;
}
urlencode() {
local LANG=C i=0 c e s="$1"
while [ $i -lt ${#1} ]; do
[ "$i" -eq 0 ] || s="${s#?}"
c=${s%"${s#?}"}
[ -z "${c#[[:alnum:].~_-]}" ] || c=$(printf '%%%02X' "'$c")
e="${e}${c}"
i=$((i + 1))
done
echo "$e"
}
## Main
while getopts 4:6::b:c:d:f:hi:l:n:o:r:s:t:v: opt
do
case "$opt" in
4) HOSTADDRESS=$OPTARG ;;
6) HOSTADDRESS6=$OPTARG ;;
b) NOTIFICATIONAUTHORNAME=$OPTARG ;;
c) NOTIFICATIONCOMMENT=$OPTARG ;;
d) LONGDATETIME=$OPTARG ;; # required
f) MAILFROM=$OPTARG ;;
h) Help ;;
i) ICINGAWEB2URL=$OPTARG ;;
l) HOSTNAME=$OPTARG ;; # required
n) HOSTDISPLAYNAME=$OPTARG ;; # required
o) HOSTOUTPUT=$OPTARG ;; # required
r) USEREMAIL=$OPTARG ;; # required
s) HOSTSTATE=$OPTARG ;; # required
t) NOTIFICATIONTYPE=$OPTARG ;; # required
v) VERBOSE=$OPTARG ;;
\?) echo "ERROR: Invalid option -$OPTARG" >&2
Error ;;
:) echo "Missing option argument for -$OPTARG" >&2
Error ;;
*) echo "Unimplemented option: -$OPTARG" >&2
Error ;;
esac
done
shift $((OPTIND - 1))
## Keep formatting in sync with mail-service-notification.sh
for P in LONGDATETIME HOSTNAME HOSTDISPLAYNAME HOSTOUTPUT HOSTSTATE USEREMAIL NOTIFICATIONTYPE ; do
eval "PAR=\$${P}"
if [ ! "$PAR" ] ; then
Error "Required parameter '$P' is missing."
fi
done
## Build the message's subject
SUBJECT="[$NOTIFICATIONTYPE] Host $HOSTDISPLAYNAME is $HOSTSTATE!"
## Build the notification message
NOTIFICATION_MESSAGE=`cat << EOF
Host Monitoring on $ICINGA2HOST
$HOSTDISPLAYNAME is $HOSTSTATE!
Info: $HOSTOUTPUT
When: $LONGDATETIME
Host: $HOSTNAME
EOF
`
## Check whether IPv4 was specified.
if [ -n "$HOSTADDRESS" ] ; then
NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
IPv4: $HOSTADDRESS"
fi
## Check whether IPv6 was specified.
if [ -n "$HOSTADDRESS6" ] ; then
NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
IPv6: $HOSTADDRESS6"
fi
## Check whether author and comment was specified.
if [ -n "$NOTIFICATIONCOMMENT" ] ; then
NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
Comment by $NOTIFICATIONAUTHORNAME:
$NOTIFICATIONCOMMENT"
fi
## Check whether Icinga Web 2 URL was specified.
if [ -n "$ICINGAWEB2URL" ] ; then
NOTIFICATION_MESSAGE="$NOTIFICATION_MESSAGE
$ICINGAWEB2URL/monitoring/host/show?host=$(urlencode "$HOSTNAME")"
fi
## Check whether verbose mode was enabled and log to syslog.
if [ "$VERBOSE" = "true" ] ; then
logger "$PROG sends $SUBJECT => $USEREMAIL"
fi
## Send the mail using the $MAILBIN command.
/usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | sendEmail -f EmailAddress -t "$USEREMAIL" -u "$SUBJECT" -s email-smtp.us-east-1.amazonaws.com:587 -xu "Access key ID" -xp "Secret access key(Converted to Password)" -o tls=yes -vvv -l /var/log/sendemail.log
##/usr/bin/printf "%b" "$NOTIFICATION_MESSAGE" | sendEmail -f EmailAddress -t "$USEREMAIL" -u "$SUBJECT" -s email-smtp.us-east-1.amazonaws.com:587 -xu "Access key ID" -xp "Secret access key(Converted to Password)" -o tls=yes -vvv -l /var/log/sendemail.log
Lastly, I tried updating using sudo apt update && sudo apt upgrade icinga2 but I kept getting an error message the following signatures couldn’t be verified because the public key is not available.
Thanks for your help and I look forward to hearing you.