Send email notifications with sendmail command (Centos 7)

Hello everyone,

I am writing a script that uses the sendmail command to send email but I am running into some issues.
When I execute the script from the shell with the icinga user it executes fine but when I try to send a notification from the web interface it returns the following:

Notification command for object 'agent1' (PID: 7467, arguments: '/usr/bin/sudo' '/usr/lib64/nagios/plugins/send-email-notification.sh') terminated with exit code 126,
output: /usr/lib64/nagios/plugins/send-email-notification.sh: line 90: /usr/sbin/sendmail: Permission denied

The SELinux flags are set as bellow:

httpd_can_connect_icinga2_api --> on
httpd_can_manage_icingaweb2_config --> on
httpd_can_write_icinga2_command --> on
icinga2_can_connect_all --> off
icinga2_run_sudo --> on
icinga2adm_exec_content --> on

The sendmail file persmissions are by default set to:

lrwxrwxrwx. 1 root root 21 Feb  1 09:24 /usr/sbin/sendmail -> /etc/alternatives/mta

Environment

icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.3)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.24.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Any help would be appreciated

Kind regards,
Mike

What is the context of /usr/lib64/nagios/plugins/send-email-notification.sh (output of ls -lZ)? Are there any entries in /var/log/audit/audit.log or what is the output of audit2allow?

The script:

#!/bin/bash

PROG="`basename $0`"
ICINGA2HOST="/usr/bin/hostname"
MAILBIN="/usr/sbin/sendmail"

## Function helpers
Usage() {
cat << EOF

Required parameters:
  -d LONGDATETIME (\$icinga.long_date_time\$)
  -g HOSTGROUP (\$host.groups\$)
  -l HOSTNAME (\$host.name\$)
  -n HOSTDISPLAYNAME (\$host.display_name\$)
  -o HOSTOUTPUT (\$host.output\$)
  -r USEREMAIL (\$user.email\$)
  -s HOSTSTATE (\$host.state\$)
  -t NOTIFICATIONTYPE (\$notification.type\$)

Optional parameters:
  -4 HOSTADDRESS (\$address\$)
  -6 HOSTADDRESS6 (\$address6\$)
  -b NOTIFICATIONAUTHORNAME (\$notification.author\$)
  -c NOTIFICATIONCOMMENT (\$notification.comment\$)
  -i ICINGAWEB2URL (\$notification_icingaweb2url\$, Default: unset)
  -f MAILFROM (\$notification_mailfrom\$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
  -v (\$notification_sendtosyslog\$, Default: false)

EOF
}


while getopts 4:6::b:c:d:f:g:hi:l:n:o:r:s:t:v: opt
do
  case "$opt" in
    4) HOSTADDRESS=$OPTARG ;;
    6) HOSTADDRESS6=$OPTARG ;;
    b) NOTIFICATIONAUTHORNAME=$OPTARG ;;
    c) NOTIFICATIONCOMMENT=$OPTARG ;;
    d) LONGDATETIME=$OPTARG ;; # required
    f) MAILFROM=$OPTARG ;;
    g) HOSTGROUP=$OPTARG ;;
    h) Help ;;
    i) ICINGAWEB2URL=$OPTARG ;;
    l) HOSTNAME=$OPTARG ;; # required
    n) HOSTDISPLAYNAME=$OPTARG ;; # required
    o) HOSTOUTPUT=$OPTARG ;; # required
    r) USEREMAIL=$OPTARG ;; # required
    s) HOSTSTATE=$OPTARG ;; # required
    t) NOTIFICATIONTYPE=$OPTARG ;; # required
    v) VERBOSE=$OPTARG ;;
   \?) echo "ERROR: Invalid option -$OPTARG" >&2
       Error ;;
    :) echo "Missing option argument for -$OPTARG" >&2
       Error ;;
    *) echo "Unimplemented option: -$OPTARG" >&2
       Error ;;
  esac
done

if [[ $HOSTSTATE == "UP" ]]; then
	COLOR="green"
elif [[ $HOSTSTATE == "DOWN" ]]; then
	COLOR="red"
fi

{
  echo "To: $USEREMAIL"
  echo "Subject: $HOSTDISPLAYNAME is $HOSTSTATE"
  echo "Mime-Version: 1.0"
  echo "Content-Type: text/html; charset='utf-8'"
  echo
  echo "<html>"
  echo "<body>"
  echo "<a style="font-size:20px\;font-weight:bold\;">$HOSTDISPLAYNAME</a> is <a style="color:$COLOR\;font-size:20px\;font-weight:bold\;">$HOSTSTATE</a><br><br>"
  echo "<p>Group: $HOSTGROUP</p>"
  echo "<p>Hostname:$HOSTNAME</p>"
  echo "<p>Info: $HOSTOUTPUT</p>"
  echo "<p>When: $LONGDATETIME</p>"
  echo "</body>"
  echo "</html>"
} | $MAILBIN -t

Output of ls -lZ:

-rwxr-xr-x. root nagios unconfined_u:object_r:lib_t:s0   /usr/lib64/nagios/plugins/send-email-notification.sh

The entries in the /var/log/audit/audit.log when I send the notification from the web interface:

type=USER_ACCT msg=audit(1619775980.916:448): pid=10910 uid=995 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="icinga" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=USER_CMD msg=audit(1619775980.916:449): pid=10910 uid=995 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='cwd="/" cmd="/usr/lib64/nagios/plugins/send-email-notification.sh" terminal=? res=success'
type=CRED_REFR msg=audit(1619775980.916:450): pid=10910 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=USER_START msg=audit(1619775980.952:451): pid=10910 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:session_open grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_krb5 acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1619775981.036:452): avc:  denied  { execute } for  pid=10918 comm="send-email-noti" name="sendmail.sendmail" dev="vda1" ino=6662602 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:sendmail_exec_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1619775981.036:452): arch=c000003e syscall=59 success=no exit=-13 a0=1e7e440 a1=1e81180 a2=1e7c510 a3=7ffdae4d9320 items=0 ppid=10915 pid=10918 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="send-email-noti" exe="/usr/bin/bash" subj=system_u:system_r:icinga2_t:s0 key=(null)
type=PROCTITLE msg=audit(1619775981.036:452): proctitle=2F62696E2F62617368002F7573722F6C696236342F6E6167696F732F706C7567696E732F73656E642D656D61696C2D6E6F74696669636174696F6E2E7368
type=AVC msg=audit(1619775981.036:453): avc:  denied  { getattr } for  pid=10918 comm="send-email-noti" path="/usr/sbin/sendmail.sendmail" dev="vda1" ino=6662602 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:sendmail_exec_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1619775981.036:453): arch=c000003e syscall=4 success=no exit=-13 a0=1e7e440 a1=7ffdae4d9ea0 a2=7ffdae4d9ea0 a3=7ffdae4d92a0 items=0 ppid=10915 pid=10918 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="send-email-noti" exe="/usr/bin/bash" subj=system_u:system_r:icinga2_t:s0 key=(null)
type=PROCTITLE msg=audit(1619775981.036:453): proctitle=2F62696E2F62617368002F7573722F6C696236342F6E6167696F732F706C7567696E732F73656E642D656D61696C2D6E6F74696669636174696F6E2E7368
type=AVC msg=audit(1619775981.036:454): avc:  denied  { getattr } for  pid=10918 comm="send-email-noti" path="/usr/sbin/sendmail.sendmail" dev="vda1" ino=6662602 scontext=system_u:system_r:icinga2_t:s0 tcontext=system_u:object_r:sendmail_exec_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1619775981.036:454): arch=c000003e syscall=4 success=no exit=-13 a0=1e7e440 a1=7ffdae4d9e80 a2=7ffdae4d9e80 a3=7ffdae4d92a0 items=0 ppid=10915 pid=10918 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="send-email-noti" exe="/usr/bin/bash" subj=system_u:system_r:icinga2_t:s0 key=(null)
type=PROCTITLE msg=audit(1619775981.036:454): proctitle=2F62696E2F62617368002F7573722F6C696236342F6E6167696F732F706C7567696E732F73656E642D656D61696C2D6E6F74696669636174696F6E2E7368
type=USER_END msg=audit(1619775981.050:455): pid=10910 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_krb5 acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=CRED_DISP msg=audit(1619775981.050:456): pid=10910 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'

I think this is the problem. Can you change the context of the file to via chcon -t nagios_notification_plugin_exec_t /usr/lib64/nagios/plugins/send-email-notification.sh and verify if this works?
If yes, you should also add it to your local policy with semanage fcontext -a -t nagios_notification_plugin_exec_t /usr/lib64/nagios/plugins/send-email-notification.sh so it can not be overwritten.

Some background: With lib_t there will happen no transition and icinga2 will try to directly executed it in its context icinga2_t what is not allowed. With nagios_notification_plugin_exec_t it will transition to nagios_notification_plugin_t which I created for the sample notification script and adjusted later on for other plugins which had problems so it should be allowed.

I change the context of the file as you said and it seems to be executing it, but I don’t receive anything.
I guess that is not related to this topic, another problem for me to solve then!

Thanks for the help!

You are welcome.

When notifications are executed successfully and Icinga shows that, next debugging step would be your local mail log. I also try to use a valid sender address so I get any bounce or similar errors in one mailbox for easier debugging.

I get this in the maillog:

NOQUEUE: SYSERR(icinga): can not chdir(/var/spool/clientmqueue/): Permission denied

This could also be SELinux-related, so if you can provide again the audit.log or audit2allow output, I will have I look into it.

The audit.log:

type=USER_ACCT msg=audit(1619791038.015:639): pid=25944 uid=995 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:accounting grantors=pam_unix,pam_localuser acct="icinga" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=USER_CMD msg=audit(1619791038.015:640): pid=25944 uid=995 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='cwd="/" cmd="/usr/lib64/nagios/plugins/send-email-notification.sh" terminal=? res=success'
type=CRED_REFR msg=audit(1619791038.015:641): pid=25944 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=USER_START msg=audit(1619791038.054:642): pid=25944 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:session_open grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_krb5 acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=AVC msg=audit(1619791038.109:643): avc:  denied  { dac_override } for  pid=25948 comm="send-email-noti" capability=1  scontext=system_u:system_r:nagios_notification_plugin_t:s0 tcontext=system_u:system_r:nagios_notification_plugin_t:s0 tclass=capability permissive=0
type=AVC msg=audit(1619791038.109:643): avc:  denied  { dac_read_search } for  pid=25948 comm="send-email-noti" capability=2  scontext=system_u:system_r:nagios_notification_plugin_t:s0 tcontext=system_u:system_r:nagios_notification_plugin_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1619791038.109:643): arch=c000003e syscall=2 success=no exit=-13 a0=2640c10 a1=441 a2=1b6 a3=fffffff0 items=0 ppid=25944 pid=25948 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="send-email-noti" exe="/usr/bin/bash" subj=system_u:system_r:nagios_notification_plugin_t:s0 key=(null)
type=PROCTITLE msg=audit(1619791038.109:643): proctitle=2F62696E2F62617368002F7573722F6C696236342F6E6167696F732F706C7567696E732F73656E642D656D61696C2D6E6F74696669636174696F6E2E7368
type=AVC msg=audit(1619791038.110:644): avc:  denied  { dac_override } for  pid=25948 comm="send-email-noti" capability=1  scontext=system_u:system_r:nagios_notification_plugin_t:s0 tcontext=system_u:system_r:nagios_notification_plugin_t:s0 tclass=capability permissive=0
type=AVC msg=audit(1619791038.110:644): avc:  denied  { dac_read_search } for  pid=25948 comm="send-email-noti" capability=2  scontext=system_u:system_r:nagios_notification_plugin_t:s0 tcontext=system_u:system_r:nagios_notification_plugin_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1619791038.110:644): arch=c000003e syscall=2 success=no exit=-13 a0=2640c10 a1=401 a2=1b6 a3=fffffff0 items=0 ppid=25944 pid=25948 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="send-email-noti" exe="/usr/bin/bash" subj=system_u:system_r:nagios_notification_plugin_t:s0 key=(null)
type=PROCTITLE msg=audit(1619791038.110:644): proctitle=2F62696E2F62617368002F7573722F6C696236342F6E6167696F732F706C7567696E732F73656E642D656D61696C2D6E6F74696669636174696F6E2E7368
type=USER_END msg=audit(1619791038.231:645): pid=25944 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:session_close grantors=pam_keyinit,pam_keyinit,pam_limits,pam_systemd,pam_unix,pam_krb5 acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'
type=CRED_DISP msg=audit(1619791038.232:646): pid=25944 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:icinga2_t:s0 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'

Ok, this one is not trivial. I am not sure why it wants permissions dac_override and dac_read_search, this would be overriding and ignoring the normal linux permissions for some reason. This is something I will not include in the default policy as it is “very risky”. So I would look into the normal permissions if I can fix it. If not add a local policy allowing it or set the context to nagios_unconfined_plugin_exec_t which should allow eveerything (but not sure for these permissions). And as work-around you can tell the context to run permissive semanage permissive -a nagios_notification_plugin_t.

Ok, so in order not to push more permissions for this script, could I do something similar with the default script?
I have tried to send html with the mail or mailx commands but doesn’t seem to work. Is it possible to do so?
(It’s a different topic I know)

I think it is somehow possible but not so easy with the version on CentOS, so perhaps a new thread where some others look into would be good.

Thank you very much for your help anyway!