Icinga2 - After Debian Upgrade Mail Notification doesnt work

Hello Everyone,

we just upgraded our Debian from 10 to 11 because of the end of support. While the maintanance i also upgrade Icinga2, Icingaweb2 and Icinga Director. While some Troubleshooting, looks like all is working fine now.

Yesterday i found out, that our Mail Notifications isnt working since the upgrade. I checked some logs and couldnt find any help. Thats why im asking here.

Here is our system:

  • Debian 11
  • Icinga: r2.14.2-1
  • Icinga Director 1.11.1
  • Icinga Web 2 Version 2.12.1

Here is what i already checked:

  • icinga2 feature list
    Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 journald livestatus opentsdb perfdata statusdata syslog
    Enabled features: api checker command ido-mysql influxdb mainlog notification

  • Checked if MTA Postfix is working:
    echo ‘This is a test’ | mail -s ‘test’ mailadress
    → Mail is sent and received

  • Checked if Mail Script is working:
    ./mail-host-notification.sh -d test -l test -n test -o test -r mailadress -s test -t test
    → Mail is sent and received

But the Mail-Adress has changed in Outlook, i dont know why. Before the update i just saw “icinga@mydomain.de” as Sender. Now i see the username “icinga icinga@mydomain.de”.

When i trigger an Notification via Icingaweb i dont see any Logs in mail.log.

icinga2 object list --type=notification shows that Notification is beeing sent.
Our Teams Notifications are still working so im sure that the director confs (User, notification and so on) works correctly.

Do You have any suggestion?
Thanks!

Thanks for your post and the details you have provided. Would you mind checking the log file under /var/log/icinga2/icinga2.log for anything notification related?

Furthermore, can you try executing both the mail command and the mail-host-notification.sh script as your Icinga 2-User, depending on your setup this might be icinga, nagios, _icinga or something the like.

Thanks for your fast reply!

Nothing related in icinga2.log.
Both commands work fine with the icinga user.

Could you clarify nothing? There should be at least something like information/Notification: Sending […] and information/Notification: Completed sending […] in there.

Would you please also grep -r through other log files under /var/log/icinga2/ and check journalctl -u icinga2.service (or however your Icinga 2 systemd unit is named). Thanks.

If there is still nothing, could you please verify that the notifications are still configured and your config wasn’t messed up during the upgrade?

i will check the files on monday :slight_smile: have a nice weekend!

On icinga.log i can see the following entry when notifaction should be sent:

[2024-05-06 09:57:25 +0200] information/Checkable: Checkable 'myfw!Firewall Health' has 1 notification(s). Checking filters for type 'Custom', sends will be logged.
[2024-05-06 09:57:25 +0200] information/HttpServerConnection: Request POST /v1/actions/send-custom-notification (from [::ffff:icinga IP]:45068), user: root, agent: , status: OK) took 3ms.

journalctl -u icinga2.service looks good, i dont know what i need to check here.

I would expect some information/Notification log entries to be present, as written above.

You can check for existence of Notification and NotificationCommand objects by:

$ icinga2 object list -t Notification
$ icinga2 object list -t NotificationCommand

Those two object queries must return something. Please feel free to paste an exemplary (and redacted, if necessary) entry of one Notification object and the (redacted, if necessary) output for all NotificationCommands.

When i do the command: icinga2 object list -t NotificationCommand i can see on the bottom the following:

warning/cli: This data is 11 days, 6 hours and 47 seconds older than the last Icinga config (re)load. It may be outdated. Consider running ‘icinga2 daemon -C --dump-objects’ first.

I think exactly on this date (11 days before) it was the last time the mail notification worked.
What should i do here?
icinga2 daemon -C --dump-objects’ will this delete the old file?

The --dump-objects flag recreates /var/cache/icinga2/icinga2.{debug,vars} and shouldn’t do any harm. Maybe you wanna backup your old /var/cache/icinga2/icinga2.debug file and compare it to the recreated one.

Please note that the icinga2 object list command requires a synchronized /var/cache/icinga2/icinga2.debug file. That’s why it warns you if it has aged somewhat.

Output of icinga2 object list -t Notification:

Object 'ExampleSW!Switch SNMP Stack!Notify Me About Services' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 1:0-1:55
  * __name = "ExampleSW!Switch SNMP Stack!Notify Me About Services"
  * command = "mail-service-notification"
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_templates.conf', lines 10:5-10:41
  * command_endpoint = ""
  * host_name = "ExampleSW"
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 1:0-1:55
  * interval = 0
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_templates.conf', lines 11:5-11:17
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 4:5-4:17
  * name = "Notify Me About Services"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 1:0-1:55
  * period = "Immerzu"
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_templates.conf', lines 12:5-12:22
  * service_name = "Switch SNMP Stack"
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 1:0-1:55
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 55
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf"
  * states = [ "Critical", "OK", "Warning" ]
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 6:5-6:38
  * templates = [ "Notify Me About Services", "Generic Service Alarm By Email" ]
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 1:0-1:55
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_templates.conf', lines 9:1-9:54
  * times = null
  * type = "Notification"
  * types = [ "Acknowledgement", "Custom", "DowntimeStart", "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_templates.conf', lines 13:5-13:56
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 7:5-13:5
  * user_groups = null
  * users = [ "Cherry" ]
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_templates.conf', lines 14:5-14:38
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 14:5-14:38
  * vars = null
  * zone = "icinga.my.domain.de"
    % = modified in '/var/lib/icinga2/api/packages/director/b39f9863-2a47-4326-b498-2ce51111a3b0/zones.d/icinga.my.domain.de/notification_apply.conf', lines 1:0-1:55

Output of icinga2 object list -t NotificationCommand:

Object 'mail-host-notification' of type 'NotificationCommand':
  % declared in '/etc/icinga2/conf.d/commands.conf', lines 9:1-9:51
  * __name = "mail-host-notification"
  * arguments
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 12:3-52:3
    * -4 = "$notification_address$"
    * -6 = "$notification_address6$"
    * -X = "$notification_hostnotes$"
    * -b = "$notification_author$"
    * -c = "$notification_comment$"
    * -d
      * required = true
      * value = "$notification_date$"
    * -f
      * description = "Set from address. Requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE)"
      * value = "$notification_from$"
    * -i = "$notification_icingaweb2url$"
    * -l
      * required = true
      * value = "$notification_hostname$"
    * -n
      * required = true
      * value = "$notification_hostdisplayname$"
    * -o
      * required = true
      * value = "$notification_hostoutput$"
    * -r
      * required = true
      * value = "$notification_useremail$"
    * -s
      * required = true
      * value = "$notification_hoststate$"
    * -t
      * required = true
      * value = "$notification_type$"
    * -v = "$notification_logtosyslog$"
  * command = [ "/etc/icinga2/scripts/mail-host-notification.sh" ]
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 10:3-10:64
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "mail-host-notification"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 51
    * last_line = 9
    * path = "/etc/icinga2/conf.d/commands.conf"
  * templates = [ "mail-host-notification", "plugin-notification-command" ]
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 9:1-9:51
    % = modified in 'methods-itl.conf', lines 22:2-22:122
  * timeout = 60
  * type = "NotificationCommand"
  * vars
    % = modified in '/etc/icinga2/conf.d/commands.conf', lines 54:3-67:3
    * notification_address = "$address$"
    * notification_address6 = "$address6$"
    * notification_author = "$notification.author$"
    * notification_comment = "$notification.comment$"
    * notification_date = "$icinga.long_date_time$"
    * notification_hostdisplayname = "$host.display_name$"
    * notification_hostname = "$host.name$"
    * notification_hostnotes = "$host.notes$"
    * notification_hostoutput = "$host.output$"
    * notification_hoststate = "$host.state$"
    * notification_type = "$notification.type$"
    * notification_useremail = "$user.email$"
  * zone = ""

And Output of icinga2 object list -t NotificationCommand from teams (works):

Object 'teams-host-notification' of type 'NotificationCommand':
  % declared in '/var/lib/icinga2/api/packages/director/85f6dccf-c2cf-4b48-ac0f-1cf1eb4066c1/zones.d/icinga.my.domain.de/commands.conf', lines 771:1-771:52
  * __name = "teams-host-notification"
  * arguments
    % = modified in '/var/lib/icinga2/api/packages/director/85f6dccf-c2cf-4b48-ac0f-1cf1eb4066c1/zones.d/icinga.my.domain.de/commands.conf', lines 774:5-786:5
    * -4 = "$address$"
    * -d = "$icinga.long_date_time$"
    * -e = "$service.name$"
    * -l = "$host.name$"
    * -n = "$host.display_name$"
    * -o = "$host.output$"
    * -p = "Webook URL Teams"
    * -s = "$host.state$"
    * -t = "$notification.type$"
    * -u = "$service.display_name$"
    * -x = "$enable_teams_notifications$"
  * command = [ "/etc/icinga2/scripts/teams-host-notification.pl" ]
    % = modified in '/var/lib/icinga2/api/packages/director/85f6dccf-c2cf-4b48-ac0f-1cf1eb4066c1/zones.d/icinga.justiz.niedersachsen.de/commands.conf', lines 773:5-773:67
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "teams-host-notification"
  * package = "director"
  * source_location
    * first_column = 1
    * first_line = 771
    * last_column = 52
    * last_line = 771
    * path = "/var/lib/icinga2/api/packages/director/85f6dccf-c2cf-4b48-ac0f-1cf1eb4066c1/zones.d/icinga.my.domain.de/commands.conf"
  * templates = [ "teams-host-notification", "plugin-notification-command", "plugin-notification-command" ]
    % = modified in '/var/lib/icinga2/api/packages/director/85f6dccf-c2cf-4b48-ac0f-1cf1eb4066c1/zones.d/icinga.my.domain.de/commands.conf', lines 771:1-771:52
    % = modified in 'methods-itl.conf', lines 22:2-22:122
    % = modified in 'methods-itl.conf', lines 22:2-22:122
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = "icinga.my.domain.de"

I checked icinga2.log again and saw this time the following:

[2024-05-06 15:21:52 +0200] warning/PluginNotificationTask: Notification command for object ‘myfw’ (PID: 2442984, arguments: ‘/etc/icinga2/scripts/mail-host-notification.sh’ ‘-4’ ‘192.168.10.132’ ‘-6’ ‘’ ‘-X’ ‘’ ‘-b’ ‘User’ ‘-c’ ‘hallo’ ‘-d’ ‘2024-05-06 15:21:52 +0200’ ‘-l’ ‘myfw’ ‘-n’ ‘myfw’ ‘-o’ ‘OK - 192.168.10.132: rta 13.296ms, lost 0%’ ‘-r’ ‘myemail@mail.de’ ‘-s’ ‘UP’ ‘-t’ ‘true’) terminated with exit code 1, output: Illegal option -X
ERROR: Invalid option -

I dont know from where the “-X” parameter comes from. do you have any clue?

That’s a good catch! The -X parameter was introduced recently in 7f0aaf9 and was first released in v2.14.0.

If I had to guess, you have manually edited your mail-{host,service}-notification.sh script and thus the package update did not touch it. There might be some .dpkg-dist file next to the scripts.

Thus, you can diff your script against the .dpkg-dist file or the latest version from git. You could also consult the change itself. Unless there are custom changes to the script itself, you might wanna just replace your version with the one being shipped from your distribution, (hopefully) resulting in automatic updates in the future.

1 Like

I changed both script with the .dpkg-dist :slight_smile:
And now Host-Notification started to work again.
I think we changed the Script because the “-t” argument with true and false in the mail was a bit confusing for us.

Sadly, the Service Notification still doesnt work. I tested the script manually on the console and it worked. The Mail is beeing sent and received:
./mail-service-notification.sh -e test -u test -d test -l test -n test -o test -r mymail@mail.de -s test -t test

I also double checked the notification in director and both looks same, only different is host and service.

The icinga2.log is showing some difference:
Hosts:

[2024-05-07 14:15:29 +0200] information/ApiListener: New client connection from [::ffff:192.168.132.5]:43422 (no client certificate)
[2024-05-07 14:15:29 +0200] information/Checkable: Checkable 'myfw' has 2 notification(s). Checking filters for type 'Custom', sends will be logged.
[2024-05-07 14:15:29 +0200] information/Notification: Sending 'Custom' notification 'myfw!Notify Me About Host' for user 'User1'
[2024-05-07 14:15:29 +0200] information/Notification: Sending 'Custom' notification 'myfw!Notify Me About Host' for user 'User3'
[2024-05-07 14:15:29 +0200] information/Notification: Sending 'Custom' notification 'myfw!Notify Me About Host' for user 'User2'
[2024-05-07 14:15:29 +0200] information/HttpServerConnection: Request POST /v1/actions/send-custom-notification (from [::ffff:192.168.132.5]:43422), user: root, agent: , status: OK) took 0ms.
[2024-05-07 14:15:29 +0200] information/Notification: Completed sending 'Custom' notification 'myfw!Notify Me About Host' for checkable 'myfw' and user 'User1' using command 'mail-host-notification'.
[2024-05-07 14:15:29 +0200] information/Notification: Completed sending 'Custom' notification 'myfw!Notify Me About Host' for checkable 'myfw' and user 'User2' using command 'mail-host-notification'.
[2024-05-07 14:15:29 +0200] information/Notification: Completed sending 'Custom' notification 'myfw!Notify Me About Host' for checkable 'myfw' and user 'User3' using command 'mail-host-notification'.
[2024-05-07 14:15:29 +0200] information/HttpServerConnection: HTTP client disconnected (from [::ffff:192.168.132.5]:43422)

Service:

[2024-05-07 14:15:01 +0200] information/ApiListener: New client connection from [::ffff:192.168.132.5]:36994 (no client certificate)
[2024-05-07 14:15:01 +0200] information/Checkable: Checkable 'myfw!Firewall CPU Load' has 1 notification(s). Checking filters for type 'Custom', sends will be logged.
[2024-05-07 14:15:01 +0200] information/HttpServerConnection: Request POST /v1/actions/send-custom-notification (from [::ffff:192.168.132.5]:36994), user: root, agent: , status: OK) took 0ms.
[2024-05-07 14:15:02 +0200] information/HttpServerConnection: HTTP client disconnected (from [::ffff:192.168.132.5]:36994)

Do you have any idea? Thanks so far!!

I’m glad to hear this.

Could you please paste your icinga2 object list -t NotificationCommand snippet for the mail-service-notification?

Furthermore, could you please manually trigger a CUSTOM notification for some Service, e.g., by clicking Notification in Icinga Web’s service view? Please post the Icinga 2 log during this time.

P.S.: I took the liberty to reformat your posts to put the logs in code blocks to ease their readability.

Object ‘mail-service-notification’ of type ‘NotificationCommand’:
% declared in ‘/etc/icinga2/conf.d/commands.conf’, lines 70:1-70:54

  • __name = “mail-service-notification”
  • arguments
    % = modified in ‘/etc/icinga2/conf.d/commands.conf’, lines 73:3-122:3
    • -4 = “$notification_address$”
    • -6 = “$notification_address6$”
    • -X = “$notification_hostnotes$”
    • -b = “$notification_author$”
    • -c = “$notification_comment$”
    • -d
      • required = true
      • value = “$notification_date$”
    • -e
      • required = true
      • value = “$notification_servicename$”
    • -f
      • description = “Set from address. Requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE)”
      • value = “$notification_from$”
    • -i = “$notification_icingaweb2url$”
    • -l
      • required = true
      • value = “$notification_hostname$”
    • -n
      • required = true
      • value = “$notification_hostdisplayname$”
    • -o
      • required = true
      • value = “$notification_serviceoutput$”
    • -r
      • required = true
      • value = “$notification_useremail$”
    • -s
      • required = true
      • value = “$notification_servicestate$”
    • -t
      • required = true
      • value = “$notification_type$”
    • -u
      • required = true
      • value = “$notification_servicedisplayname$”
    • -v = “$notification_logtosyslog$”
    • -x = “$notification_servicenotes$”
  • command = [ “/etc/icinga2/scripts/mail-service-notification.sh” ]
    % = modified in ‘/etc/icinga2/conf.d/commands.conf’, lines 71:3-71:67
  • env = null
  • execute
    % = modified in ‘methods-itl.conf’, lines 23:3-23:30
    • arguments = [ “notification”, “user”, “cr”, “itype”, “author”, “comment”, “resolvedMacros”, “useResolvedMacros” ]
    • deprecated = false
    • name = “Internal#PluginNotification”
    • side_effect_free = false
    • type = “Function”
  • name = “mail-service-notification”
  • package = “_etc”
  • source_location
    • first_column = 1
    • first_line = 70
    • last_column = 54
    • last_line = 70
    • path = “/etc/icinga2/conf.d/commands.conf”
  • templates = [ “mail-service-notification”, “plugin-notification-command” ]
    % = modified in ‘/etc/icinga2/conf.d/commands.conf’, lines 70:1-70:54
    % = modified in ‘methods-itl.conf’, lines 22:2-22:122
  • timeout = 60
  • type = “NotificationCommand”
  • vars
    % = modified in ‘/etc/icinga2/conf.d/commands.conf’, lines 124:3-140:3
    • notification_address = “$address$”
    • notification_address6 = “$address6$”
    • notification_author = “$notification.author$”
    • notification_comment = “$notification.comment$”
    • notification_date = “$icinga.long_date_time$”
    • notification_hostdisplayname = “$host.display_name$”
    • notification_hostname = “$host.name$”
    • notification_hostnotes = “$host.notes$”
    • notification_servicedisplayname = “$service.display_name$”
    • notification_servicename = “$service.name$”
    • notification_servicenotes = “$service.notes$”
    • notification_serviceoutput = “$service.output$”
    • notification_servicestate = “$service.state$”
    • notification_type = “$notification.type$”
    • notification_useremail = “$user.email$”
  • zone = “”

I already trigger the custom Notification, thats the log you see above.

Thanks! Didnt know how to do that :slight_smile: