Mail Notifications

Hey,
im trying to configure a Mail notification for my Icinga2 version 2.8.1 on Ubuntu 18.04.04 LTS.
I went with the “official” Guide, but i still got the Problem, that i won´t get a notification via Mail.
Surprisingly, i do get a Mail notification, when Icinga2 requires an update.

Postfix and bsd-mailx are already installed.

Here are my config files:

notifications.conf

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = host.vars.notification.mail.groups
  users = host.vars.notification.mail.users

  //interval = 2h

  //vars.notification_logtosyslog = true

  assign where host.vars.notification.mail
}

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = host.vars.notification.mail.groups
  users = host.vars.notification.mail.users

  //interval = 2h

  //vars.notification_logtosyslog = true

  assign where host.vars.notification.mail
}

users.conf

object User "icingaadmin" {
  import "generic-user"

  display_name = "Icinga 2 Admin"
  groups = [ "icingaadmins" ]

  enable_notifications = true
  states = [ OK, Warning, Critical ]
  types = [ Problem, Recovery ]
  email = "testmail@test.de"
}

object UserGroup "icingaadmins" {
  display_name = "Icinga 2 Admin Group"
}

hosts.conf

object Host "vCenterServer" {
        import "generic-host"
        address = "10.***.***.**"
        check_command = "hostalive"
        vars.notification["mail"] = {
        groups = [ "icingaadmins" ]
        users = [ "icingaadmin" ]
}
}

I´m still an apprentice and doesn´t have much experience with Ubuntu in general.
Does anyone have an idea, if i missed something or is my configuration just wrong?

Hi and welcome,
that’s a recurring question here in the forum. So e.g. take a look in this old threads:

Short Summary:
Icinga delivers two bash scripts for notifications. You can use this or you use your own one. Icinga triggers the notification script. So would you have to do is to check if these scripts are running correctly on your systems (maybe something is mising e.g. installation, configuration etc.)

Hey,
thank you for this fast reply.
i worked through many old threads and tried all different solutions, but it still doesn´t work quite right.

i checked the notification script and it seems to work correctly. Interestingly, when i send out a Notfication through Icinga Web with the “forced” option, then i do get a notification. But when i doesn´t use the forced option, i won´t get a Mail.

What does the icinga2.log say around the time the notifications should be sent?

[2020-07-01 10:50:10 +0200] information/ExternalCommandListener: Executing external command: [1593593410] SEND_CUSTOM_HOST_NOTIFICATION;zimbra.diginet.gmbh;0;admin;test
[2020-07-01 10:50:10 +0200] information/Checkable: Checking for configured notifications for object ‘zimbra.diginet.gmbh’
[2020-07-01 10:51:40 +0200] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’

and when i send it with the forced option

[2020-07-01 10:56:07 +0200] information/Checkable: Checking for configured notifications for object ‘zimbra.diginet.gmbh’
[2020-07-01 10:56:07 +0200] information/Notification: Sending ‘Custom’ notification ‘zimbra.diginet.gmbh!mail-icingaadmin’ for user ‘icingaadmin’
[2020-07-01 10:56:07 +0200] information/Notification: Completed sending ‘Custom’ notification ‘zimbra.diginet.gmbh!mail-icingaadmin’ for checkable ‘zimbra.diginet.gmbh’ and user ‘icingaadmin’.

Do you get notifications when the host goes down?
Simply change the IP to something not reachable and test it.

unfortunately no.

I tested it by shutting down the host:

[2020-07-01 11:41:06 +0200] information/Checkable: Checking for configured notifications for object ‘zimbra.diginet.gmbh’
[2020-07-01 11:41:40 +0200] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’

Please show the output of
icinga2 object list --type notification --name 'vCenterServer!mail-icingaadmin'

Object ‘vCenterServer!mail-icingaadmin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:45

  • __name = “vCenterServer!mail-icingaadmin”
  • command = “mail-host-notification”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 47:3-47:36
  • command_endpoint = “”
  • host_name = “vCenterServer”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:45
  • interval = 1800
  • name = “mail-icingaadmin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:45
  • period = “24x7”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 60:3-60:17
  • service_name = “”
  • source_location
    • first_column = 1
    • first_line = 11
    • last_column = 45
    • last_line = 11
    • path = “/etc/icinga2/conf.d/notifications.conf”
  • states = [ “Up”, “Down” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 49:3-49:23
  • templates = [ “mail-icingaadmin”, “mail-host-notification” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:45
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 46:1-46:46
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd”, “DowntimeStart”, “DowntimeEnd”, “DowntimeRemoved” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 50:3-52:57
  • user_groups = [ “icingaadmins” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 13:3-13:50
  • users = [ “icingaadmin” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 14:3-14:43
  • vars
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 54:3-58:3
    • notification_logtosyslog = true
  • zone = “”

Hmmm…
all looks ok so far…

Please also post what a call to the API shows:

https://yourmonitoringserver:5665/v1/objects/notifications?filter=match("vCenterServer", host.name)
When prompted for the login take the root user from /etc/icinga2/conf.d/api-users.conf

{“results”:[{“attrs”:{"__name":“vCenterServer!ping4!mail-icingaadmin”,“active”:true,“command”:“mail-service-notification”,“command_endpoint”:"",“ha_mode”:0.0,“host_name”:“vCenterServer”,“interval”:1800.0,“last_notification”:0.0,“last_problem_notification”:0.0,“name”:“mail-icingaadmin”,“next_notification”:1593603009.3741290569,“no_more_notifications”:false,“notification_number”:0.0,“notified_problem_users”:[],“original_attributes”:null,“package”:"_etc",“paused”:false,“period”:“24x7”,“service_name”:“ping4”,“source_location”:{“first_column”:1.0,“first_line”:23.0,“last_column”:48.0,“last_line”:23.0,“path”:"/etc/icinga2/conf.d/notifications.conf"},“states”:[“OK”,“Warning”,“Critical”,“Unknown”],“templates”:[“mail-icingaadmin”,“mail-service-notification”],“times”:null,“type”:“Notification”,“types”:[“Problem”,“Acknowledgement”,“Recovery”,“Custom”,“FlappingStart”,“FlappingEnd”,“DowntimeStart”,“DowntimeEnd”,“DowntimeRemoved”],“user_groups”:[“icingaadmins”],“users”:[“icingaadmin”],“vars”:{“notification_logtosyslog”:true},“version”:0.0,“zone”:""},“joins”:{},“meta”:{},“name”:“vCenterServer!ping4!mail-icingaadmin”,“type”:“Notification”},{“attrs”:{"__name":“vCenterServer!mail-icingaadmin”,“active”:true,“command”:“mail-host-notification”,“command_endpoint”:"",“ha_mode”:0.0,“host_name”:“vCenterServer”,“interval”:1800.0,“last_notification”:1593598687.3699960709,“last_problem_notification”:0.0,“name”:“mail-icingaadmin”,“next_notification”:1593603009.3741400242,“no_more_notifications”:false,“notification_number”:0.0,“notified_problem_users”:[],“original_attributes”:null,“package”:"_etc",“paused”:false,“period”:“24x7”,“service_name”:"",“source_location”:{“first_column”:1.0,“first_line”:11.0,“last_column”:45.0,“last_line”:11.0,“path”:"/etc/icinga2/conf.d/notifications.conf"},“states”:[“Up”,“Down”],“templates”:[“mail-icingaadmin”,“mail-host-notification”],“times”:null,“type”:“Notification”,“types”:[“Problem”,“Acknowledgement”,“Recovery”,“Custom”,“FlappingStart”,“FlappingEnd”,“DowntimeStart”,“DowntimeEnd”,“DowntimeRemoved”],“user_groups”:[“icingaadmins”],“users”:[“icingaadmin”],“vars”:{“notification_logtosyslog”:true},“version”:0.0,“zone”:""},“joins”:{},“meta”:{},“name”:“vCenterServer!mail-icingaadmin”,“type”:“Notification”}]}

I don´t know if this is somehow related to this problem, but when i try to add a ressource to the icingadirector, i always get the message No CURL extension detected, it must be installed and enabled (RestApiClient.php:84)

Even though i installed the php curl extension and checked the php.ini if its enabled.

Did you retstart the apache2 service after installing?

Also what I see just now:

Is this really the icinga2 version (icinga2 --version?)
If so, please update to 2.11.4 so you are not using an outdated version.

Yes, and tried different php curl versions

i´ll try it. Can i just upgrade it, or do i need to consider anything else, before the upgrade? (except doing a backup of course :stuck_out_tongue: )

https://icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/
There everything that is necessary when/prior to updating is mentioned.

I saw @log1c wrote you some tips already.
in addition to this I have following questions/notes:

  • Did you test the notification script with the icinga, your or root-user?
  • You wrote you are using Ubuntu. I am not familiar with it, we are using CentOS. But we have sometimes problems with SELinux, maybe if you are using AppAmor, this is blocking something?
1 Like

@stevie-sy

  • i tested the script as root-user

@log1c
i couldn´t upgrade it because of corrupted packages, so i reinstalled the icinga. Now it´s on the newest Version, but the same Problem still exists

Maybe a stupid idea: did you enable notification? What is the output of icinga2 feature list?

It´s enabled, otherwise it wouldn´t even send out a forced notficiation:

Disabled features: compatlog debuglog elasticsearch gelf graphite influxdb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification

Hm… I’m pretty much out of ideas.

Please enable the debug log and test the notifications again (manually, manually forced and “automatic” via host down) and check the log.