Email Notifications - nothing is sent to user

Hi,
I’m attempting to setup email notifications through Director, unfortunately my configuration doesn’t want to send mail through to our relay.

Icinga2 is sat on Ubuntu. I’ve also installed mailtools/postfix on the same server.

ii  libmailutils5:amd64                 1:3.4-1
ii  mailutils                                   1:3.4-1        
ii  mailutils-common                    1:3.4-1         
ii  mime-support  
ii  postfix 

I’ve tested sending via the command line
mail -s "Test Email" emailaddr@mydomain.com < /dev/null which worked.

I’ve even managed to get email to local mail store on the server, not sure how.

In Director I’ve setup templates for and used Timeperiods, User, and Notifications.

zones.d/director-global/user_templates.conf
template User "MIS User" {
    enable_notifications = true
    states = [ Critical, Down, OK, Up, Warning ]
    types = [
        Acknowledgement,
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        Problem,
        Recovery
    ]
    groups = [ "MIS" ]
}
zones.d/cyllene/users.conf
object User "myname" {
    display_name = "my name"
    email = "email@fqdn. com"
    enable_notifications = true
    states = [ Critical, Down, OK, Up, Warning ]
    types = [
        Acknowledgement,
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        Problem,
        Recovery
    ]
    groups = [ "MIS" ]
}
## zones.d/director-global/timeperiod_templates.conf
template TimePeriod "Week Core Hours" {
      import "[legacy-timeperiod](https://cyllene.fqdn/icingaweb2/director/timeperiod?name=legacy-timeperiod)" 
       display_name = "Week Core Hours" 
       ranges = { "monday, tuesday, wednesday, thursday, friday" = "08:00-18:00" } }
## zones.d/cyllene/notification_templates.conf

template Notification "template name" { 
          command = "[mail-service-notification](https://cyllene.fqdncom/icingaweb2/director/command?name=mail-service-notification)" 
          interval = 15s 
          period = "Week Core" 
          states = [ Critical, OK, Warning ] 
          types = [ Acknowledgement, Custom, DowntimeEnd, DowntimeRemoved, DowntimeStart, Problem, Recovery ] 
          users = [ "myname" ] }
## zones.d/cyllene/notification_apply.conf

apply Notification "Test notification Hosts" to Host { 
     command = "[mail-service-notification](https://cyllene.fqdn.com/icingaweb2/director/command?name=mail-service-notification)" 
     interval = 15s 
     period = "Week Core" 
     assign where host.enable_notifications != "false" 
     states = [ Down, Up ] 
     types = [ Acknowledgement, Custom, Problem, Recovery ] 
     users = [ "myname" ] }

I’ve enabled debug and can see the following log msg.

[information/Checkable: Checkable 'hostname.fqdn.com' has 1 notification(s). Checking filters for type 'Custom', sends will be logged.

The host has notifications enabled, but nothing is sent.

I think I’ve worked out what might be missing but I can’t workout how to fix it.
My Icinga server which is also being monitored has the following listed on hosts parameters

Notifications
Notifications	Send notification
Contacts	Icinga 2 Admin, MyName
Contactgroups	Icinga 2 Admin Group

I can’t seem to apply contacts to the other hosts which I’ve setup via Director… Some help would be appreciated.

I’ve solved the problem for the hosts notifications by adding the Assign Where filter, I’ve tried repeating for the service notification, but the assignment doesnt work and the listed contacts for the services remain blank.

Now solved the assignment for the service, the contacts are now list against the service, however in the history Notifications are left as ```Not sent out to any contact```` I’m totally stuck as to why they aren’t being sent to the contacts. Host notifications work.

Could someone please offer some advice. Thanks

CAn you show a screenshot of the history? Or tell us on what states/types the “Not sent out to any contact” is showing?

Also: Do you really want to send notifications in a 15 second interval?

1 Like

15secs? No I hadn’t spotted it was in seconds, i’ll adjust that for sure.


|Type|Notification|
| --- | --- |
|Start time|2020-01-23 15:29:58|
|End time|2020-01-23 15:29:58|
|Reason|Normal notification|
|State|CRITICAL|
|Escalated|No|
|Contacts notified|0|
|Output|SERVICE "W32Time" CRITICAL NOT RUNNING|

All values are i seconds by default. You can use s, m, h, d to specify seconds, minutes, hours, days explicitly.

As to you posted history:
How does the template/apply rule for the service notification look like?
Did you check the log file /var/log/icinga2/icinga2.log around the time the notification should have been sent? What does it say?
Maybe even enable the debug log.

Also what I missed yesterday:
You have defined the mail-service-notification command for your host notifications

And also also:
What is this?! :thinking:
command = "[mail-service-notification](https://cyllene.fqdn.com/icingaweb2/director/command?name=mail-service-notification)"
import "[legacy-timeperiod](https://cyllene.fqdn/icingaweb2/director/timeperiod?name=legacy-timeperiod)"
Why are there URLs in the command/import definition? Is this valid, never seen that!?

1 Like

This is my current configuration, I think the confusion above is resolved.
I have also got the debug enabled but it doesn’t show anything in the log, I case as the system doesnt want to send a message in the first place. The email infrastructure is fine, as host alerts work.

zones.d/cyllene/users.conf
object User "MyUser" {
    import "MisUser"

    display_name = "Full Name"
    email = "emailaddr@fqdn.com"
    groups = [ "misGroup" ]
}

__

zones.d/cyllene/usergroups.conf
object UserGroup "misGroup" {
    display_name = "msiGroup"
}

__

zones.d/cyllene/notification_templates.conf
template Notification "MIS_Gen_Services" {
    command = "mail-service-notification"
    user_groups = [ "misGroup" ]
    users = [ "Myuser", "MIS" ]
}

__

zones.d/cyllene/notification_apply.conf
apply Notification "MIS_MainNotification_Service" to Service {
    import "MIS_Gen_Services"

    assign where service.name
    states = [ Critical, OK, Warning ]
    types = [
        Acknowledgement,
        Custom,
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        FlappingEnd,
        FlappingStart,
        Problem,
        Recovery
    ]
    user_groups = [ "misGroup" ]
    users = [ "Myuser", "MIS" ]
}

I can’t find anything wrong, assuming that your config objects, where you replaced the names are all correct in the real config.
You have so many differently written objects, I’m a bit confused.
User: MyUser, Myuser, myname, MIS
User template: MIS User, MisUser

You did not, by accident, disable the notifications inside the service template? Not sure atm if this would be indicated on the webinterface

1 Like

You have so many differently written objects, I’m a bit confused.

Why do you say this?

According to the Icinga2 web interface the services are enabled for Notification. :weary:

Because the naming in the config snippets you posted is not consistent and the config is case-sensitive.
But as I said, I assume you copy&pasted the snippets from the Icinga Director preview and renamed parts not suitable for the public.

I can’t see anything wrong with the configs you posted.

1 Like

Could this be a Director bug. I’ve just noticed a service email from my Icinga server which isn’t configured through Director…

Hm, do you still have the conf.d directory included in your icinga2.conf?
Check, if there is a line like
// include_recursive "conf.d"
which incates that the directory is not included. Normally this will be excluded when setting up the master server with icinga2 node wizard and then answer y on the exclusion question at the end.

Example how one of my /etc/icinga2/icinga2.conf looks like:

// Disabled by the node setup CLI command on 2020-01-15 10:57:19 +0100
// include_recursive "conf.d"
// Added by the node setup CLI command on 2020-01-15 10:57:19 +0100
include "conf.d/api-users.conf"
include "conf.d/commands.conf"

Both files, api-users.conf and commands.conf, could be placed somewhere else and then included with their full path as well.

1 Like

You might be correct, not sure I want to change that at the moment incase in breaks other stuff, its been a bun fight to get this far.

My hosts are notifying too, and the config for that seems to be in the same files.

Notification "MIS_MainNotification_Host" to Host {
    import "MIS_General_Host"

    assign where host.address || host.enable_notifications
    states = [ Down, Up ]
    types = [
        Acknowledgement,
        Custom,
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        FlappingEnd,
        FlappingStart,
        Problem,
        Recovery
    ]
    user_groups = [ "misGroup" ]
    users = [ "MIS" ]
}

apply Notification "MIS_MainNotification_Service" to Service {
    import "MIS_Gen_Services"

    assign where service.name
    states = [ Critical, OK, Warning ]
    types = [
        Acknowledgement,
        Custom,
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        FlappingEnd,
        FlappingStart,
        Problem,
        Recovery
    ]
    user_groups = [ "misGroup" ]
    users = [ "MIS" ]
}

Should be safe to exclude all but the api-users and commands file, like posted above.
Try it, do a icinga2 daemon -C on the command line to check for config errors.
If there are none, restart icinga2 and check your notifications again.

You will maybe have to add you icinga2 host via the director again, as this is part of the conf files as a example host as well.

I’ve made the change and checked the config, all good. Still not service emails… :weary:

icinga2 daemon -C
[2020-01-24 11:00:54 +0000] information/cli: Icinga application loader (version: r2.11.2-1)
[2020-01-24 11:00:54 +0000] information/cli: Loading configuration file(s).
[2020-01-24 11:00:54 +0000] information/ConfigItem: Committing config item(s).
[2020-01-24 11:00:54 +0000] information/ApiListener: My API identity: cyllene
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 2 FileLoggers.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 637 Notifications.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 17 HostGroups.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 36 Hosts.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 2 Comments.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 27 Zones.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 25 Endpoints.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 2 ApiUsers.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 235 CheckCommands.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 1 TimePeriod.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 2 Users.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 601 Services.
[2020-01-24 11:00:55 +0000] information/ConfigItem: Instantiated 23 ServiceGroups.
[2020-01-24 11:00:55 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2020-01-24 11:00:55 +0000] information/cli: Finished validating the configuration file(s).

Check via the API if the attribute no_more_notifications for the services is set to true:

For host:
https://localhost:5665/v1/objects/notifications?filter=match(%22myhostname%22,%20host.name)
For service:
https://localhost:5665/v1/objects/notifications?filter=match(%22myservicename%22,%20service.name)

Just access those links in the browser, put in the icinga2 host ip and the desired host/service name, enter the root API credentials and check the output

1 Like

Looks good.

 { 
         "attrs":{ 
            "__name":"host.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service",
            "active":true,
            "command":"mail-service-notification",
            "command_endpoint":"",
            "ha_mode":0.0,
            "host_name":"host.fqdn.com",
            "interval":1800.0,
            "last_notification":1579863202.971238,
            "last_notification_result":null,
            "last_problem_notification":1579862510.601198,
            "name":"MIS_MainNotification_Service",
            "next_notification":1579873326.905912,
            "no_more_notifications":false,
            "notification_number":0.0,
            "notified_problem_users":[ 

            ],
            "original_attributes":null,
            "package":"director",
            "paused":false,
            "period":"OfficeHours",
            "service_name":"winservice Windows Remote Desktop Services",
            "source_location":{ 
               "first_column":1.0,
               "first_line":21.0,
               "last_column":60.0,
               "last_line":21.0,
               "path":"/var/lib/icinga2/api/packages/director/b2ea1f6e-5a39-46b9-8303-952cd18598f7/zones.d/cyllene/notification_apply.conf"
            },
            "states":[ 
               "Critical",
               "OK",
               "Warning"
            ],
            "templates":[ 
               "MIS_MainNotification_Service",
               "MIS_Gen_Services"
            ],
            "times":null,
            "type":"Notification",
            "types":[ 
               "Acknowledgement",
               "Custom",
               "DowntimeEnd",
               "DowntimeRemoved",
               "DowntimeStart",
               "FlappingEnd",
               "FlappingStart",
               "Problem",
               "Recovery"
            ],
            "user_groups":[ 
               "misGroup"
            ],
            "users":[ 
               "MIS"
            ],
            "vars":null,
            "version":0.0,
            "zone":"cyllene"
         },
         "joins":{ 

         },
         "meta":{ 

         },
         "name":"host.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service",
         "type":"Notification"
      },