Problem getting host/service notifications with Icinga Director

Good afternoon,

I’ve been trying for a while to configure the Notifications with Icinga Director using Marianne M. Spiller’s tutorial and scripts (https://www.unixe.de/icinga2-director-notifications/), but I still didn’t achieve to get the notifications automatically. I tried many other suggestions from different posts, FAQs, books, etc., but I still could not find out where the problem could be.

As you can see I already created the timeperiod template, the timeperiod, the user template, my own user, the notification template and the notifications. I am using Debian 10 with Icinga Web 2 Version 2.8.2 and the latest version of Icinga Director.

Time Period Template

zones.d/director-global/timeperiod_templates.conf

template TimePeriod “Generic Time Period Template” {
import “legacy-timeperiod”
includes = [ “Always” ]
ranges = {
“friday” = “00:00-24:00”
“monday” = “00:00-24:00”
“saturday” = “00:00-24:00”
“sunday” = “00:00-24:00”
“thursday” = “00:00-24:00”
“tuesday” = “00:00-24:00”
“wednesday” = “00:00-24:00”
}
}

Time Period

zones.d/director-global/timeperiods.conf

object TimePeriod “Always” {
import “legacy-timeperiod”
import “Generic Time Period Template”

display_name = "Always"
ranges = {
    "friday"	        = "00:00-24:00"
    "monday"	= "00:00-24:00"
    "saturday"	= "00:00-24:00"
    "sunday"	        = "00:00-24:00"
    "thursday"	= "00:00-24:00"
    "tuesday"	= "00:00-24:00"
    "wednesday"	= "00:00-24:00"
}

}

User Template

zones.d/director-global/user_templates.conf

template User “User Template” {
enable_notifications = true
period = “Always”
states = [ Critical, Down, OK, Unknown, Up, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
}

My User

zones.d/Master/users.conf

object User “Javier” {
import “User Template”

display_name = "javier"
email = "my e-mail address"
enable_notifications = true
period = "Always"

}

Host Notification Template

zones.d/Master/notification_templates.conf

template Notification “Host Alarm By Email Template” {
command = “host alarm e-mail”
interval = 15s
period = “Always”
states = [ Critical, Down, OK, Unknown, Up, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
users = [ “Javier” ]
}

Service Notification Template

zones.d/Master/notification_templates.conf

template Notification “Service Alarm By Email Template” {
command = “service alarm e-mail”
interval = 15s
period = “Always”
states = [ Critical, OK, Unknown, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
users = [ “Javier” ]
}

Host Notification

zones.d/Master/notification_apply.conf

apply Notification “Host Alarm By Email” to Host {
import “Host Alarm By Email Template”

assign where host.enable_notifications == "true" || host.vars.notification == "mail"
users = [ "Javier" ]

}

Service Notification

zones.d/Master/notification_apply.conf

apply Notification “Service Alarm By Email” to Service {
import “Service Alarm By Email Template”

assign where service.enable_notifications == "true" || service.vars.notification == "mail" || host.vars.os == "Linux"
users = [ "Javier" ]

}

Scripts

object NotificationCommand “host alarm e-mail” {
import “plugin-notification-command”
command = [ PluginDir + “/etc/icinga2/scripts/host-by-mail.sh” ]
arguments += {
“-4” = {
required = true
value = “$address$”
}
“-6” = {
required = true
value = “$address6$”
}
“-d” = {
required = true
value = “$icinga.long_date_time$”
}
“-l” = {
required = true
value = “$host.name$”
}
“-n” = {
required = true
value = “$host.display_name$”
}
“-o” = {
required = true
value = “$host.output$”
}
“-r” = {
required = true
value = “$user.email$”
}
“-s” = {
required = true
value = “$host.state$”
}
“-t” = {
required = true
value = “$notification.type$”
}
}
}

object NotificationCommand “service alarm e-mail” {
import “plugin-notification-command”
command = [ PluginDir + “/etc/icinga2/scripts/service-by-mail.sh” ]
arguments += {
“-4” = {
required = true
value = “$address$”
}
“-6” = “$address6$”
“-d” = {
required = true
value = “$icinga.long_date_time$”
}
“-e” = {
required = true
value = “$service.name$”
}
“-l” = {
required = true
value = “$host.name$”
}
“-n” = {
required = true
value = “$host.display_name$”
}
“-o” = {
required = true
value = “$service.output$”
}
“-r” = {
required = true
value = “$user.email$”
}
“-s” = {
required = true
value = “$service.state$”
}
“-t” = {
required = true
value = “$notification.type$”
}
“-u” = {
required = true
value = “$service.display_name$”
}
}
}

icinga2 daemon -C Output

[2020-10-14 14:04:20 +0200] information/cli: Icinga application loader (version: r2.12.0-1)
[2020-10-14 14:04:20 +0200] information/cli: Loading configuration file(s).
[2020-10-14 14:04:20 +0200] warning/config: Ignoring directory ‘/etc/icinga2/zones.d/master’ for unknown zone ‘master’.
[2020-10-14 14:04:20 +0200] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/NodeName’ for unknown zone ‘NodeName’.
[2020-10-14 14:04:20 +0200] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/mon01.akademie.bildendekunst.ac.at’ for unknown zone ‘mon01.akademie.bildendekunst.ac.at’.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Committing config item(s).
[2020-10-14 14:04:20 +0200] information/ApiListener: My API identity: mon01.akademie.bildendekunst.ac.at
[2020-10-14 14:04:20 +0200] warning/ApplyRule: Apply rule ‘Host Alarm By Email’ (in /var/lib/icinga2/api/packages/director/05940512-2888-4d33-aff3-b00647ad6031/zones.d/Master/notification_apply.conf: 1:0-1:47) for type ‘Notification’ does not match anywhere!
[2020-10-14 14:04:20 +0200] warning/ApplyRule: Apply rule ‘Service Alarm By Email’ (in /var/lib/icinga2/api/packages/director/05940512-2888-4d33-aff3-b00647ad6031/zones.d/Master/notification_apply.conf: 8:1-8:54) for type ‘Notification’ does not match anywhere!
[2020-10-14 14:04:20 +0200] warning/ApplyRule: Apply rule ‘swap’ (in /var/lib/icinga2/api/packages/director/05940512-2888-4d33-aff3-b00647ad6031/zones.d/director-global/service_apply.conf: 121:1-121:20) for type ‘Service’ does not match anywhere!
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 ElasticsearchWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 122 Hosts.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 3 Zones.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 GelfWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 OpenTsdbWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 CompatLogger.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 288 CheckCommands.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 LivestatusListener.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 User.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 496 Services.
[2020-10-14 14:04:20 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2020-10-14 14:04:20 +0200] information/cli: Finished validating the configuration file(s).

Please let me know if you need any further information.

Thank you for your help.

Best regards,

Javier

Hello,

It’s been a week since my initial post and I wanted to ask if you have any suggestion. Please let me know if you need any further information.

Thank you in advance and best regards,

Javier

Hi,

pls be patient. The most of us are helping in the community either if there is a time window during work or in the leisure time. Also if someone has a solution or idea, because he/she struggled in a similar problem in the own company or still has the knowledge about this topic. :wink:

Now to your question:
Do you see in the host/service history in icingaweb2 if icinga is trying to send a mail or give you a hint why it’s not working? Do you see something in the icinga2.log? e.g. an error message like the mail program is missing, not configured or working well or the script is skipping with an error?
Does your mail script working on command line?

Hi Stevie,

thank you very much for your answer. It doesn’t look like Icinga is trying to send a mail automatically, although we have quite often a service/host problem and the whole configuration looks alright to me. As you can see from my enclosed screenshots, the “Notifications” section is completely empty. It doesn’t give me neither a hint and I cannot really see why it’s not working:

I’ve been also having a look at the logs very often just to look for a hint, but I didn’t really find anything related to the notifications. I only get something related to it if I send manually a custom/forced notification directly from Icingaweb2.

The “notification” icinga2 feature is also enabled (actually everything is enabled):

Please find enclosed the following outputs:

tail -f /var/log/icinga2/icinga2.log

tail -f /var/log/icinga2/debug.log | grep "notice/Process"

The mail scripts work well from the command line. I tried this a few times and I got always the e-mails:

./service-by-mail.sh -4 ‘127.0.0.1’ -r ‘my e-mail’
./host-by-mail.sh -4 ‘127.0.0.1’ -r ‘my e-mail’

image

Please let me know if you need any further information.

Thank you again for your help.

Best regards,

Javier

Hi

I meant the history tab in at the host object. if the host or service is in a non ok state you should see if the notification rule is working:
e.g.


here is no notification sent, because nobody in our company wants to have “unkown”-messages.
image

In the last time there were some threads with a similar problems. Maybe they could help:

If you can send a forced notification via icingaweb2 it couldn’t be something like SELinux or wrong file permisisons. the same if you can send messages via the bash. I think something is wrong with the apply rule. In the first link is a way how to check if the rules are applied

Hello Stevie,

thank you very much for your answer and the links you sent me. I am still looking for the problem.

Well, there are a few hosts and services failing from time to time, but if I have a look at the history tab (as you wrote last time), I cannot see anything related to the notifications (looks like I am not even able to send a forced notification via icingaweb2):

I checked many host failing in the past and it looked always the same. The forced comments seem to work (I don’t receive any e-mails or notifications for this neither) and I am still able to send (test) messages via the bash (so I assume the scrips are working), but there is no trace of notifications.

I checked the whole configuration and did not really find anything wrong. What do you think about this “assigns”?

Assign where
OR
host.enable_notifications = true
host.vars.notification.mail = mail
host.vars.notification = mail

Assign where
OR
service.enable_notifications = true
service.vars.notification = mail
service.vars.os = Linux

Any further help will be greatly appreciated.

Best regards,

Javier

Hi again,

just for testing - delete these lines from your assign rules

It happens quickly to switch off this option in the host. Some times ago, it happend to us that wie switched off “active checks” and we were wondering the host wasn’t checked anymore. Small things with a big impact.
If you delete these lines you can rule out that this is possibly deactivated in your config.

I also didn’t find an example for a host object definition. So be sure that your hosts has these vars defined:

Hello Stevie,

thank you for your last reply.

I have removed the lines you suggested me from the assign rules and I’ve also checked all hosts and the notifications are everywhere activated. As l mentioned above, the “notification” feature from the icinga feature list is activated, but I am still not able to receive notifications.

In your last post you mentioned “be sure that your hosts has these vars defined”, but I am not really sure how to do that in Icinga Director. I found something similar in one of the links you posted last time (Need to setup Icinga2/Icingaweb2 to send email alerts only for ping service) but I did not find the way to set custom variables within Icinga Director:

Screenshot_2020-11-06 Need to setup Icinga2 Icingaweb2 to send email alerts only for ping service

Thank you again and have a nice weekend.

Best regards,

Javier

Hi,

how you define custom vars in the director is described here: https://icinga.com/docs/director/latest/doc/12-Handling-custom-variables/ and here with an here you’ll find an example with a custom vars for using as an Array for Interfaces: https://icinga.com/docs/director/latest/doc/14-Fields-example-interfaces-array/. Here is an example for using custom vars for snmp https://icinga.com/docs/director/latest/doc/16-Fields-example-SNMP/. The nice thing is that the principle is always very the same or similar.

Hello Stevie,

thank you for your last message. Sorry that I could not reply a bit earlier.

I am still facing the same issue and I am still not receiving the notifications.

Following the information from the links you gave me, I defined the “notification” data field:

And afterwards changed my host templates, where I add this new field:

These are the custom vars defined:

mail = host.vars.notification
mail = service.vars.notification
mail = host.vars.notification.mail
mail = service.vars.notification.mail

If I deploy everything, I choose a certain host and go to preview, I don’t see any changes there:

Screenshot_2021-01-15 Icinga Web(2)

Do I am still missing something?

Thank you and regards,

Javier

Hi,
you set the field “notification” as Array. I don’t know if this work. I never tried it. And also I never tried it in that way you configured it. Maye others know more …

Hello Stevie,

well, I did it following the links you sent me last time (specially https://icinga.com/docs/icinga-director/latest/doc/14-Fields-example-interfaces-array/), but since it still does not work there must be another way to do that.

As I already said, this is the first time I am configuring the notifications and I would appreciate some further help according to the information I have enclosed. I searched the web already for weeks but unfortunately I could not find the way to get this done and working.

Thank you in advance for your help and best regards,

Javier

Having only quickly skipped through the thread it left me quite confused on the current state of your configuration, so I will try to give some basic tips.

Though the tutorial by unixe is good, it is also old and outdated.
The Director should have the default icinga2 notifications commands under “external commands” after the kickstart.
Nevertheless the other commands and scripts should work as well.

Why is everything enabled? I don’t think you are using every feature.
Though having everything enabled might not be problematic (and has nothing to do with the notifications) it still could make problems later. Like huge debug logfiles, unnecessary perfdata files clogging the disk and more.

This does not make sense, because you have configured the fields “mail” to only be displayed when the variable “mail” is set to the strings on the right side of the =.

What is you goal with host.vars.notification and host.vars.notification.mail?
Do you want to configure the contacts that should get sent the mail with these?

With your current “assign where” statements the notifications will only be deployed to hosts that have the string “mail” in either host.vars.notification or host.vars.notification.mail, and service notifications will only be deployed to services with the string “mail” in service.vars.notification or the string “Linux” in service.vars.os.

I assume that you currently have no notification deployed, because none of the variables are filled with what the assign rule expects.

I suggest you start fresh, use the “official” icinga2 notification commands (you maybe need to include the config file commands.conf under /etc/icinga2/conf.d in you /etc/icinga2/icinga2.conf), create notifications templates for both commands, create a user template and user that has your desired notifications states and types and then create a SIMPLE notification apply rule based on a host name and/or service name (“assign where host.name = myhostname”)

HI,

you mixed two things. The link you wrote in your answer is a part for the service checks to generate apply for rules from values from an array like in the docs with Interfaces.
So this array has nothing to do with the notification.

To configure notifications in the director you have to create the users here first:

After you can create notification rules here:

A notification rule could look like this:


You can assign specific user and/or groups.

Of course the notification command have to configured as well and the script behind work.

Hello log1c and Stevie,

thank you very much for your replies. A colleague of mine is going to carry on with the notification stuff.

We will let you know that anyway as soon as we get this working.

Best regards,

Javier