Problem getting host/service notifications with Icinga Director

Good afternoon,

I’ve been trying for a while to configure the Notifications with Icinga Director using Marianne M. Spiller’s tutorial and scripts (https://www.unixe.de/icinga2-director-notifications/), but I still didn’t achieve to get the notifications automatically. I tried many other suggestions from different posts, FAQs, books, etc., but I still could not find out where the problem could be.

As you can see I already created the timeperiod template, the timeperiod, the user template, my own user, the notification template and the notifications. I am using Debian 10 with Icinga Web 2 Version 2.8.2 and the latest version of Icinga Director.

Time Period Template

zones.d/director-global/timeperiod_templates.conf

template TimePeriod “Generic Time Period Template” {
import “legacy-timeperiod”
includes = [ “Always” ]
ranges = {
“friday” = “00:00-24:00”
“monday” = “00:00-24:00”
“saturday” = “00:00-24:00”
“sunday” = “00:00-24:00”
“thursday” = “00:00-24:00”
“tuesday” = “00:00-24:00”
“wednesday” = “00:00-24:00”
}
}

Time Period

zones.d/director-global/timeperiods.conf

object TimePeriod “Always” {
import “legacy-timeperiod”
import “Generic Time Period Template”

display_name = "Always"
ranges = {
    "friday"	        = "00:00-24:00"
    "monday"	= "00:00-24:00"
    "saturday"	= "00:00-24:00"
    "sunday"	        = "00:00-24:00"
    "thursday"	= "00:00-24:00"
    "tuesday"	= "00:00-24:00"
    "wednesday"	= "00:00-24:00"
}

}

User Template

zones.d/director-global/user_templates.conf

template User “User Template” {
enable_notifications = true
period = “Always”
states = [ Critical, Down, OK, Unknown, Up, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
}

My User

zones.d/Master/users.conf

object User “Javier” {
import “User Template”

display_name = "javier"
email = "my e-mail address"
enable_notifications = true
period = "Always"

}

Host Notification Template

zones.d/Master/notification_templates.conf

template Notification “Host Alarm By Email Template” {
command = “host alarm e-mail”
interval = 15s
period = “Always”
states = [ Critical, Down, OK, Unknown, Up, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
users = [ “Javier” ]
}

Service Notification Template

zones.d/Master/notification_templates.conf

template Notification “Service Alarm By Email Template” {
command = “service alarm e-mail”
interval = 15s
period = “Always”
states = [ Critical, OK, Unknown, Warning ]
types = [
Acknowledgement,
Custom,
DowntimeEnd,
DowntimeRemoved,
DowntimeStart,
FlappingEnd,
FlappingStart,
Problem,
Recovery
]
users = [ “Javier” ]
}

Host Notification

zones.d/Master/notification_apply.conf

apply Notification “Host Alarm By Email” to Host {
import “Host Alarm By Email Template”

assign where host.enable_notifications == "true" || host.vars.notification == "mail"
users = [ "Javier" ]

}

Service Notification

zones.d/Master/notification_apply.conf

apply Notification “Service Alarm By Email” to Service {
import “Service Alarm By Email Template”

assign where service.enable_notifications == "true" || service.vars.notification == "mail" || host.vars.os == "Linux"
users = [ "Javier" ]

}

Scripts

object NotificationCommand “host alarm e-mail” {
import “plugin-notification-command”
command = [ PluginDir + “/etc/icinga2/scripts/host-by-mail.sh” ]
arguments += {
“-4” = {
required = true
value = “$address$”
}
“-6” = {
required = true
value = “$address6$”
}
“-d” = {
required = true
value = “$icinga.long_date_time$”
}
“-l” = {
required = true
value = “$host.name$”
}
“-n” = {
required = true
value = “$host.display_name$”
}
“-o” = {
required = true
value = “$host.output$”
}
“-r” = {
required = true
value = “$user.email$”
}
“-s” = {
required = true
value = “$host.state$”
}
“-t” = {
required = true
value = “$notification.type$”
}
}
}

object NotificationCommand “service alarm e-mail” {
import “plugin-notification-command”
command = [ PluginDir + “/etc/icinga2/scripts/service-by-mail.sh” ]
arguments += {
“-4” = {
required = true
value = “$address$”
}
“-6” = “$address6$”
“-d” = {
required = true
value = “$icinga.long_date_time$”
}
“-e” = {
required = true
value = “$service.name$”
}
“-l” = {
required = true
value = “$host.name$”
}
“-n” = {
required = true
value = “$host.display_name$”
}
“-o” = {
required = true
value = “$service.output$”
}
“-r” = {
required = true
value = “$user.email$”
}
“-s” = {
required = true
value = “$service.state$”
}
“-t” = {
required = true
value = “$notification.type$”
}
“-u” = {
required = true
value = “$service.display_name$”
}
}
}

icinga2 daemon -C Output

[2020-10-14 14:04:20 +0200] information/cli: Icinga application loader (version: r2.12.0-1)
[2020-10-14 14:04:20 +0200] information/cli: Loading configuration file(s).
[2020-10-14 14:04:20 +0200] warning/config: Ignoring directory ‘/etc/icinga2/zones.d/master’ for unknown zone ‘master’.
[2020-10-14 14:04:20 +0200] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/NodeName’ for unknown zone ‘NodeName’.
[2020-10-14 14:04:20 +0200] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/mon01.akademie.bildendekunst.ac.at’ for unknown zone ‘mon01.akademie.bildendekunst.ac.at’.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Committing config item(s).
[2020-10-14 14:04:20 +0200] information/ApiListener: My API identity: mon01.akademie.bildendekunst.ac.at
[2020-10-14 14:04:20 +0200] warning/ApplyRule: Apply rule ‘Host Alarm By Email’ (in /var/lib/icinga2/api/packages/director/05940512-2888-4d33-aff3-b00647ad6031/zones.d/Master/notification_apply.conf: 1:0-1:47) for type ‘Notification’ does not match anywhere!
[2020-10-14 14:04:20 +0200] warning/ApplyRule: Apply rule ‘Service Alarm By Email’ (in /var/lib/icinga2/api/packages/director/05940512-2888-4d33-aff3-b00647ad6031/zones.d/Master/notification_apply.conf: 8:1-8:54) for type ‘Notification’ does not match anywhere!
[2020-10-14 14:04:20 +0200] warning/ApplyRule: Apply rule ‘swap’ (in /var/lib/icinga2/api/packages/director/05940512-2888-4d33-aff3-b00647ad6031/zones.d/director-global/service_apply.conf: 121:1-121:20) for type ‘Service’ does not match anywhere!
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 ElasticsearchWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 122 Hosts.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 3 Zones.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 GelfWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 OpenTsdbWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 CompatLogger.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 288 CheckCommands.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 LivestatusListener.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 1 User.
[2020-10-14 14:04:20 +0200] information/ConfigItem: Instantiated 496 Services.
[2020-10-14 14:04:20 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2020-10-14 14:04:20 +0200] information/cli: Finished validating the configuration file(s).

Please let me know if you need any further information.

Thank you for your help.

Best regards,

Javier

Hello,

It’s been a week since my initial post and I wanted to ask if you have any suggestion. Please let me know if you need any further information.

Thank you in advance and best regards,

Javier

Hi,

pls be patient. The most of us are helping in the community either if there is a time window during work or in the leisure time. Also if someone has a solution or idea, because he/she struggled in a similar problem in the own company or still has the knowledge about this topic. :wink:

Now to your question:
Do you see in the host/service history in icingaweb2 if icinga is trying to send a mail or give you a hint why it’s not working? Do you see something in the icinga2.log? e.g. an error message like the mail program is missing, not configured or working well or the script is skipping with an error?
Does your mail script working on command line?

Hi Stevie,

thank you very much for your answer. It doesn’t look like Icinga is trying to send a mail automatically, although we have quite often a service/host problem and the whole configuration looks alright to me. As you can see from my enclosed screenshots, the “Notifications” section is completely empty. It doesn’t give me neither a hint and I cannot really see why it’s not working:

I’ve been also having a look at the logs very often just to look for a hint, but I didn’t really find anything related to the notifications. I only get something related to it if I send manually a custom/forced notification directly from Icingaweb2.

The “notification” icinga2 feature is also enabled (actually everything is enabled):

Please find enclosed the following outputs:

tail -f /var/log/icinga2/icinga2.log

tail -f /var/log/icinga2/debug.log | grep "notice/Process"

The mail scripts work well from the command line. I tried this a few times and I got always the e-mails:

./service-by-mail.sh -4 ‘127.0.0.1’ -r ‘my e-mail’
./host-by-mail.sh -4 ‘127.0.0.1’ -r ‘my e-mail’

image

Please let me know if you need any further information.

Thank you again for your help.

Best regards,

Javier

Hi

I meant the history tab in at the host object. if the host or service is in a non ok state you should see if the notification rule is working:
e.g.


here is no notification sent, because nobody in our company wants to have “unkown”-messages.
image

In the last time there were some threads with a similar problems. Maybe they could help:

If you can send a forced notification via icingaweb2 it couldn’t be something like SELinux or wrong file permisisons. the same if you can send messages via the bash. I think something is wrong with the apply rule. In the first link is a way how to check if the rules are applied