Director Notifications.conf

Hello everyone,

I am new in the icinga2 world. The last two days I have setup my Centos 7 with Icinga2 and installed the Director. Ater adding my first hosts, I want to create Telegram notifications.

I have read several tutorials (for example) on the web but I think I make something wrong :confused:

I want to user this scripts

1.) I have created the
/etc/icinga2/conf.d/notifications.conf and commands.conf
2.) downloaded the scrips and set chmod u+x

3.) restart icinga2 and kickstart on director

Now I want to create a notification template on director but I can’t find my import “Notify Host By Telegram”

Can you tell me what I am doing wrong :frowning:

Yours Alex

Are the notification commands visible in the Directors command section under external commands (like the normal mail-host/service-notification should be)?

If not, check if your conf.d folder in /etc/icinga2 is excluded from the config.
If you setup your master server with the node wizard it asks if you want to exclude this folder. If you replied with “yes” you will have something like

// Disabled by the node setup CLI command on 2018-11-13 16:01:13 +0100
// include_recursive "conf.d"

If so include the commands.conf with include "conf.d/commands.conf"

What have you put into the notifications.conf?

Another way, instead of defining the commands in the files, would be to define them by hand in the Director.

1 Like

Hello,

thank you, I have found the solution last night.

But now I have the Problem that when a notification get started I got Permission denied
‘PROBLEM’ ‘-v’ ‘true’) terminated with exit code 128, output: execvpe(/etc/icinga2/scripts/host-by-telegram.sh) failed: Permission denied

I have set the script chmod +u and sudo semanage permissive -a nagios_notification_plugin_t
but still get Permission denied
Yours Alex

Can you run the scripts as the icinga2 executing user?
sudo -u icinga as I assume you use some distribution like CentOS because of the SELinux.

No further help can be expected of me according to SELinux though :smiley: as I mostly work with Ubuntu (or set SELinux to permissive :crazy_face: )

1 Like

thx I have set
SELinux to permissive

Now It can send via the script :slight_smile:

I have only the problem it sends all notifications but on the bot I only get the recover (UP) message :confused:

[2019-04-12 10:50:17 +0200] information/Checkable: Checking for configured notifications for object ‘test2’
[2019-04-12 10:50:17 +0200] information/Notification: Sending ‘Problem’ notification ‘test2!Host by Telegram’ for user ‘telegram_unixe’
[2019-04-12 10:50:17 +0200] information/Notification: Completed sending ‘Problem’ notification ‘test2!Host by Telegram’ for checkable ‘test2’ and user ‘telegram_unixe’.

But on the bot I got no notification

When it goes up again:
[2019-04-12 11:23:34 +0200] information/Notification: Sending ‘Recovery’ notification ‘test2!Host by Telegram’ for user ‘telegram_unixe’
[2019-04-12 11:23:34 +0200] information/Notification: Completed sending ‘Recovery’ notification ‘test2!Host by Telegram’ for checkable ‘test2’ and user ‘telegram_unixe’.

I’ll get a notification

Hm, your user templates and objects have the DOWN/Problem states and types defined, correct?

1 Like

yes, on log you can see there ist sending a command but I think there is a bug in the script

when down then (i think this is a problem)
‘-o’ ‘<Terminated by signal 9 (Killed).>’ ‘-p’

when up
‘-o’ ‘PING OK - Packet loss = 20%, RTA = 4.10 ms’ ‘-p’

Hello everyone,

I am right the problem is working with hostalive and the “<Terminated by signal 9 (Killed).>’” in the down message. If I use ping all is working fine

Yours Alex

Looks like the ping timeout is higher than the CheckCommand timeout, resulting in Icinga killing the process before ping itself returns. Try setting the ping_timeout variable to a possible low value of e.g. 30s or similar.

Cheers,
Michael

What was your solution?
I was following the same guide and I’m kinda stuck on it. I’m trying to use the scripts that spiller provided in her tutorial and it works if I do it from cli and I can’t get it to even attempt to send a notification on its own.

Hi,

can you please share the manual tests as well as how the error looks like exactly inside the web interface or the (debug) logs? How does the Director configuration look like?

Also, please add the versions involved for Icinga 2 and the Director.

Cheers,
Michael

Hi!

By testing I meant two things, running Spiller’s host-by-mail script from the cli with manually added parameters, that manages to send out a test email. Also when on the web interface if I select a host and click “Notification” and send a custom host notification that also works. For now, all I want is to get an email if a host goes down. I have a host added for testing that I can simulate problems with. If I could get that working I could probably figure out the rest…

Ok so from the top:

I have a notification template called “host_notify_template”, looks like this:

zones.d/myserver.local/notification_templates.conf

template Notification “host_notify_template” {
command = “host-by-mail”
period = “always”
states = [ Down, Up ]
types = [ Acknowledgement, Custom, Problem, Recovery ]
users = [ “myuser” ]
}

Then I have a notification with that template that looks like this (I’m not sure about the assign rule here tbh):

zones.d/myserver.local/notification_apply.conf

apply Notification “host-notification” to Host {
import “host_notify_template”

assign where host.enable_notifications == "true"
users = [ "myuser" ]

}

These call for a command called “host-by-mail”. That looks like this:

zones.d/icinga.srvs.blkfpark.local/commands.conf

object NotificationCommand “host-by-mail” {
import “plugin-notification-command”
command = [ “/etc/icinga2/scripts/host-by-mail.sh” ]
arguments += {
“-4” = “$address$”
“-6” = “$address6$”
“-d” = “$icinga.long_date_time$”
“-l” = “$host.name$”
“-n” = “$host.display_name$”
“-o” = “$host.output$”
“-r” = “$user.email$”
“-s” = “$host.state$”
“-t” = “$notification.type$”
}
}

Versions:

icinga2 - The Icinga 2 network monitoring daemon (version: r2.10.4-1)
icininga web2 version: 2.6.2
icinga director version: 1.6.2

What else do you need in regard of config files?
Also regarding logs, when I make the test host unavailable the notification doesn’t fire at all as far as I can tell.

Thanks!

With running icinga2 object list --type Notification --name *host-notification* you are not getting a notification object, right?

Cheers,
Michael

Indeed, I’m not getting anything. So basically the statement is not true for any of my hosts and that’s why it’s not firing?

Yep. I wanted you to ask yourself this question before directly pointing to it :wink:

A notification object is required being linked to the host/service object. When a hard state change happens, Icinga loops over all notification objects related here and triggers the notification command to execute - for each user of course.

That being said, enable_notifications is of the boolean value, which is true or false without the quotes. By using the quotation, Icinga checks for equal strings which never is the case because of the type mismatch.

You can test such with the debug console, if you’re not sure.

$ icinga2 console
Icinga 2 (version: v2.10.4-644-gb37f87929)
Type $help to view available commands.
<1> => host.enable_notifications = true
null
<2> => host.enable_notifications == "true"
false
<3> => host.enable_notifications == true
true

Cheers,
Michael

Thanks for taking the time and helping me!
I changed my configuration to this now:

zones.d/myserver.local/notification_apply.conf

    apply Notification "host-notification" to Host {
    import "host_notify_template"

    assign where host.enable_notifications
    users = [ "myuser" ]

}

It still doesn’t fire however and the log is giving me nothing. Is there a way to enforce more detailed logging? This is all I get:

[2019-05-08 14:51:43 +0200] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/f10d9609-5536-4b29-87f9-e4ff1cfbfd5c/zones.d/myserver/notification_apply.conf
[2019-05-08 14:51:43 +0200] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/f10d9609-5536-4b29-87f9-e4ff1cfbfd5c/zones.d/myserver/notification_templates.conf
[2019-05-08 14:52:48 +0200] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ef30ef3c-6fcb-46cc-939d-fb7aa0f3ef7f/zones.d/myserver/notification_apply.conf
[2019-05-08 14:52:48 +0200] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/ef30ef3c-6fcb-46cc-939d-fb7aa0f3ef7f/zones.d/myserver/notification_templates.conf
[2019-05-08 14:53:21 +0200] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/1b9c583d-4921-474b-8e04-c1be549ad2f2/zones.d/myserver/notification_apply.conf
[2019-05-08 14:53:21 +0200] information/ConfigPackageUtility: Updating configuration file: /var/lib/icinga2/api/packages/director/1b9c583d-4921-474b-8e04-c1be549ad2f2/zones.d/myserver/notification_templates.conf
[2019-05-08 14:53:23 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/myserver//director/notification_apply.conf
[2019-05-08 14:53:23 +0200] information/NotificationComponent: ‘notification’ started.

It basically logged the changes in configuration and that’s all. I made my test host unavailable and didn’t do anything.

Hm, what exactly happens if you run icinga2 daemon -C, are there any warnings for unmatched notifications?

If there are not, and icinga2 object list --type Notification returns the created object, then I’d advise to enable the debug log and check on why the notification event doesn’t happen. Some instructions on that can be found in the troubleshooting docs.

Cheers,
Michael

icinga2 daemon -C returns all green, no errors there.

icinga2 object list --type Notification returns basically all my hosts. I’ll look into the debug log thing to see if I can learn something. Thanks!

[2019-05-08 15:16:39 +0200] information/Checkable: Checking for configured notifications for object 'testserver.local!ping4'
[2019-05-08 15:16:39 +0200] information/Checkable: Checkable 'testserver.local!ping4' does not have any notifications.

This is what happens when the host goes down. On the web interface the host is show as being in CRITICAL state and “Host unreachable”.

Iterating back to the notification configuration, this is one for host objects.

The log shows a notification attempt for a service, and it seems you don’t have any.

So you’ll need to create a second apply rule for notifications which applies to the Service type.

Cheers,
Michael