Need to setup Icinga2/Icingaweb2 to send email alerts only for ping service

Icinga Web 2
1

Hello there,

I have installed Icinga2 and setup Icingaweb2 on RHEL7, currently I have a test host and have configured it in the hosts.conf as an object host. I configured the notification.conf and users.conf for the ping4 service alerts. but when I stop the n/w service on the test host, Icingaweb2 shows that host is down, but it is not sending any notifications. I have also installed mailx, configured, tested sending emails and I am able receive the test emails using mailx. Could someone please help me fix the email notifications?

Config details are as follows…

users.conf

object User “Admin” {
import “generic-user”

display_name = “Admin”
email = “removed email ID”
enable_notifications = true
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
}

notifications.conf

apply Notification “mail-Admin” to Host {
import “mail-host-notification”
users = [ “Admin” ]
interval = 30s
period = “24x7”
assign where host.vars.notification.mail
}

apply Notification “mail-Admin” to Service {
import “mail-service-notification”
command = “mail-service-notification”
users = [ “Admin” ]
interval = 30s
period = “24x7”
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
assign where service.name == “ping4”
}

icinga2 daemon -C Output

[root@rhel7-pingmon-tst conf.d]# icinga2 daemon -C
[2020-09-24 16:38:38 -0500] information/cli: Icinga application loader (version: 2.12.0-1)
[2020-09-24 16:38:38 -0500] information/cli: Loading configuration file(s).
[2020-09-24 16:38:38 -0500] information/ConfigItem: Committing config item(s).
[2020-09-24 16:38:38 -0500] information/ApiListener: My API identity: “Removed hostname”
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 2 Hosts.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 FileLogger.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 3 Notifications.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 2 HostGroups.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 3 Zones.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 Endpoint.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 ApiUser.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 ApiListener.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 235 CheckCommands.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 TimePeriod.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 1 User.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 3 Services.
[2020-09-24 16:38:38 -0500] information/ConfigItem: Instantiated 3 ServiceGroups.
[2020-09-24 16:38:38 -0500] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2020-09-24 16:38:38 -0500] information/cli: Finished validating the configuration file(s).

Thanks and regards,
Balaji.

2

Hi and welcome,

did you check with

icinga2 feature list

if notificiation is activated?

image
image919×78 6.84 KB

3

is it ping4 or Ping4? Not sure if thats important.

you can check if the notification is applied with:

icinga2 object list --type=Notification
4

Hello Rafael Voss,

Following is the output of icinga2 object list --type=Notification.

[root@removed-hostname]# icinga2 object list --type=Notification
Object ‘removed-hostname!ping4!mail-Admin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43

  • __name = “removed-hostname!ping4!mail-Admin”
  • command = “mail-service-notification”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 69:3-69:39
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 22:3-22:39
  • command_endpoint = “”
  • host_name = “removed-hostname”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
  • interval = 30
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 24:3-24:16
  • name = “mail-Admin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
  • period = “24x7”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 82:3-82:17
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 25:3-25:17
  • service_name = “ping4”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
  • source_location
    • first_column = 2
    • first_line = 20
    • last_column = 43
    • last_line = 20
    • path = “/etc/icinga2/conf.d/notifications.conf”
  • states = [ “OK”, “Warning”, “Critical”, “Unknown” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 71:3-71:45
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 26:3-26:45
  • templates = [ “mail-Admin”, “mail-service-notification” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 68:1-68:49
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd”, “DowntimeStart”, “DowntimeEnd”, “DowntimeRemoved” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 72:3-74:57
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 27:3-29:57
  • user_groups = null
  • users = [ “Admin” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 23:3-23:21
  • vars
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 76:3-80:3
    • notification_logtosyslog = false
  • zone = “”

Object ‘removed-hostname!icinga!mail-Admin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43

  • __name = “rhel7-pingmon-tst.chicagopolice.local!icinga!mail-Admin”
  • command = “mail-service-notification”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 69:3-69:39
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 22:3-22:39
  • command_endpoint = “”
  • host_name = “removed-hostname”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
  • interval = 30
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 24:3-24:16
  • name = “mail-Admin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
  • period = “24x7”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 82:3-82:17
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 25:3-25:17
  • service_name = “icinga”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
  • source_location
    • first_column = 2
    • first_line = 20
    • last_column = 43
    • last_line = 20
    • path = “/etc/icinga2/conf.d/notifications.conf”
  • states = [ “OK”, “Warning”, “Critical”, “Unknown” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 71:3-71:45
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 26:3-26:45
  • templates = [ “mail-Admin”, “mail-service-notification” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 20:2-20:43
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 68:1-68:49
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd”, “DowntimeStart”, “DowntimeEnd”, “DowntimeRemoved” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 72:3-74:57
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 27:3-29:57
  • user_groups = null
  • users = [ “Admin” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 23:3-23:21
  • vars
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 76:3-80:3
    • notification_logtosyslog = false
  • zone = “”

Object ‘removed-hostname!mail-Admin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:39

  • __name = “removed-hostname!mail-Admin”
  • command = “mail-host-notification”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 47:3-47:36
  • command_endpoint = “”
  • host_name = “removed-hostname”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:39
  • interval = 30
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 14:3-14:16
  • name = “mail-Admin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:39
  • period = “24x7”
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 60:3-60:17
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 15:3-15:17
  • service_name = “”
  • source_location
    • first_column = 1
    • first_line = 11
    • last_column = 39
    • last_line = 11
    • path = “/etc/icinga2/conf.d/notifications.conf”
  • states = [ “Up”, “Down” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 49:3-49:23
  • templates = [ “mail-Admin”, “mail-host-notification” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 11:1-11:39
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 46:1-46:46
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd”, “DowntimeStart”, “DowntimeEnd”, “DowntimeRemoved” ]
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 50:3-52:57
  • user_groups = null
  • users = [ “Admin” ]
    % = modified in ‘/etc/icinga2/conf.d/notifications.conf’, lines 13:3-13:21
  • vars
    % = modified in ‘/etc/icinga2/conf.d/templates.conf’, lines 54:3-58:3
    • notification_logtosyslog = false
  • zone = “”

Thanks and regards,
Balaji

5

Hello Stevie Sy,

The following is the output of icinga2 feature list…
The notifications feature is already enabled.

image
image1316×67 8.39 KB

Thanks and regards,
Balaji.

6

Ok, the notification feature is enabled and the notification rules are assigned.
You wrote that you are using RHEL and you configured mailx.
Is the icinga user allowed to start mailx? Do you enabled SELinux? Maybe icinga is not allowed to start mailx.

7

Hi Stevie,

No, SELinux is disabled. I switched to Icinga shell, was successfully able to send mail using mailx and received the same.

image

Thanks & regards,
Balaji.

8

Hi Stevie,

When I send the custom notifications, I am able to receive them, but not the standard notifications for issues.

Thanks & regards,
Balaji.

9

ok, just to be sure: do you have an example for us what is a custom notification and what is a standard notificiation for you?

10

Hi Stevie,

Following it the screen shot of a custom notification from icingaweb2…

image
image933×464 4.65 KB

Following is the custom notification email I received from icinga…

image
image668×593 18.6 KB

I meant a notification sent out automatically for a state(Down, Critical, etc…) of the host is a standard notification.

I also see the notifications are not being sent out to any contacts…

image
image879×597 8.32 KB

Thanks & regards,
Balaji.

11

The “was not sent … to any contact” error texts are often output if the contacts you configured in the notification rule have not got the receiving of notifications enabled! Check your contacts and perhaps set “Disable notifications” to no…

Also check that the notified states in the rule are appropriate (e.g. “Up” & “Down” for a host notification). The SAME states may have to be explicitly allowed in the contacts, too.

12

Hi Kai,

I see the following in the contact, it looks like it has sent the notifications, but I did not get any notification emails, not sure what is wrong.

image
image1131×707 10.8 KB

So far it is sending only custom notifications successfully, when I send one by clicking “Send Notification”
It has sent 36 notifications by itself to the contact Admin, but I am not receiving them.

image
image610×933 8.96 KB

Best regards,
Balaji.

13

Let’s have a look at the mail notification script this notification rule (that’s not behaving) actually uses! Click through the previews shown in the GUI from notification rule via all imports (always via the previews: click, click, …) until you arrive at the actual command definition. Then have a look at the actual notify command you use, at the actual script and identify the line in it that’s sending the mail…

In our case that’s always std. command mail! If so, set up a test “mail” call-up from your Bash (e.g. to yourself); if that mail sent from the shell works, so should the one in your notify script.

14

Hi Kai,

I am unable to find previews to check command definitions.
Could you please provide me an example to test mail and call up notification bash script?

Thanks & regards,
Balaji.

15

You need to identify the actual notify script…

Go to your (misbehaving) notification, i.e. open it in Director. At the top the 2nd tab will be the preview label (“Vorschau” in this example):
grafik
(This is the German interface we in my firm use; you will probably have the English labels.)

Click on that, and then click again on the “imports” linked in that “raw” syntaxed variant of your notification rule.

Once you have clicked on it, again click the “preview” tab. And so on…

In the final window you will see the command path, describing the script file. Switch to the shell of your Icinga server, locate that script, open it & look for the part that actually mails something.

If it is a “mail” command, try this at the shell prompt: mail -s "Test mail #1" your.own@address
Type in some text and then end it with the usual <Ctrl-D> at a line begin to end text input in Linux.

Then check that mail got passed on, using e.g. “mailq”. (That’s Postfix. If all went correctly, that output should be empty!) Or whatever steps you take to check correct transport of your test mail.

If that all worked & you received the mail, your basic infrastructure from notification rule to sending the mails is okay. Then it can only be due to the notify script - either being buggy or not getting the arguments it needs… (As a rule, notify scripts are not too complex. :wink:)

If it seemed to work, but you got no mail, check your sendmailing configuration or that of the mail relay host (Exchange?) you are likely to be using. (Or do a re-check of your Director config’s - cf. following P.S…)

P.S.: Did you check that the (Director-shown) states mentioned in your receiving contact and in your notif. rule agree?

16

Hi Kai,

I know the mailx command to test email and it is working fine for the test emails using mailx.
I thought you were asking me to run the notification bash script using mailx, which I am not sure how to.

I enabled the debug and found out from the debug logs that the state “Down” was not defined in the filters for notifications, once I added the state “Down” I started receiving notifications for Down state, I also added the state Down for the filters in the notifications.conf and updated the notifications.conf to the following from my original notifications.conf…

apply Notification “mail-Admin” to Host {
import “mail-host-notification”
command = “mail-host-notification”
users = [ “Admin” ]
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
assign where host.vars.notification.mail
vars += {
notification_icingaweb2url = “https://remved-host-name/icingaweb2”
notification_from = “Icinga 2 Host Monitoring icinga@remved-host-name”
notification_logtosyslog = false
}

period = “24x7”
}

Thank you for your help.

Balaji.

17

Hi Stevie,

I enabled the debug and found out from the debug logs that the state “Down” was not defined in the filters for notifications, once I added the state “Down” I started receiving notifications for Down state, I also added the state Down for the filters in the notifications.conf and updated the notifications.conf to the following from my original notifications.conf…

apply Notification “mail-Admin” to Host {
import “mail-host-notification”
command = “mail-host-notification”
users = [ “Admin” ]
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeStart, DowntimeEnd, DowntimeRemoved ]
assign where host.vars.notification.mail
vars += {
notification_icingaweb2url = “https://remved-host-name/icingaweb2”
notification_from = “Icinga 2 Host Monitoring icinga@remved-host-name”
notification_logtosyslog = false
}

period = “24x7”
}

Thank you for your help.

Balaji.

1 Like
18

I’m glad to read, that it works now

19

(Yes, as I noted - twice - you need to pay attention to the the states allowed in contact & notification rule.)