Icinga2 not sending notification for services only for host down

Service Monitoring
1

Hi All,
I encountered on this problem amd working with the icinga2 logs but didn’t find the right configuration that should solve it.

The icinga2 not sending emails on change state of services in monitored clients only on the hosts.
Which means if i stopped service on Linux client , we getting the alert on the dashboard but not receiving email.

I tested it on several monitored clients and it’s same behavior,

For example by stop the ssh on monitored client we can see the alert on dashboard, but not getting notification and if i will shutdown the host i’m getting alert that host is down.

In addition I added a custom check that monitor if Postifx service of icinga2 client is down,
When I stopping the postifx we getting the service alert on dashboard but not alerting by mail.

Custom check below:

/etc/icinga2/zones.d/global-templates/services.conf

apply Service “Check Postfix status” {
import “generic-service”
check_command = “Check Postfix status”
assign where host.vars.servicename == “SMTP_Servers”
}

##/etc/icinga2/zones.d/global-templates/commands.conf

object CheckCommand “Check Postfix status” {
import “by_ssh”
vars.by_ssh_command = [ “sudo”, “/usr/lib64/nagios/plugins/check_postfix_status.sh” ]
}

cat /etc/icinga2/zones.d/Site/FQDN.conf
object Host “FQDN” {
import “generic-host”
address = “FQDN”
check_command = “hostalive”
vars.agent = “ssh”
vars.os = “Linux”
vars.servicename = “SMTP_Servers”
vars.notification[“mail”] = {
groups = [ “icingaadmins” ]
}
}

Since it does not notify comprehensively on all services, and not just for custom
i’m sure it’s a general configuration but i didn’t get to it by investigate from icinga2 log.

2

What does the cli say about the notification objects for the services?

3

Hi Dominik ,
Which command execution output you ask?

4 
sudo icinga2 object list --type Notification  --name  "your_host_name*"

This should return all notifications on the host and his services.

5

Thank you , attached .

[root@icinga2 ~]# sudo icinga2 object list --type Notification --name “icinga2-fqdn*”
Object ‘icinga2-fqdn!disk /!mail-icingaadmin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

  • __name = “icinga2-fqdn!disk /!mail-icingaadmin”
  • command = “mail-service-notification”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 64:3-64:39
  • command_endpoint = “”
  • host_name = “icinga2-fqdn”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • interval = 0
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 27:1-27:12
  • name = “mail-icingaadmin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • period = “24x7”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 72:3-72:17
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 31:3-31:17
  • service_name = “disk /”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • source_location
    • first_column = 1
    • first_line = 22
    • last_column = 48
    • last_line = 22
    • path = “/etc/icinga2/zones.d/global-templates/notifications.conf”
  • states = [ “OK”, “Warning”, “Critical” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 66:3-66:37
  • templates = [ “mail-icingaadmin”, “mail-service-notification” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 63:1-63:49
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 67:3-68:40
  • user_groups = [ “icingaadmins” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 25:3-25:50
  • users = null
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 26:3-26:43
  • vars = null
  • zone = “SITE”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

Object ‘icinga2-fqdn!ping4!mail-icingaadmin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

  • __name = “icinga2-fqdn!ping4!mail-icingaadmin”
  • command = “mail-service-notification”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 64:3-64:39
  • command_endpoint = “”
  • host_name = “icinga2-fqdn”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • interval = 0
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 27:1-27:12
  • name = “mail-icingaadmin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • period = “24x7”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 72:3-72:17
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 31:3-31:17
  • service_name = “ping4”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • source_location
    • first_column = 1
    • first_line = 22
    • last_column = 48
    • last_line = 22
    • path = “/etc/icinga2/zones.d/global-templates/notifications.conf”
  • states = [ “OK”, “Warning”, “Critical” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 66:3-66:37
  • templates = [ “mail-icingaadmin”, “mail-service-notification” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 63:1-63:49
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 67:3-68:40
  • user_groups = [ “icingaadmins” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 25:3-25:50
  • users = null
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 26:3-26:43
  • vars = null
  • zone = “SITE”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

Object ‘icinga2-fqdn!mail-icingaadmin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 11:1-11:45

  • __name = “icinga2-fqdn!mail-icingaadmin”
  • command = “mail-host-notification”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 53:3-53:36
  • command_endpoint = “”
  • host_name = “icinga2-fqdn”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 11:1-11:45
  • interval = 0
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 16:2-16:13
  • name = “mail-icingaadmin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 11:1-11:45
  • period = “24x7”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 58:3-58:17
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 19:3-19:17
  • service_name = “”
  • source_location
    • first_column = 1
    • first_line = 11
    • last_column = 45
    • last_line = 11
    • path = “/etc/icinga2/zones.d/global-templates/notifications.conf”
  • states = [ “Up”, “Down” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 55:3-55:24
  • templates = [ “mail-icingaadmin”, “mail-host-notification” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 11:1-11:45
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 52:1-52:46
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 56:3-56:57
  • user_groups = [ “icingaadmins” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 14:3-14:50
  • users = null
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 15:3-15:43
  • vars = null
  • zone = “SITE”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 11:1-11:45

Object ‘icinga2-fqdn!cluster!mail-icingaadmin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

  • __name = “icinga2-fqdn!cluster!mail-icingaadmin”
  • command = “mail-service-notification”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 64:3-64:39
  • command_endpoint = “”
  • host_name = “icinga2-fqdn”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • interval = 0
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 27:1-27:12
  • name = “mail-icingaadmin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • period = “24x7”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 72:3-72:17
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 31:3-31:17
  • service_name = “cluster”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • source_location
    • first_column = 1
    • first_line = 22
    • last_column = 48
    • last_line = 22
    • path = “/etc/icinga2/zones.d/global-templates/notifications.conf”
  • states = [ “OK”, “Warning”, “Critical” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 66:3-66:37
  • templates = [ “mail-icingaadmin”, “mail-service-notification” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 63:1-63:49
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 67:3-68:40
  • user_groups = [ “icingaadmins” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 25:3-25:50
  • users = null
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 26:3-26:43
  • vars = null
  • zone = “SITE”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

Object ‘icinga2-fqdn!icinga!mail-icingaadmin’ of type ‘Notification’:
% declared in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48

  • __name = “icinga2-fqdn!icinga!mail-icingaadmin”
  • command = “mail-service-notification”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 64:3-64:39
  • command_endpoint = “”
  • host_name = “icinga2-fqdn”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • interval = 0
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 27:1-27:12
  • name = “mail-icingaadmin”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • period = “24x7”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 72:3-72:17
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 31:3-31:17
  • service_name = “icinga”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
  • source_location
    • first_column = 1
    • first_line = 22
    • last_column = 48
    • last_line = 22
    • path = “/etc/icinga2/zones.d/global-templates/notifications.conf”
  • states = [ “OK”, “Warning”, “Critical” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 66:3-66:37
  • templates = [ “mail-icingaadmin”, “mail-service-notification” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 63:1-63:49
  • times = null
  • type = “Notification”
  • types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 67:3-68:40
  • user_groups = [ “icingaadmins” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 25:3-25:50
  • users = null
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 26:3-26:43
  • vars = null
  • zone = “SITE”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/notifications.conf’, lines 22:1-22:48
    [root@icinga2 ~]#
6

The Check Postfix status is not appeard , how i can added/ enable the notification ?

7

You need to add

vars.notification[“mail”] = {
groups = [ “icingaadmins” ]

to the postfix service, as this will probably trigger a apply rule that will create the notification object.
Or do you attach the ones from the host to all services if nothing is explicitly set on the service?

8

Hi Dominik,
The following ;
vars.notification[“mail”] = {
groups = [ “icingaadmins” ]

Were already attached to the host confiugration…
Iexpected it to issue alerts both for host down and it’s services.

cat /etc/icinga2/zones.d/Site/FQDN.conf
object Host “FQDN” {
import “generic-host”
address = “FQDN”
check_command = “hostalive”
vars.agent = “ssh”
vars.os = “Linux”
vars.servicename = “SMTP_Servers”
vars.notification[“mail”] = {
groups = [ “icingaadmins” ]
}
}

Do I need to change something else?

9

Where do the notifications for the other services of the host come from?
Can you look in /etc/icinga2/zones.d/global-templates/notifications.conf for the apply rules of the mail-service-notification?

10

Hi Dominik , attached

[root@icinga2 ~]# cat /etc/icinga2/zones.d/global-templates/notifications.conf
/**

  • The example notification apply rules.
  • Only applied if host/service objects have
  • the custom attribute notification defined
  • and containing mail as key.
  • Check hosts.conf for an example.
    */

apply Notification “mail-icingaadmin” to Host {
import “mail-host-notification”

user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users
interval = 0
#assign where host.vars.agent == “ssh”
assign where host.vars.notification.mail
period = “24x7”
}

apply Notification “mail-icingaadmin” to Service {
import “mail-service-notification”

user_groups = host.vars.notification.mail.groups
users = host.vars.notification.mail.users
interval = 0
assign where host.vars.agent == “ssh”
assign where host.address
assign where host.vars.notification.mail
period = “24x7”
}

11

Hi Dominik,
just wanted to update that I took a test server i simulated events like /root getting full, ssh stopped, and notification didn’t received.
I saw the incidents only on dashboard

12

why isn’t this enough and

assign where host.vars.agent == “ssh”
assign where host.address

are added?