Slack And Pagerduty Notification on Icinga2

Manali · September 27, 2023, 9:17am

Hello Team,

I have configured icinga2 on linux server. Everything is working as expected but for Host and Service I am receiving notification only for ping4 service. For other services I am not getting notification on slack and PagerDuty. Here is the output of icinga2 object list --type Notification
I want notification for all services.

Object ‘saaskarma!ping4!pagerduty-service’ of type ‘Notification’:
% declared in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 75:1-75:49

__name = “saaskarma!ping4!pagerduty-service”
command = “notify-service-by-pagerduty”
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 76:3-76:41
command_endpoint = “”
host_name = “saaskarma”
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 75:1-75:49
interval = 1800
name = “pagerduty-service”
package = “_etc”
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 75:1-75:49
period = “24x7”
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 79:3-79:17
service_name = “ping4”
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 75:1-75:49
source_location
- first_column = 1
- first_line = 75
- last_column = 49
- last_line = 75
- path = “/etc/icinga2/conf.d/pagerduty-icinga2.conf”
states = [ “OK”, “Warning”, “Critical”, “Unknown” ]
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 77:3-77:45
templates = [ “pagerduty-service” ]
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 75:1-75:49
times = null
type = “Notification”
types = [ “Problem”, “Acknowledgement”, “Recovery” ]
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 78:3-78:48
user_groups = null
users = [ “pagerduty” ]
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 80:3-80:25
vars = null
zone = “master”
% = modified in ‘/etc/icinga2/conf.d/pagerduty-icinga2.conf’, lines 75:1-75:49

Object ‘saaskarma!ping4!slack-notifications-notification-services’ of type ‘Notification’:
% declared in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 66:1-66:73

__name = “saaskarma!ping4!slack-notifications-notification-services”
command = “slack-notifications-command”
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 45:5-45:43
command_endpoint = “”
host_name = “saaskarma”
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 66:1-66:73
interval = 900
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 12:5-12:18
name = “slack-notifications-notification-services”
package = “_etc”
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 66:1-66:73
period = “24x7”
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 10:5-10:19
service_name = “ping4”
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 66:1-66:73
source_location
- first_column = 1
- first_line = 66
- last_column = 73
- last_line = 66
- path = “/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf”
states = [ “OK”, “Warning”, “Critical”, “Unknown” ]
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 51:5-51:47
templates = [ “slack-notifications-notification-services”, “slack-notifications-user-configuration-services”, “slack-notifications-default-configuration-services”, “slack-notifications-configuration”, “slack-notifications-user-configuration”, “slack-notifications-default-configuration” ]
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 66:1-66:73
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf’, lines 12:1-12:71
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 48:1-48:74
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 42:1-42:57
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf’, lines 1:0-1:61
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 1:0-1:64
times = null
type = “Notification”
types = [ “Problem”, “Acknowledgement”, “Recovery”, “Custom”, “FlappingStart”, “FlappingEnd” ]
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 6:5-7:42
user_groups = null
users = [ “slack-notification-user” ]
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 8:5-8:41
vars
- slack_notifications_botname = “icinga”
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 15:5-15:47
- slack_notifications_channel = “#icinga-rawneg”
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 14:5-14:55
- slack_notifications_color_dictionary
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 17:5-24:5
  - CRITICAL = “danger”
  - DOWN = “danger”
  - OK = “good”
  - UNKNOWN = “grey”
  - UP = “good”
  - WARNING = “warning”
- slack_notifications_icinga2_base_url = “https://icinga.raweng.net”
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf’, lines 5:5-5:75
- slack_notifications_icon_dictionary
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 25:5-39:5
  - ACKNOWLEDGEMENT = “ballot_box_with_check”
  - CRITICAL = “red_circle”
  - CUSTOM = “speaking_head_in_silhouette”
  - DOWNTIMEEND = “arrow_down_small”
  - DOWNTIMEREMOVED = “leftwards_arrow_with_hook”
  - DOWNTIMESTART = “arrow_up_small”
  - FLAPPINGEND = “small_red_triangle_down”
  - FLAPPINGSTART = “small_red_triangle”
  - OK = “large_blue_circle”
  - PROBLEM = “red_circle”
  - RECOVERY = “large_blue_circle”
  - UNKNOWN = “grey_question”
  - WARNING = “bell”
- slack_notifications_plugin_output_max_length = 3500
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 16:5-16:60
- slack_notifications_webhook_url = “https://hooks.slack.com/services/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”
  % = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-user-configuration.conf’, lines 4:5-4:124
zone = “master”
% = modified in ‘/etc/icinga2/conf.d/slack-notifications/slack-notifications-configuration.conf’, lines 66:1-66:73

Al2Klimov · September 28, 2023, 8:03am

Hello Dave!

Please share /etc/icinga2/conf.d/pagerduty-icinga2.conf .

Best,
A/K

Manali · September 29, 2023, 6:05am

object User “pagerduty” {
pager = “token”
groups = [ “sports-pd-users” ]
display_name = “Sports PagerDuty Notification User”
states = [ OK, Warning, Critical, Unknown, Up, Down ]
types = [ Problem, Acknowledgement, Recovery ]
}

object NotificationCommand “notify-service-by-pagerduty” {
import “plugin-notification-command”
command = [ “/usr/share/pdagent-integrations/bin/pd-nagios” ]
arguments = {
“-n” = {
order = 0
value = “service”
}
“-k” = {
order = 1
value = “$user.pager$”
}
“-t” = {
order = 2
value = “$notification.type$”
}
“-f” = {
order = 3
repeat_key = true
value = “$f_args$”
}
}

vars.f_args = [
“SERVICEDESC=$service.name$”,
“SERVICEDISPLAYNAME=$service.display_name$”,
“HOSTNAME=$host.name$”,
“HOSTSTATE=$host.state$”,
“HOSTDISPLAYNAME=$host.display_name$”,
“SERVICESTATE=$service.state$”,
“SERVICEPROBLEMID=$service.state_id$”,
“SERVICEOUTPUT=$service.output$”
]
}

object NotificationCommand “notify-host-by-pagerduty” {
import “plugin-notification-command”
command = [ “/usr/share/pdagent-integrations/bin/pd-nagios” ]
arguments = {
“-n” = {
order = 0
value = “host”
}
“-k” = {
order = 1
value = “$user.pager$”
}
“-t” = {
order = 2
value = “$notification.type$”
}
“-f” = {
order = 3
repeat_key = true
value = “$f_args$”
}
}

vars.f_args = [
“HOSTNAME=$host.name$”,
“HOSTSTATE=$host.state$”,
“HOSTPROBLEMID=$host.state_id$”,
“HOSTOUTPUT=$host.output$”
]
}

apply Notification “pagerduty-service” to Service {
command = “notify-service-by-pagerduty”
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery ]
period = “24x7”
users = [ “pagerduty” ]
assign where service.vars.enable_pagerduty == true
}

apply Notification “pagerduty-host” to Host {
command = “notify-host-by-pagerduty”
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery ]
period = “24x7”
users = [ “pagerduty” ]
assign where host.vars.enable_pagerduty == true
}

Al2Klimov · September 29, 2023, 9:04am

Are you sure you didn’t forget some vars.enable_pagerduty=true?

Manali · September 29, 2023, 11:24am

Where is suppose to mention this? Actually I am new to Icinga2 setup and only for ping4 service I am getting slack and PagerDuty notification but for the other services and host I am not receiving any notification.
Also I have observed one thing on icinga2 web . Please find this screenshot for your reference.
For the host and check-ssh service notification tab is empty and for the ping4 notification tab contains slack-notification-user
I dot have any idea how to assign notification user.
![Screenshot 2023-09-29 at 4.51.45 PM|690x431](upload://gYT

Al2Klimov · September 29, 2023, 12:40pm

Please share the config of

a service you receive such notifications for
a service you don’t (but would like to) receive such notifications for

Manali · September 29, 2023, 12:50pm

Command configuration
For ping4 service: Which is working fine
object CheckCommand “ping4” {
import “plugin-check-command”
command = [ PluginDir + “/check_ping”, “-4” ]
timeout = 1m
arguments += {
“-H” = {
description = “host to ping”
value = “$ping_address$”
}
“-c” = {
description = “critical threshold pair”
value = “$ping_crta$,$ping_cpl$%”
}
“-p” = {
description = “number of ICMP ECHO packets to send (Default: 5)”
value = “$ping_packets$”
}
“-t” = {
description = “Seconds before connection times out (default: 10)”
value = “$ping_timeout$”
}
“-w” = {
description = “warning threshold pair”
value = “$ping_wrta$,$ping_wpl$%”
}
}
vars.ping_address = “$address$”
vars.ping_cpl = 15
vars.ping_crta = 200
vars.ping_wpl = 5
vars.ping_wrta = 100
}

For check_memory command notification is not working
object CheckCommand “check_memory” {
import “plugin-check-command”
command = [ “/opt/icinga2-plugins/check_memory.pl”, “-u”, “-C” ]
timeout = 30s
arguments += {
“-c” = {
description = “CRITICAL alert threshold limit”
repeat_key = false
required = true
value = “$mem_crit$”
}
“-w” = {
description = “Warning threshold value”
repeat_key = false
required = true
value = “$mem_warn$”
}
}

vars.mem_crit = "90"
vars.mem_warn = "85"

}

Manali · September 29, 2023, 12:53pm

Ping4 service conf working fine for notification

apply Service “ping4” {
import “generic-service”

check_command = “ping4”

assign where host.address
}

Check_memory for which notification not working

apply Service “check_memory” {
import “check_memory”

assign where "nonprod-generic-active-sports" in host.templates
zone = "master"

import DirectorOverrideTemplate

}

Manali · September 29, 2023, 1:05pm

I have created all the commands, services and host templates from Icinga Web.

Al2Klimov · October 4, 2023, 9:29am

Strange, you reported it’s working somewhere, but I don’t see vars.enable_pagerduty = true anywhere. Please could you search for a such setting in the output of icinga2 object list -t service and share respective services?

Manali · October 4, 2023, 1:36pm

Output of icinga2 object list -t service

root@icinga:/home/ubuntu# icinga2 object list -t service
Object ‘saaskarma!check-ssh’ of type ‘Service’:
% declared in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 1:0-1:24

__name = “saaskarma!check-ssh”
action_url = “”
check_command = “ssh”
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_templates.conf’, lines 2:5-2:25
check_interval = 300
check_period = “”
check_timeout = null
command_endpoint = “saaskarma”
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_templates.conf’, lines 3:5-3:32
display_name = “check-ssh”
enable_active_checks = true
enable_event_handler = true
enable_flapping = false
enable_notifications = true
enable_passive_checks = true
enable_perfdata = true
event_command = “”
flapping_ignore_states = null
flapping_threshold = 0
flapping_threshold_high = 30
flapping_threshold_low = 25
groups =
host_name = “saaskarma”
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 1:0-1:24
icon_image = “”
icon_image_alt = “”
max_check_attempts = 3
name = “check-ssh”
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 1:0-1:24
notes = “”
notes_url = “”
package = “director”
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 1:0-1:24
retry_interval = 60
source_location
- first_column = 0
- first_line = 1
- last_column = 24
- last_line = 1
- path = “/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf”
templates = [ “check-ssh”, “check_ssh”, “host var overrides (Director)” ]
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 1:0-1:24
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_templates.conf’, lines 1:0-1:27
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/director-global/001-director-basics.conf’, lines 28:3-28:43
type = “Service”
vars = null
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/director-global/001-director-basics.conf’, lines 43:7-43:50
volatile = false
zone = “master”
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 1:0-1:24
% = modified in ‘/var/lib/icinga2/api/packages/director/cf8654a7-a0ee-4e70-853d-23bc6366dc6f/zones.d/master/service_apply.conf’, lines 6:5-6:19

Object ‘saaskarma!ping4’ of type ‘Service’:
% declared in ‘/etc/icinga2/conf.d/services.conf’, lines 26:1-26:21

__name = “saaskarma!ping4”
action_url = “”
check_command = “ping4”
% = modified in ‘/etc/icinga2/conf.d/services.conf’, lines 29:3-29:25
check_interval = 60
% = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 35:3-35:21
check_period = “”
check_timeout = null
command_endpoint = “”
display_name = “ping4”
enable_active_checks = true
enable_event_handler = true
enable_flapping = false
enable_notifications = true
enable_passive_checks = true
enable_perfdata = true
event_command = “”
flapping_ignore_states = null
flapping_threshold = 0
flapping_threshold_high = 30
flapping_threshold_low = 25
groups =
host_name = “saaskarma”
% = modified in ‘/etc/icinga2/conf.d/services.conf’, lines 26:1-26:21
icon_image = “”
icon_image_alt = “”
max_check_attempts = 5
% = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 34:3-34:24
name = “ping4”
% = modified in ‘/etc/icinga2/conf.d/services.conf’, lines 26:1-26:21
notes = “”
notes_url = “”
package = “_etc”
% = modified in ‘/etc/icinga2/conf.d/services.conf’, lines 26:1-26:21
retry_interval = 30
% = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 36:3-36:22
source_location
- first_column = 1
- first_line = 26
- last_column = 21
- last_line = 26
- path = “/etc/icinga2/conf.d/services.conf”
templates = [ “ping4”, “generic-service” ]
% = modified in ‘/etc/icinga2/conf.d/services.conf’, lines 26:1-26:21
% = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 33:1-33:34
type = “Service”
vars
- enable_email_for = “web”
  % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 39:3-39:31
- enable_pagerduty = true
  % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 37:3-37:30
- slack_notifications = “enabled”
  % = modified in ‘/etc/icinga2/zones.d/global-templates/templates.conf’, lines 38:3-38:38
volatile = false
zone = “master”
% = modified in ‘/etc/icinga2/conf.d/services.conf’

Manali · October 4, 2023, 1:38pm

Just a question is there any way to modify slack and PagerDuty variable using Icinga Web as I have created all the objects , commands, services, host templete using web.?

Attac

hing image for your reference.

Al2Klimov · October 4, 2023, 2:05pm

There seems to be a template in templates.conf you imported only for ping4.

Manali · October 4, 2023, 2:09pm

So one question is the template.conf is responsible for slack_notification and PagerDuty?
If Yes then how to do for other services? Also I have 2 zones “global-template” and “master” so in which zone template.conf should be placed?
Is it possible using Icinga Web?

Al2Klimov · October 4, 2023, 2:38pm

OK, first of all, what is there?

Manali · October 4, 2023, 2:40pm

template Host “generic-host” {
check_command = “hostalive”
max_check_attempts = 3
check_interval = 1m
retry_interval = 30s
vars.enable_pagerduty = true
vars.slack_notifications = “enabled”
vars.enable_email_for = “web”
}

/**

Provides default settings for services. By convention
all services should import this template.
*/
template Service “generic-service” {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s
vars.enable_pagerduty = true
vars.slack_notifications = “enabled”
vars.enable_email_for = “web”
}

/**

Provides default settings for users. By convention
all users should inherit from this template.
*/

template User “generic-user” {

}

Al2Klimov · October 5, 2023, 7:58am

The services you miss notifications for seem to miss import “generic-service” as in ping4. If you created some via Director, you should add the generic-service template there.

Manali · October 5, 2023, 9:26am

But how to add notification in web?
bca the service for which import is missing is created using web.

Al2Klimov · October 5, 2023, 9:36am

On which page of web exactly?

Manali · October 5, 2023, 9:38am

this page where notification is empty but for ping4 service notification has user name.