ICINGA2 integrating with Pagerduty working except for 'acknowledge' event types

gh001 · January 11, 2022, 6:49am

icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1)
CentOS Linux release 7.9.2009 (Core)
icinga2 daemon -C - Returns clean

So we have the pdagent integration between our ICINGA2 installation and Pagerduty. It used to Create, Acknowledge and Resolve just fine on ICINGA. When upgraded to ICINGA2, only the Acknowledge stopped working.

I did verify that manually acknowledging an alert in PagerDuty using the pdagent utility pd-send works.

It appears that ICINGA2 is not calling pdagent for acknowledgements. I did the following to reproduce and find errors:

I enabled icinga2 in debug mode.
Toggled off and on an acknowledgment on an issue in ICINGA2 to see if anything triggered in the logs.
I got the below in the debug log. Note, that when acknowledging an alert, ICINGA2 wants me to put in a comment (which is fine). I put in the comment “re-ack Gregor”. Api listener is dropping the update (acknowledgment). It appears the ICINGA2 API drops it due to some misconfig?

[2022-01-11 06:12:12 +0000] warning/ApiListener: Ignoring config update from ‘master2-icinga.int.cloud.ruckuswireless.com’…’ of type ‘Comment’. ‘api’ does not accept config.

Looking for help understanding what may be the issue.

Thank You!

stevie-sy · January 11, 2022, 7:54am

Hi and welcome,

first of all I’m not familiar with pagerduty - I think others can help here more. But maybe you should check following first:

the configured user für notification
your notificiation rule

Maybe you don’t set the configuraiton type “acknowledgement” to this. Check also the docs about this topic: Monitoring Basics - Icinga 2.

After check this:

your notification script, which is sending the message to pagerduty. Maybe a typing or some characters in the message which pagerduty doesn’t like?

If you don’t find anything it would help to post the config objects here

MarcusCaepio · January 11, 2022, 2:57pm

Could be related to Scheduled downtime in with HA masters - #5 by sethlyons
Do you have set accept_config = true on (both) masters?

gh001 · January 11, 2022, 4:35pm

Hello Stevie,
Appreciate the welcome. The notification feature is enabled and I have placed our conf file below. This all works, just not the Acknowledgement type.

Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql mainlog notification

object User “pagerduty” {
pager = “R0BLAH”
groups = [ “icingaadmins” ]
display_name = “PagerDuty Notification User”
states = [ Critical, Up, Down ]
types = [ Custom, Problem, Acknowledgement, Recovery ]
}
object NotificationCommand “notify-service-by-pagerduty” {
import “plugin-notification-command”
command = [ “/usr/share/pdagent-integrations/bin/pd-nagios” ]
arguments = {
“-n” = {
order = 0
value = “service”
}
“-k” = {
order = 1
value = “$user.pager$”
}
“-t” = {
order = 2
value = “$notification.type$”
}
“-f” = {
order = 3
repeat_key = true
value = “$f_args$”
}
}

vars.f_args = [
“SERVICEDESC=$service.name$”,
“SERVICEDISPLAYNAME=$service.display_name$”,
“HOSTNAME=$host.name$”,
“HOSTSTATE=$host.state$”,
“HOSTDISPLAYNAME=$host.display_name$”,
“SERVICESTATE=$service.state$”,
“SERVICEPROBLEMID=$service.state_id$”,
“SERVICEOUTPUT=$service.output$”,
“ICINGA_URL=$service.vars.icinga_url$/icingaweb2/monitoring/service/show?host=$host.name$&service=$service.name$”
]
}
object NotificationCommand “notify-host-by-pagerduty” {
import “plugin-notification-command”
command = [ “/usr/share/pdagent-integrations/bin/pd-nagios” ]
arguments = {
“-n” = {
order = 0
value = “host”
}
“-k” = {
order = 1
value = “$user.pager$”
}
“-t” = {
order = 2
value = “$notification.type$”
}
“-f” = {
order = 3
repeat_key = true
value = “$f_args$”
}
}

vars.f_args = [
“HOSTNAME=$host.name$”,
“HOSTSTATE=$host.state$”,
“HOSTPROBLEMID=$host.state_id$”,
“HOSTOUTPUT=$host.output$”,
“ICINGA_URL=$service.vars.icinga_url$/icingaweb2/monitoring/host/show?host=$host.name$”
]
}

apply Notification “pagerduty-service” to Service {
command = “notify-service-by-pagerduty”
states = [ Critical ]
types = [ Custom, Problem, Acknowledgement, Recovery ]
period = “24x7”
users = [ “pagerduty” ]
assign where service.vars.enable_pagerduty == true
}

apply Notification “pagerduty-host” to Host {
command = “notify-host-by-pagerduty”
states = [ Up, Down ]
types = [ Custom, Problem, Acknowledgement, Recovery ]
period = “24x7”
users = [ “pagerduty” ]
assign where host.vars.enable_pagerduty == true
}

stevie-sy · January 12, 2022, 9:56am

Hi,

first of all it would be great to format your code like written in the FAQ FAQ - Icinga Community. It’s better readable for everybody

I’ll see you put everywhere the type “Acknowledgement”. From this point it must be sent. Btw: looks like this is the official config file from pagerduty what is decribed here Icinga 2 Integration Guide | PagerDuty, right?

What happens if you run the notification command manualy at the console like Icinga would trigger it?
like


/usr/share/pdagent-integrations/bin/pd-nagios -n [hostname] -k [pagerdutyuser] - t Acknowledgement -f...

gh001 · January 13, 2022, 5:34am

Hello Stevie,
Yes, the config above is very vanila and from the official config. As mentioned only the Acknowledgment is not working.

If I do call it manually with pd-agent it works. So I know the configuration/integration between the pagerduty agent and pagerduty is working fine. When calling pd-agent manually I can see pd-queue pick up the TXT file and send the payload to Pager Duty and it picks it up.

gh001 · January 13, 2022, 5:38am

Hello Marcus,
I made the changes as you mentioned. It did not fix my issue BUT the error messages mentioned went away. I saw in your helpful link that is the ‘new’ standard and the previous setting of accept_config=false was a workaround. Thank you for that. I suppose you have helped me fix a separate issue

gh001 · January 13, 2022, 6:14am

Also, to answer any questions regarding if the Host or Service are setup correctly I did follow the chain of inheritance from any defined object Host or apply Service config back to the following generic templates where var.enable_pagerduty = true as per the integration guide.

template Host "generic-host" {
  check_command = "hostalive"
  check_command = "hostalive"
  retry_interval = 1m
  max_check_attempts = 2
  vars.slack_notifications = "enabled"
  vars.enable_pagerduty = true
  enable_flapping = true
  flapping_threshold_low = 5
  flapping_threshold_high = 10
}

template Service "generic-service" {
  retry_interval = 1m
  max_check_attempts = 2
  check_interval = 5m
  vars.slack_notifications = "enabled"
  vars.enable_pagerduty = true
  enable_flapping = true
  flapping_threshold_low = 5
  flapping_threshold_high = 10
}

Also, debug log is clean now after applying @MarcusCaepio suggestion. So no errors now in the debug log but Acknowledgements still not working.

stevie-sy · January 13, 2022, 7:35am

Ok, if everything seems to be correct and you write there is no hint in the debug log, this situation is weird. Because we don’t use pagerduty to test it, I have no more ideas. I hope for you others have more ideas

MarcusCaepio · January 14, 2022, 8:44am

If I do call it manually with pd-agent it works.

Silly question maybe, but did you test it as the Icinga user? Or is something missing in the Notification? E.g. the Docs describe, that ACKS over API must have a filter:

In addition to these parameters a filter must be provided. The valid types for this action are Host and Service .

https://icinga.com/docs/icinga-2/latest/doc/12-icinga2-api/#acknowledge-problem