Icinga2 integration with pagerduty(go-pdagent)

Icinga 2
,
1
  • Version used (icinga2 --r2.13.6-1)
  • Operating System and version : debian 12 bookworm
  • managed by puppet
  • however, for initiate testing just working on single icinga2 server without puppet.
    Hi ,

I am trying to integrate icinga2 with pagerduty (go-pdagent)

my pagerduty-icinga2.cfg file is as follow :

object User "pdagent" {
pager = "MY API INTEGRATION KEY "
# groups = [ "icingaadmins" ]
display_name = "PagerDuty Notification User"
states = [ OK, Warning, Critical, Unknown, Up, Down ]
types = [ Problem, Acknowledgement, Recovery ]
}

object NotificationCommand "notify-service-by-pagerduty" {
import "plugin-notification-command"
command = [ "/usr/local/bin/pdagent" ]
arguments = {
"enqueue" = {
skip_key = true
order = 0
value = "enqueue"
}
"-f" = {
order = 1
value = "pd_nagios_object=service"
}
}

env = {
"ICINGA_CONTACTPAGER" = "$user.pager$"
"ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
"ICINGA_SERVICEDESC" = "$service.name$"
"ICINGA_HOSTNAME" = "$host.name$"
"ICINGA_HOSTALIAS" = "$host.display_name$"
"ICINGA_SERVICESTATE" = "$service.state$"
"ICINGA_SERVICEOUTPUT" = "$service.output$"
}
}

object NotificationCommand "notify-host-by-pagerduty" {
import "plugin-notification-command"
command = [ "/usr/local/bin/pdagent" ]
arguments = {
"enqueue" = {
skip_key = true
order = 0
value = "enqueue"
}
"-f" = {
order = 1
value = "pd_nagios_object=host"
}
}

env = {
"ICINGA_CONTACTPAGER" = "$user.pager$"
"ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
"ICINGA_HOSTNAME" = "$host.name$"
"ICINGA_HOSTALIAS" = "$host.display_name$"
"ICINGA_HOSTSTATE" = "$host.state$"
"ICINGA_HOSTOUTPUT" = "$host.output$"
}
}

apply Notification "pagerduty-service" to Service {
command = "notify-service-by-pagerduty"
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Acknowledgement, Recovery ]
# period = "24x7"
users = [ "pdagent" ]

assign where service.vars.enable_pagerduty == true
}

apply Notification "pagerduty-host" to Host {
command = "notify-host-by-pagerduty"
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery ]
# period = "24x7"
users = [ "pdagent" ]

assign where host.vars.enable_pagerduty == true
}

and my /etc/pdagent/config.yaml is :

address: 127.0.0.1:49463
database: /var/db/pdagent/pdagent.db
pidfile: /var/run/pdagent/pidfile
region: us
secret: MY API INTEGRATION KEY
logging:
level: debug

the icinga2 logs are :

[2024-04-19 15:58:30 +1000] information/Checkable: Checkable 'icinga01-bom.trellian.com!ssh' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2024-04-19 15:58:30 +1000] information/Notification: Sending 'Problem' notification 'icinga01-bom.trellian.com!ssh!pagerduty-service' for user 'pdagent'
[2024-04-19 15:58:30 +1000] information/Notification: Completed sending 'Problem' notification 'icinga01-bom.trellian.com!ssh!pagerduty-service' for checkable 'icinga01-bom.trellian.com!ssh' and user 'pdagent' using command 'notify-service-by-pagerduty'.

and my pdagent logs are :

"level":"info","ts":1713508896.3805995,"logger":"Heartbeat","caller":"server/heartbeat.go:58","msg":"Starting heartbeat."}
{"level":"info","ts":1713508927.964813,"logger":"Server","caller":"server/middleware.go:12","msg":"Handling request: /send"}
{"level":"error","ts":1713508927.9654932,"logger":"PersistentQueue","caller":"persistentqueue/enqueue.go:23","msg":"Failed to validate event in queue .%!(EXTRA *errors.errorString=invalid routing key)","stacktrace":["github.com/PagerDuty/go-pdagent/pkg/persistentqueue.(*PersistentQueue).Enqueue\n\t/home/circleci/project/pkg/persistentqueue/enqueue.go:23\ngithub.com/PagerDuty/go-pdagent/pkg/server.(*Server).SendHandler\n\t/home/circleci/project/pkg/server/send_handler.go:24\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2041\ngithub.com/PagerDuty/go-pdagent/pkg/server.authMiddleware.func1.1\n\t/home/circleci/project/pkg/server/middleware.go:35\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2041\ngithub.com/PagerDuty/go-pdagent/pkg/server.loggingMiddleware.func1.1\n\t/home/circleci/project/pkg/server/middleware.go:13\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2041\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2836\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1924"](mailto:github.com/PagerDuty/go-pdagent/pkg/persistentqueue.(*PersistentQueue).Enqueue\n\t/home/circleci/project/pkg/persistentqueue/enqueue.go:23\ngithub.com/PagerDuty/go-pdagent/pkg/server.(*Server).SendHandler\n\t/home/circleci/project/pkg/server/send_handler.go:24\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2041\ngithub.com/PagerDuty/go-pdagent/pkg/server.authMiddleware.func1.1\n\t/home/circleci/project/pkg/server/middleware.go:35\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2041\ngithub.com/PagerDuty/go-pdagent/pkg/server.loggingMiddleware.func1.1\n\t/home/circleci/project/pkg/server/middleware.go:13\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2041\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/mux@v1.7.4/mux.go:210\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2836\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1924)}

Please assist if i am missing on something or any other dependencies or any other things i have to do…

Regards,
usman

2

Hello,

Do you succeed using the command line to invoke pdagent?

You should have a command looking like this:

pdagent enqueue
-k your_key_goes_here
-t trigger
-d “This is only a test”
-u “http://pagerduty.com”
-e “error”
-f some_field=some_value

Looking at your implementation, I do not see where this command would be constructed. Perhaps the yaml config file would provide all default values, but I am unsure.

I would:

  • find a pdagent command line that works
  • wrap the command line into a bash script
  • invoke the bash script from Icinga (not the pdagent command)
  • add processing in the bash script to transfer arguments to pdagent

Wishing you success,

Jean

1 Like
3

thanks for your response, the test command works from CLI,

i am new to icinga2 and pagerduty…
Is this the command which would be default binary installed as pdagent in /usr/local/bin ?
or the something else you meant ?

4

We will go step by step :slight_smile:

It is already very good news that the test command works from cli. Which user is logged in when running the test command? Does it also work when run as the icinga user (or whatever user is running the icinga process)?

2 Likes
5

yes it sends alert using nagios user only as test command,

sudo -u nagios pdagent enqueue   -k "integration key here"   -t trigger   -d "This is only a test"   -u "http://pagerduty.com"   -e "error"   -f some_field=some_value
{"key":"lg8rnw3i35evu98lmoht9zzixxujg69a"}

got an alert on pagerduty web interface as test alert otherwise it does not show up in pagerduty web.

6

hi,
any other information you require ?

8

Hello,

That’s already very good new. I assume Icinga is running as user nagios, right? (we run it as icinga).

I saw an email from you starting with “Hi, i have tried few steps”, but I do not see the content here.

I saw the bash script encapsulating the command. Does that bash script work when executed from the command line as “nagios”?

In the bash script, you have log statements using the “echo” command to stdout. Can you redirect the echo output to a log file, and check if the log file is populated with the expected information, when triggered from Icinga?

I would log the passed arguments and the environment, as a first debug step. Always good to log a timestamp for each line, too.

Best of luck,

Jean

2 Likes
9

Thanks for your reply.
this is the bash script and i guess it was deleted by icinga…

#!/bin/bash
# Script to check service status using go-pdagent and Icinga 2 plugin format

# Define exit codes
OK=0
WARNING=1
CRITICAL=2
UNKNOWN=3

# Check if service name is provided as argument
if [ $# -ne 1 ]; then
    echo "Usage: $0 <service_name>"
    exit $UNKNOWN
fi

# Add debug statements
echo "Notification type: $ICINGA_NOTIFICATIONTYPE"
echo "Service description: $ICINGA_SERVICEDESC"
# Redirect output to a log file
exec &>> /var/log/icinga_sh.log


# Debug output to log received arguments and command execution
echo "Received arguments: $@"
echo "Executing command: pdagent query service status $@"

# Execute the pdagent command and log its output
pdagent queue status "$@" > /tmp/pdagent_output.log 2>&1
cat /tmp/pdagent_output.log

# Check the exit code of pdagent and return an appropriate exit code
if [ $? -eq 0 ]; then
    echo "pdagent command executed successfully"
    exit 0
else
    echo "pdagent command failed with exit code $?"
    exit 3
fi


# Service name provided as argument
SERVICE_NAME=$1

# Function to query service status using go-pdagent
query_service_status() {
    # Use go-pdagent to query service status
    status=$(pdagent query service status $SERVICE_NAME)

    # Check the status and return appropriate exit code and message
    case $status in
        "OK")
            echo "Service $SERVICE_NAME is OK"
            exit $OK
            ;;
        "WARNING")
            echo "Service $SERVICE_NAME is in WARNING state"
            exit $WARNING
            ;;
        "CRITICAL")
            echo "Service $SERVICE_NAME is in CRITICAL state"
            exit $CRITICAL
            ;;
        "UNKNOWN")
            echo "Service $SERVICE_NAME is in UNKNOWN state"
            exit $UNKNOWN
            ;;
        *)
            echo "Unknown response from pdagent"
            exit $UNKNOWN
            ;;
    esac
}

# Main execution
query_service_status

and the file log output was

Executing command: pdagent query service status enqueue
{"status_items":[{"routing_key":"iNTEGRATION KEY","pending":0,"success":2,"error":0}}
pdagent command executed successfully

but still nothing in pdagent logs or web interface.
also is it possible to share the sample pdagent/config.yaml & icinga2-pagerduty.conf and the command script for go-pdagent integration.
is there anything else i need to change or check in order to make it working ?

10

Hi,

Can you please share all output and results of the bash script when run from the command line?

Thank you,

Jean

11

hi,

icinga01-bom:/usr/local/bin# ./icinga.sh ssh
output is : 
Notification type: 
Service description:

its output just have 2 lines. only the echo command i believe and its not getting the variable value.
so i added this line to the script :

# Debug output to log all environment variables
echo "All environment variables:"
printenv

and when i triggered an alert by making the ssh service down the output of the /var/log/icinga_sh.log

Received arguments: enqueue
Executing command: pdagent query service status enqueue
{"status_items":[{"routing_key":"INTEGRATION KEY HERE","pending":0,"success":2,"error":0}]}
pdagent command executed successfully
All environment variables:
ICINGA_CONTACTPAGER=INTEGRATION KEY HERE
ICINGA_SERVICEOUTPUT=connect to address 192.168.5.192 and port 22: Connection refused
ICINGA_HOSTALIAS=icinga01-bom.trellian.com
ICINGA_SERVICEDESC=ssh
PIDFILE=/run/icinga2/icinga2.pid
PWD=/
SYSTEMD_EXEC_PID=4045255
LANG=en_AU.UTF-8
INVOCATION_ID=a625737bfe204b7087b389fa32d05f0a
ICINGA_SERVICESTATE=CRITICAL
SHLVL=1
ICINGA_NOTIFICATIONTYPE=PROBLEM
ICINGA_HOSTNAME=icinga01-bom.trellian.com
JOURNAL_STREAM=8:19416131
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LC_NUMERIC=C
_=/usr/bin/printenv
Received arguments: enqueue
Executing command: pdagent query service status enqueue
{"status_items":[{"routing_key":"INTEGRATION KEY HERE","pending":0,"success":2,"error":0}]}
pdagent command executed successfully
12

Hello,

This is great, I believe the mechanics on the Icinga side are in place.

In the script below:

# Execute the pdagent command and log its output
pdagent query service status "$@" > /tmp/pdagent_output.log 2>&1
cat /tmp/pdagent_output.log
# Check the exit code of pdagent and return an appropriate exit code
if [ $? -eq 0 ]; then

The test in the if will be on the return code of the cat, not of the pdagent command.
You should adapt the script to report on the pdagent command return code. Something like:

PDAGENT_COMMAND="pdagent query service status $@"
PDAGENT_OUTPUTFILE="/tmp/pdagent_output.log"
echo "Executing command: $PDAGENT_COMMAND"
$PDAGENT_COMMAND > $PDAGENT_OUTPUTFILE 2>&1
PDAGENT_RETCODE=$?
cat $PDAGENT_OUTPUTFILE

if [ $PDAGENT_RETCODE -eq 0 ]; then

Probably the return code will not be 0, and might inform you on what is the issue (looking at pdagent documentation).

Best regards,

Jean

13

thanks,
my scripts is now as :

#!/bin/bash
# Script to check service status using go-pdagent and Icinga 2 plugin format

# Define exit codes
OK=0
WARNING=1
CRITICAL=2
UNKNOWN=3

# Check if service name is provided as argument
if [ $# -ne 1 ]; then
    echo "Usage: $0 <service_name>"
    exit $UNKNOWN
fi

# Add debug statements
echo "Notification type: $ICINGA_NOTIFICATIONTYPE"
echo "Service description: $ICINGA_SERVICEDESC"
# Redirect output to a log file
exec &>> /var/log/icinga_sh.log

# Debug output to log all environment variables
echo "All environment variables:"
printenv


# Debug output to log received arguments and command execution
echo "Received arguments: $@"
echo "Executing command: pdagent enqueue service status $@"

# Execute the pdagent command and log its output
pdagent enqueue status "$@" > /tmp/pdagent_output.log 2>&1
cat /tmp/pdagent_output.log

# Check the exit code of pdagent and return an appropriate exit code
PDAGENT_COMMAND="pdagent enqueue service status $@"
PDAGENT_OUTPUTFILE="/tmp/pdagent_output.log"
echo "Executing command: $PDAGENT_COMMAND"
$PDAGENT_COMMAND > $PDAGENT_OUTPUTFILE 2>&1
PDAGENT_RETCODE=$?
cat $PDAGENT_OUTPUTFILE

if [ $PDAGENT_RETCODE -eq 0 ]; then
    echo "pdagent command executed successfully"
    exit 0
else
    echo "pdagent command failed with exit code $?"
    exit 3
fi


# Service name provided as argument
SERVICE_NAME=$1

# Function to query service status using go-pdagent
query_service_status() {
    # Use go-pdagent to query service status
    status=$(pdagent enqueue service status $SERVICE_NAME)

    # Check the status and return appropriate exit code and message
    case $status in
        "OK")
            echo "Service $SERVICE_NAME is OK"
            exit $OK
            ;;
        "WARNING")
            echo "Service $SERVICE_NAME is in WARNING state"
            exit $WARNING
            ;;
        "CRITICAL")
            echo "Service $SERVICE_NAME is in CRITICAL state"
            exit $CRITICAL
            ;;
        "UNKNOWN")
            echo "Service $SERVICE_NAME is in UNKNOWN state"
            exit $UNKNOWN
            ;;
        *)
            echo "Unknown response from pdagent"
            exit $UNKNOWN
            ;;
    esac
}

# Main execution
enqueue_service_status

and the pdagent/config.yaml is same

the /tmp/pdagent_output_log shows :

Post "http://127.0.0.1:49463/send": dial tcp 127.0.0.1:49463: connect: connection refused

and the icnga2.log shows :

[2024-04-29 21:07:32 +1000] information/Checkable: Checkable 'icinga01-bom.trellian.com!ssh' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2024-04-29 21:07:32 +1000] information/Notification: Sending 'Problem' notification 'icinga01-bom.trellian.com!ssh!pagerduty-service' for user 'pdagent'
[2024-04-29 21:07:32 +1000] information/Notification: Completed sending 'Problem' notification 'icinga01-bom.trellian.com!ssh!pagerduty-service' for checkable 'icinga01-bom.trellian.com!ssh' and user 'pdagent' using command 'notify-service-by-pagerduty'.
[2024-04-29 21:07:32 +1000] warning/PluginNotificationTask: Notification command for object 'icinga01-bom.trellian.com!ssh' (PID: 737, arguments: '/usr/local/bin/icinga.sh' 'enqueue') terminated with exit code 3, output: Notification type: PROBLEM
Service description: ssh

and the script logs shows :

Received arguments: enqueue
Executing command: pdagent enqueue service status enqueue
Post "http://127.0.0.1:49463/send": dial tcp 127.0.0.1:49463: connect: connection refused
Executing command: pdagent enqueue service status enqueue
Post "http://127.0.0.1:49463/send": dial tcp 127.0.0.1:49463: connect: connection refused
pdagent command failed with exit code 1
All environment variables:
ICINGA_CONTACTPAGER=INTEGRATION KEY HERE
ICINGA_SERVICEOUTPUT=connect to address 192.168.5.192 and port 22: Connection refused
ICINGA_HOSTALIAS=icinga01-bom.trellian.com
ICINGA_SERVICEDESC=ssh
PIDFILE=/run/icinga2/icinga2.pid
PWD=/
SYSTEMD_EXEC_PID=429
LANG=en_AU.UTF-8
INVOCATION_ID=af68af5ff61347209b9e3c93f173a024
ICINGA_SERVICESTATE=CRITICAL
SHLVL=1
ICINGA_NOTIFICATIONTYPE=PROBLEM
ICINGA_HOSTNAME=icinga01-bom.trellian.com
JOURNAL_STREAM=8:19120
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LC_NUMERIC=C
_=/usr/bin/printenv
Received arguments: enqueue
Executing command: pdagent enqueue service status enqueue
Post "http://127.0.0.1:49463/send": dial tcp 127.0.0.1:49463: connect: connection refused
Executing command: pdagent enqueue service status enqueue
Post "http://127.0.0.1:49463/send": dial tcp 127.0.0.1:49463: connect: connection refused
pdagent command failed with exit code 1
14

Please try executing the exact same command from the command line, as user icinga:

sudo -u icinga pdagent enqueue service status enqueue

Please confirm the output is the same, or report the exact output.

15

Looking at a previous post (#4 of 22 April), I realise you need to invoke the pdagent command with more parameters in order to have it working:

Can you adapt the script so it does that, based on environment variables?

16 
sudo -u icinga pdagent enqueue service status enqueue
sudo: unknown user icinga
sudo: error initializing audit plugin sudoers_audit
17

i added the parameter to pagerduty-icinga2.conf

object NotificationCommand "notify-service-by-pagerduty" {
  import "plugin-notification-command"
  command = [ "/usr/local/bin/icinga.sh" ]
 arguments = {
    "enqueue" = {
      skip_key = true
      order = 0
      value = "enqueue"
    }
    "integration_key" = {
      skip_key = true
      order = 1
      value = "INTEGRATION KEY HERE"
    }
    "trigger" = {
      skip_key = true
      order = 2
      value = "trigger"
    }
    "message" = {
      skip_key = true
      order = 3
      value = "This is only a test"
    }
    "details" = {
      skip_key = true
      order = 4
      value = "some_field=some_value"
    }
    // Add more arguments as needed
  }

  env = {
    "ICINGA_CONTACTPAGER" = "$user.pager$"
    "ICINGA_NOTIFICATIONTYPE" = "$notification.type$"
    "ICINGA_SERVICEDESC" = "$service.name$"
    "ICINGA_HOSTNAME" = "$host.name$"
    "ICINGA_HOSTALIAS" = "$host.display_name$"
    "ICINGA_SERVICESTATE" = "$service.state$"
    "ICINGA_SERVICEOUTPUT" = "$service.output$"
  }
}

and the output for icinga2.logs is

[2024-04-30 13:59:03 +1000] information/Notification: Completed sending 'Problem' notification 'icinga01-bom.trellian.com!ssh!pagerduty-service' for checkable 'icinga01-bom.trellian.com!ssh' and user 'pdagent' using command 'notify-service-by-pagerduty'.
[2024-04-30 13:59:03 +1000] warning/PluginNotificationTask: Notification command for object 'icinga01-bom.trellian.com!ssh' (PID: 63691, arguments: '/usr/local/bin/icinga.sh' 'enqueue' 'b30b453a31e44f0ed06a260060616483' 'trigger' 'This is only a test' 'some_field=some_value') terminated with exit code 3, output:

and the icinga_sh.log :

TERM=xterm-256color
USER=root
SHLVL=2
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SUDO_UID=11342
MAIL=/var/mail/root
OLDPWD=/var/run/pdagent
_=/usr/bin/printenv
Received arguments: ssh
Executing command: pdagent enqueue service status ssh
{"errors":["invalid routing key"]}
Executing command: pdagent enqueue service status ssh
{"errors":["invalid routing key"]}
pdagent command executed successfully

just to confirm the routing key is the same as integration key ?
and it is i have the same one entered

18

I think in your case the icinga user is “nagios”. Can you please do it again with:

sudo -u nagios pdagent enqueue service status enqueue

And post here the full output, please.

Looking at one of the earliest posts, what you really wanted to do is:

sudo -u nagios pdagent enqueue \
-k "integration key here"   \
-t trigger   \
-d "This is only a test"   \
-u "http://pagerduty.com"   \
-e "error"   \
-f some_field=some_value  \
{"key":"lg8rnw3i35evu98lmoht9zzixxujg69a"}

Has the scope changed in the meantime, or is the “status” command just a way to simplify the problem?

19

yes , the first one is just for checking the status actual have to do the 2nd one only
sudo -u nagios pdagent enqueue service status enqueue
{“errors”:[“Unauthorized, expected matching secret token in Authorization header.”]}

and
sudo -u nagios pdagent enqueue -k Integration key -t trigger -d “This is only a test” -u “https://events.pagerduty.com/integration/integrationkey/enqueue” -e “error” -f some_field=some_value

{“errors”:[“Unauthorized, expected matching secret token in Authorization header.”]}

20

Hi,

First step is to have a command line that works. This requires pdagent competencies, which I do not have.

Second step is to have your script that works, i.e., generating the correct command. This requires Linux/bash competencies.

Third step is to have Icinga invoke the script with the correct parameters. This requires Icinga competencies, and this community will surely be able to help.

So I think now you need to address the first step, as you still have errors there.

Best of luck,

Jean

1 Like
21

Hi Jean,
thanks for that,
i have completed the first 2 steps as mentioned and sharing below the same
used the GitHub - PagerDuty/pdagent-integrations
and hence have script for pd-nagios as per the Icinga 2 Integration Guide | PagerDuty
and logs for icinga says successfully send command afetr some script modifications :

information/Checkable: Checkable 'icinga01-mel.trellian.com!ssh' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2024-05-03 17:06:15 +1000] information/Notification: Sending 'Problem' notification 'icinga01-mel.trellian.com!ssh!pagerduty-service' for user 'pdagent'
[2024-05-03 17:06:15 +1000] information/Notification: Completed sending 'Problem' notification 'icinga01-mel.trellian.com!ssh!pagerduty-service' for checkable 'icinga01-mel.trellian.com!ssh' and user 'pdagent' using command 'notify-service-by-pagerduty'

and the pd-nagios script is :

#!/usr/bin/python3
#
# Python script to enqueue an event from Nagios to send to PagerDuty.
#
# Copyright (c) 2013-2014, PagerDuty, Inc. <info@pagerduty.com>
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
#   * Redistributions of source code must retain the above copyright
#     notice, this list of conditions and the following disclaimer.
#   * Redistributions in binary form must reproduce the above copyright
#     notice, this list of conditions and the following disclaimer in the
#     documentation and/or other materials provided with the distribution.
#   * Neither the name of the copyright holder nor the
#     names of its contributors may be used to endorse or promote products
#     derived from this software without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
#


class NagiosEvent:
    HOST_NAME = "HOSTNAME"
    HOST_STATE = "HOSTSTATE"
    SERVICE_DESCRIPTION = "SERVICEDESC"
    SERVICE_STATE = "SERVICESTATE"

    REQUIRED_FIELDS = {
        "host": set([HOST_NAME, HOST_STATE]),
        "service": set([HOST_NAME, SERVICE_DESCRIPTION, SERVICE_STATE])
    }

    def __init__(self, service_key, event_type, notification_type, details, incident_key=None):
        self._service_key = service_key
        self._event_type = event_type
        self._notification_type = notification_type
        self._details = dict(list(details.items()) +
                             [("pd_nagios_object", self._notification_type)])
        self._given_incident_key = incident_key

    def enqueue(self):
        from pdagent.config import load_agent_config
        from pdagent.pdagentutil import queue_event
        agent_config = load_agent_config()
        self._check_required_fields()
        return queue_event(
            agent_config.get_enqueuer(),
            self._pagerduty_event_type(),
            self._service_key,
            self._incident_key(),
            self._event_description(),
            "",
            "",
            self._details,
            agent_config.get_agent_id(),
            "pd-nagios"
        )

    def _event_description(self):
        if self._notification_type == "service":
            fields = [NagiosEvent.SERVICE_DESCRIPTION,
                      NagiosEvent.SERVICE_STATE,
                      NagiosEvent.HOST_NAME
                      ]
        elif self._notification_type == "host":
            fields = [NagiosEvent.HOST_NAME, NagiosEvent.HOST_STATE]
        else:
            return None

        pairs = ["{0}={1}".format(field, self._details[field])
                 for field in fields]
        return "; ".join(pairs)

    def _incident_key(self):
        if self._given_incident_key is None:
            if self._notification_type == "service":
                return "event_source=service;host_name={0};service_desc={1}".format(
                    self._details[NagiosEvent.HOST_NAME],
                    self._details[NagiosEvent.SERVICE_DESCRIPTION]
                )
            elif self._notification_type == "host":
                return "event_source=host;host_name={0}".format(self._details[NagiosEvent.HOST_NAME])
            else:
                raise ValueError(
                    "notification_type must be one of 'service' or 'host'")
        else:
            return self._given_incident_key

    def _check_required_fields(self):
        if not NagiosEvent.REQUIRED_FIELDS[self._notification_type].issubset(list(self._details.keys())):
            msg = "Missing fields for type '{0}'.  {1} required".format(
                self._notification_type,
                ", ".join(NagiosEvent.REQUIRED_FIELDS[self._notification_type])
            )
            raise ValueError(msg)

    def _pagerduty_event_type(self):
        return {
            "PROBLEM": "trigger",
            "ACKNOWLEDGEMENT": "acknowledge",
            "RECOVERY": "resolve"
        }[self._event_type]


def main():
    description = "Enqueue an event from Nagios to PagerDuty."
    parser = build_queue_arg_parser(description)
    args = parser.parse_args()
    details = parse_fields(args.fields)

    event = NagiosEvent(
        args.service_key,
        args.event_type,
        args.notification_type,
        details,
        args.incident_key
    )

    try:
        incident_key = event.enqueue()
        print("Event processed. Incident Key:", incident_key)
    except ValueError as e:
        parser.error(e.message)


def build_queue_arg_parser(description):
    from pdagent.thirdparty.argparse import ArgumentParser
    parser = ArgumentParser(description=description)
    parser.add_argument(
        "-k",
        "--service-key",
        dest="service_key",
        required=True,
        help="Service API Key"
    )
    parser.add_argument(
        "-t",
        "--event-type",
        dest="event_type",
        required=True,
        help="Event type",
        choices=["PROBLEM", "ACKNOWLEDGEMENT", "RECOVERY"]
    )
    parser.add_argument(
        "-i",
        "--incident-key",
        dest="incident_key",
        help="Incident Key"
    )
    parser.add_argument(
        "-f",
        "--field",
        action="append",
        dest="fields",
        help="Add given KEY=VALUE pair to the event details"
    )
    parser.add_argument(
        "-n",
        "--notification-type",
        dest="notification_type",
        required=True,
        help="Nagios notification type (host or service)",
        choices=["service", "host"]
    )
    return parser


def parse_fields(fields):
    if fields is None:
        return {}
    return dict(f.split("=", 1) for f in fields)

#if __name__ == "__main__":
#    try:
#        import pdagent.config
#    except ImportError:
#        # Fix up for dev layout
#        import sys
#        from os.path import realpath, dirname
#        sys.path.append(dirname(dirname(realpath(__file__))))
#        import pdagent.config
#    main()


import subprocess

if __name__ == "__main__":
    command = ["/usr/local/bin/pdagent"]
    arguments = [
        "-n", "service",
        "-k", "$user.pager$",
        "-t", "$notification.type$",
        "-f", "SERVICEDESC=$service.name$",
        "-f", "SERVICEDISPLAYNAME=$service.display_name$",
        "-f", "HOSTNAME=$host.name$",
        "-f", "HOSTSTATE=$host.state$",
        "-f", "HOSTDISPLAYNAME=$host.display_name$",
        "-f", "SERVICESTATE=$service.state$",
        "-f", "SERVICEPROBLEMID=$service.state_id$",
        "-f", "SERVICEOUTPUT=$service.output$"
    ]

    # Replace $user.pager$ and other placeholders with actual values as needed
    # For example:
    # arguments[arguments.index("-k") + 1] = "actual_pager_value"

    # Call pd-nagios binary with the constructed command and arguments
    subprocess.run(command + arguments)


#import subprocess

#if __name__ == "__main__":
#    try:
#        # Call pdagent binary with appropriate arguments
#        subprocess.run(['/usr/local/bin/pdagent', ])
#    except FileNotFoundError:
#        print("Error: pdagent binary not found at /usr/local/bin/pdagent")



#if __name__ == "__main__":
#    try:
#        import config.yaml
#    except ImportError:
#        # Fix up for dev layout
#        import sys
#        from os.path import realpath, dirname
#        sys.path.append(dirname(dirname(realpath(__file__))))
#        import config.yaml
#    main()

#if __name__ == "__main__":
#    import sys
#    from os.path import realpath, dirname, join
#
#    # Add the directory containing pdagent to the Python path
#    pdagent_dir = '/usr/local/bin/pdagent'  # Update this with the correct path
#    sys.path.append(pdagent_dir)
#
#    try:
#        import pdagent
#        from pdagent import load_config
#    except ImportError:
#        # Fix up for dev layout
#        sys.path.append(dirname(dirname(realpath(__file__))))
#        import pdagent
#        from pdagent import load_config
#
#    # Load configuration from pdagent.yaml
#    config = load_config('/etc/pdagent/config.yaml')
#
#    # Now you can access configuration values like config['key']
#
#    # Call your main function here
#    main()

======
and
command line :

sudo -u nagios pdagent enqueue \
-k "Integration KEY HERE"   \
-t trigger   \
-d "This is only a test"   \
-u "https://events.pagerduty.com/integration/Integration KEY HERE/enqueue"   \
-e "error"   \
-f HOSTDISPLAYNAME=icinga01-mel.trellian.com

pdagent logs:

{"level":"info","ts":1714712333.736411,"logger":"PersistentQueue","caller":"persistentqueue/enqueue.go:31","msg":"Enqueuing to Integration KEY HERE with key 70kd9qp0rin6ook62eyb2nos693wk4s4."}
{"level":"info","ts":1714712333.7454152,"logger":"PersistentQueue","caller":"persistentqueue/enqueue.go:37","msg":"Event enqueued with key 70kd9qp0rin6ook62eyb2nos693wk4s4, ID 1."}
{"level":"info","ts":1714712333.7455108,"logger":"PersistentQueue","caller":"persistentqueue/enqueue.go:50","msg":"Enqueuing 70kd9qp0rin6ook62eyb2nos693wk4s4 with EventQueue."}
{"level":"info","ts":1714712333.7498248,"logger":"EventQueue.Integration KEY HERE","caller":"eventqueue/eventqueue.go:138","msg":"Worker started."}
{"level":"info","ts":1714712333.7498968,"logger":"EventQueue.Integration KEY HERE","caller":"eventqueue/eventqueue.go:140","msg":"Job started, 0 pending."}
{"level":"info","ts":1714712334.638593,"logger":"PersistentQueue","caller":"persistentqueue/enqueue.go:63","msg":"EventQueue returned success for 70kd9qp0rin6ook62eyb2nos693wk4s4. "}
{"level":"info","ts":1714712334.6490712,"logger":"PersistentQueue","caller":"persistentqueue/enqueue.go:70","msg":"Set status of 70kd9qp0rin6ook62eyb2nos693wk4s4 to success."}

now need assistance with 3rd step…