How to store the output of a notification command in a variable

We use a notification command to log a ticket on an external ticketing system, in the same way that notifications can be sent via email and sms etc. When the custom notification command executes for the first time on a service, it returns a ticket number from the external service desk system. This ticket number is the reference that is required to find the specific ticket on that system. Our problem is: How can this reference number be stored in Icinga, so that when the the service executes again, it is available and can be used to update the existing ticket on the external service desk, instead of logging a duplicate (second) ticket for the same problem?

Hi & welcome to the icinga community,

I’d do this within your notifications e.g. store host and service name together with the received ticket number somewhere locally (as text file or db entry depending on HA or not). Each time a notification is triggered then, your script check if there is a ticket number first and update notification accordingly.

Another option would be using icinga’s Rest API to store the ticket number as acknowledge comment. In this case I have no clue what will happen to it by state change e.g. WARNING to CRITICAL or CRITICAL to OK.

BTW: Are you aware that icinga triggers a notification on a state change only?

Thank you. Will be looking into the first option. I am aware that state changes triggers notifications - that is sufficient for this requirement. E.g. change from OK->CRITICAL will log a new external ticket (status=OPEN) and result in manual intervention, once the problem is fixed the state change from CRITICAL->OK will trigger a status update on the external ticket (status=RESOLVED)

PS: is there a way to store the response from a check/event /notification commands in an icinga runtime variable?

You don’t need to. Check “Runtime Attributes:” under
https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#service
and
https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#notification

Antony.

Not really. You could do this indirectly by adding it to the plugin output (which is then also stored in the db). Another approach is to send the last check result to the plugin so that the plugin can include it in its logic.

@jack we use this exact solution. Icinga generates a notification (based on rules we have) that puts a message into RabbitMQ. Another service that we wrote pulls the message from RabbitMQ, checks Icinga comments on the service to see if any existing comments tie back to tickets, checks the ticketing system (Microsoft Dynamics CRM :face_vomiting: ) for said tickets. If tickets do not exist or are close, we generate a new one and acknowledge the serve with a comment containing that ticket number. If a ticket does exist, we update the ticket with the new alert output, and re-acknowledge the ticket.

This logic (for us) works because we have a helper script that checks comments (and acknowledgements) for tickets, and then checks all tickets returned in this way on the CRM to see if they’re closed. If so, the comment data gets removed from the service.