Automatically removes a service acknowledgement when plugin output changes

Icinga 2
1

I have created an Icinga 2 event command that automatically removes a service acknowledgement when the plugin output changes.

While searching for this functionality, I found many users asking for a similar feature, but I was unable to find an existing solution. To address this, I developed an event command that monitors the service output and removes the acknowledgement whenever the output text changes.

Use Case

A service enters a CRITICAL state and is acknowledged while you investigate the issue. The service remains CRITICAL, but later the plugin output changes because the underlying problem has changed.

Under normal circumstances, the acknowledgement remains in place, which can prevent the new condition from receiving the attention it deserves. This event command automatically removes the acknowledgement when the service output changes, allowing the new issue to be reviewed and, if required, generate notifications again.

GitHub Repository

The code is available on GitHub:

I hope this project is useful to other Icinga users who are looking for similar functionality. Feedback, suggestions, and contributions are welcome.

2

I really like the idea of that.

I ran it through ChatGPT GPT-5.6 Sol and it recognised a few potential problems, i removed the ones I didn’t agree with:

  • An empty plugin output is treated the same as having no previous observation. It would be safer to check whether the tracking file exists rather than whether the previous output is empty.

  • The state filename can potentially collide because it is made from the sanitised host and service names joined with _. Using a hash of the host/service combination would make this safer.

  • A successful curl exit code only confirms that the HTTP request succeeded. It would be better to check the Icinga API JSON response as well to make sure the acknowledgement was actually removed.

  • Passing the API username/password directly to curl --user may expose the credentials in some environments. A protected config/netrc file would be safer.

The name collision is very unlikely for most scenarios.

ChatGPT also questioned the api reachability of the API and proposed a retry but since icinga2 starts the eventcommand the api should be available anyway.

you could do certificate based authentication which will remove the whole api user password part.

  • just create new certificates from the icinga-ca with some nodename like acknuser.
  • make it readonly for nagios:nagios.
  • use the cert instead of the credentials.
  • add a new apiuser that uses certs instead of the password.

Otherwise, I really like the approach.