Title: How to generate dynamic service display_name
Description: We are running PromQLs from ICINGA service by using check_command and getting details related failed pods as plugin output of service.
We want pod-name and namespace to be included as part of service display name. So that it is easy to identify alert. As we further sending pagerduty notifications. So monitoring team can also easily identify issue from title by having granular information
Currently service display name :- “Containers are not ready”
Required service display name :- “Containers are not ready for $pod_name in $namespace”
Please let us know the possible solution of generating dynamic service name.
Other information:
Icinga Web 2 version - 2.9.5
Used modules and their versions (System - About) -
However, Icinga notifications are sufficiently flexible that you can definitely
include the hostname and any other dynamic attributes you want in a
notification sent out by email (which I use and have heavily customised) or to
pagerduty (which I have no familiarity with).
Thanks Antony, I am checking at notifications also.
Its just that PD/notifications is not where we want to tackle this. We need to have an alert in icinga per object that isn’t healthy.
Icinga wouldn’t care about large number of services in non OK state.
If the user care about it, we can maybe provide a filter. Even if we can’t provide a filter and UI becomes messy it’s still better than what we have now which just masks reality. Since each service plugin output has multiple pods and if more pod failures then its difficult track if any further failure happened.
you could adapt that or create a new one, add new parameters to it and.
the script above gets the name by argument -n, you can edit the command template not to get the display name but to bulid a new display name from icinga_vars and add it to the script argument -n
Actually I want plugin output to be used in such a way that I can generate multiple services based on that output. So how we can utilize plugin output json response in variable and that variable can be used to generate dynamic service display name.