How to generate dynamic service display_name

Title: How to generate dynamic service display_name

Description: We are running PromQLs from ICINGA service by using check_command and getting details related failed pods as plugin output of service.

We want pod-name and namespace to be included as part of service display name. So that it is easy to identify alert. As we further sending pagerduty notifications. So monitoring team can also easily identify issue from title by having granular information

Currently service display name :- “Containers are not ready”
Required service display name :- “Containers are not ready for $pod_name in $namespace”

Please let us know the possible solution of generating dynamic service name.

Other information:

  • Icinga Web 2 version - 2.9.5
  • Used modules and their versions (System - About) -
  • Web browser used - chrome(100.0.4896.127)
  • Icinga 2 version used -
  • PHP version used -
  • Server operating system and version - GKE cloud

Service.display_name is not a runtime attribute, therefore I doubt that you
can define it to be dynamic and display differently for different hosts.

https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#service

However, Icinga notifications are sufficiently flexible that you can definitely
include the hostname and any other dynamic attributes you want in a
notification sent out by email (which I use and have heavily customised) or to
pagerduty (which I have no familiarity with).

https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#notification
https://icinga.com/docs/icinga-2/latest/doc/09-object-types/
#notificationcommand

Antony.

Thanks Antony, I am checking at notifications also.

Its just that PD/notifications is not where we want to tackle this. We need to have an alert in icinga per object that isn’t healthy.
Icinga wouldn’t care about large number of services in non OK state.
If the user care about it, we can maybe provide a filter. Even if we can’t provide a filter and UI becomes messy it’s still better than what we have now which just masks reality. Since each service plugin output has multiple pods and if more pod failures then its difficult track if any further failure happened.

which notification script do you use?
https://github.com/Icinga/icinga2/blob/master/etc/icinga2/scripts/mail-service-notification.sh

you could adapt that or create a new one, add new parameters to it and.

the script above gets the name by argument -n, you can edit the command template not to get the display name but to bulid a new display name from icinga_vars and add it to the script argument -n

Actually I want plugin output to be used in such a way that I can generate multiple services based on that output. So how we can utilize plugin output json response in variable and that variable can be used to generate dynamic service display name.

That is a very strange idea but here are my thoughts:

  1. Adapt your notification script to parse the plugin output
  2. Use IcingaApi inside your notification script to update the displayname
  3. If you use icinga Director: Use IcingaDirectorApi inside your notification script to update the displayname