Exploring globals & custom debug function

Hello,
I’m not too sure this post belongs here, correct me if i’m wrong !

So, i’d like to share you a trick and a custom function that can come handy in debugging or developping user functions.

Dump and explore Icinga internals

globals

Typing this in the console, or using it directly in a function will return you a big array of the whole execution context of user made function declared within globals. Since at first glance it looks unreadable, you can peacefully explore it using globals.keys()
You should have a result like this :
[ "ActiveStages", "Icinga", "Internal", "ManubulonPluginDir", "MaxConcurrentChecks", "NodeName", "NscpPath", "PluginContribDir", "PluginDir", "ReloadTimeout", "StatsFunctions", "System", "TicketSalt", "Types", "ZoneName" ]

So, it returns you static constants, whether they are defined in the constants.conf file or somewhere else, internal objects, other declared user functions and internal DSL.

For example, you can list undocumented but implemented functions for a given type of object, which can come very handy to develop specific user functions sometime.
Alongside Service object definition, you’ll find those functions that may come handy :
globals["Types"]["Service"]

modify_attribute = {
        arguments = [ "attr", "value" ]
        deprecated = false
        name = "ConfigObject#modify_attribute"
        side_effect_free = false
        type = "Function"
}
restore_attribute = {
        arguments = [ "attr", "value" ]
        deprecated = false
        name = "ConfigObject#restore_attribute"
        side_effect_free = false
        type = "Function"
}

Other example :
globals["Icinga"]
You can find icinga internal constants, which may be useful to know for debugging/understanding some parts.

{
        Acknowledgement = "Acknowledgement"
        Critical = "Critical"
        Custom = "Custom"
        DbCatAcknowledgement = 4.000000
        DbCatCheck = 256.000000
        DbCatComment = 8.000000
        DbCatConfig = 1.000000
        DbCatDowntime = 16.000000
        DbCatEventHandler = 32.000000
        DbCatEverything = -1.000000
        DbCatExternalCommand = 64.000000
        DbCatFlapping = 128.000000
        DbCatLog = 512.000000
        DbCatNotification = 1024.000000
        DbCatProgramStatus = 2048.000000
        DbCatRetention = 4096.000000
        DbCatState = 2.000000
        DbCatStateHistory = 8192.000000
[...]

Troubleshoot late checks in cluster
I got inspired by the function already present in the documentation as example and enhanced it to output more informations to tell precisely what Service or Host check is late and on which satellite based on the last check source known, Here is the function :

   globals.debug_late_checks = function(verbosity) {
   var res = {}
   var details = {}
   if (typeof(verbosity)["name"] != "Number" || verbosity < 0 || verbosity > 2 ) {
   res.set("set verbosity properly","0 = No details about devices, 1 = check_command and display_name, 2 = full details")
   return res
   }
   for (o in [Host, Service]) {
       for (s in get_objects(o).filter(s => s.last_check < get_time() - 2 * s.check_interval).filter(s => s.last_reachable==true) ) {
           var check_source = s.last_check_result.check_source
           if (check_source == null) {
               check_source = "PENDING"
           }
           if (!details.contains(check_source)) {
               details.set(check_source,{})
           }
           if (!details[check_source].contains(o.name)) {
               details[check_source].set(o.name,{})
           }
           if (verbosity==1){
               details[check_source][o.name].set(s.__name,{"check_command"=s.check_command,"display_name"=s.display_name})
           }
           if (verbosity==2){
               details[check_source][o.name].set(s.__name,s)
           }
           if(!res.contains (check_source)){
              res.set(check_source,{})
              res[check_source].set(o.name,1)
           } else {
               res[check_source][o.name] = res[check_source][o.name]+1
           }
       }
   }
   if (verbosity>0){
       res.set("Check details variable for further informations about late objects","use 'debug_late_checks___get_details.keys()' to get started")
       globals.debug_late_checks___get_details = details
   }
   return res
   }

Example of result :
debug_late_checks (0)

   {
           PENDING = {
                   Host = 1919.000000
                   Service = 649.000000
           }
           "sat-01.localdomain" = {
                   Host = 4.000000
           }
           "sat-02.localdomain" = {
                   Host = 36104.000000
                   Service = 13594.000000
           }
   }

With slightly increased verbosity, you’ll have detailled results exported in a variable.
debug_late_checks (1)

   {
           "Check details variable for further informations about late objects" = "use 'debug_late_checks___get_details.keys()' to get started"
           PENDING = {
                   Host = 1919.000000
                   Service = 649.000000
           }
           "sat-01.localdomain" = {
                   Host = 4.000000
           }
           "sat-02.localdomain" = {
                   Host = 36104.000000
                   Service = 13594.000000
           }
}

You can access this variable this and directly target a delayed poller :
debug_late_checks___get_details["sat-02.localdomain"]

   {
           Host = {
                   "host1" = {
                           check_command = "dummy"
                           display_name = "some name"
                   }
                   "host2" = {
                           check_command = "hostalive"
                           display_name = "an other name"
                   }
   [...]
           }
   		Service = {
   		           "host1!service1" = {
                           check_command = "dummy"
                           display_name = "my custom name"
                   }
                   "host2!service2" = {
                           check_command = "dummy"
                           display_name = "an other custom name"
                   }
   [...]				
   		}
   }

You could also just target the services if you would need :
debug_late_checks___get_details["sat-02.localdomain"]["Service"]

  	Service = {
  	           "host1!service1" = {
                       check_command = "dummy"
                       display_name = "my custom name"
               }
               "host2!service2" = {
                       check_command = "dummy"
                       display_name = "an other custom name"
               }

[…]
}

You can have the fully detailled objects by using debug_late_checks (2) followed by debug_late_checks___get_details, but i would strongly advise to filter out the result with your needs to be more comfortable since it can be highly verbose.

Where to install custom function
Depending on where you will need it, you can either declare it in /etc/icinga2/conf.d/functions.conf if you need it locally or /etc/icinga2/zones.d/myzone on your master if you want to pin it on satellites in a specific zone.

How to use custom function
You can manually execute it from icinga console, you can also reuse it in your own defined functions thought the interest seems limited to me.
You can access console using this documentation :
https://icinga.com/docs/icinga2/latest/doc/11-cli-commands/#cli-command-console

2 Likes

Tbh I don’t quite get what you are doing, as I seemingly haven’t work to those depths of Icinga 2 until now.
But anyhow, thank you for sharing!

I would put the link to the icinga2 console doc at the top, bc that “where do I do this” was confusing me at first.
But maybe this is obvious and my missing experience here is to blame :wink: