Hello,
I’m not too sure this post belongs here, correct me if i’m wrong !
So, i’d like to share you a trick and a custom function that can come handy in debugging or developping user functions.
Dump and explore Icinga internals
globals
Typing this in the console, or using it directly in a function will return you a big array of the whole execution context of user made function declared within globals. Since at first glance it looks unreadable, you can peacefully explore it using globals.keys()
You should have a result like this :
[ "ActiveStages", "Icinga", "Internal", "ManubulonPluginDir", "MaxConcurrentChecks", "NodeName", "NscpPath", "PluginContribDir", "PluginDir", "ReloadTimeout", "StatsFunctions", "System", "TicketSalt", "Types", "ZoneName" ]
So, it returns you static constants, whether they are defined in the constants.conf file or somewhere else, internal objects, other declared user functions and internal DSL.
For example, you can list undocumented but implemented functions for a given type of object, which can come very handy to develop specific user functions sometime.
Alongside Service object definition, you’ll find those functions that may come handy :
globals["Types"]["Service"]
modify_attribute = { arguments = [ "attr", "value" ] deprecated = false name = "ConfigObject#modify_attribute" side_effect_free = false type = "Function" } restore_attribute = { arguments = [ "attr", "value" ] deprecated = false name = "ConfigObject#restore_attribute" side_effect_free = false type = "Function" }
Other example :
globals["Icinga"]
You can find icinga internal constants, which may be useful to know for debugging/understanding some parts.
{ Acknowledgement = "Acknowledgement" Critical = "Critical" Custom = "Custom" DbCatAcknowledgement = 4.000000 DbCatCheck = 256.000000 DbCatComment = 8.000000 DbCatConfig = 1.000000 DbCatDowntime = 16.000000 DbCatEventHandler = 32.000000 DbCatEverything = -1.000000 DbCatExternalCommand = 64.000000 DbCatFlapping = 128.000000 DbCatLog = 512.000000 DbCatNotification = 1024.000000 DbCatProgramStatus = 2048.000000 DbCatRetention = 4096.000000 DbCatState = 2.000000 DbCatStateHistory = 8192.000000 [...]
Troubleshoot late checks in cluster
I got inspired by the function already present in the documentation as example and enhanced it to output more informations to tell precisely what Service or Host check is late and on which satellite based on the last check source known, Here is the function :
globals.debug_late_checks = function(verbosity) { var res = {} var details = {} if (typeof(verbosity)["name"] != "Number" || verbosity < 0 || verbosity > 2 ) { res.set("set verbosity properly","0 = No details about devices, 1 = check_command and display_name, 2 = full details") return res } for (o in [Host, Service]) { for (s in get_objects(o).filter(s => s.last_check < get_time() - 2 * s.check_interval).filter(s => s.last_reachable==true) ) { var check_source = s.last_check_result.check_source if (check_source == null) { check_source = "PENDING" } if (!details.contains(check_source)) { details.set(check_source,{}) } if (!details[check_source].contains(o.name)) { details[check_source].set(o.name,{}) } if (verbosity==1){ details[check_source][o.name].set(s.__name,{"check_command"=s.check_command,"display_name"=s.display_name}) } if (verbosity==2){ details[check_source][o.name].set(s.__name,s) } if(!res.contains (check_source)){ res.set(check_source,{}) res[check_source].set(o.name,1) } else { res[check_source][o.name] = res[check_source][o.name]+1 } } } if (verbosity>0){ res.set("Check details variable for further informations about late objects","use 'debug_late_checks___get_details.keys()' to get started") globals.debug_late_checks___get_details = details } return res }
Example of result :
debug_late_checks (0)
{ PENDING = { Host = 1919.000000 Service = 649.000000 } "sat-01.localdomain" = { Host = 4.000000 } "sat-02.localdomain" = { Host = 36104.000000 Service = 13594.000000 } }
With slightly increased verbosity, you’ll have detailled results exported in a variable.
debug_late_checks (1)
{ "Check details variable for further informations about late objects" = "use 'debug_late_checks___get_details.keys()' to get started" PENDING = { Host = 1919.000000 Service = 649.000000 } "sat-01.localdomain" = { Host = 4.000000 } "sat-02.localdomain" = { Host = 36104.000000 Service = 13594.000000 } }
You can access this variable this and directly target a delayed poller :
debug_late_checks___get_details["sat-02.localdomain"]
{ Host = { "host1" = { check_command = "dummy" display_name = "some name" } "host2" = { check_command = "hostalive" display_name = "an other name" } [...] } Service = { "host1!service1" = { check_command = "dummy" display_name = "my custom name" } "host2!service2" = { check_command = "dummy" display_name = "an other custom name" } [...] } }
You could also just target the services if you would need :
debug_late_checks___get_details["sat-02.localdomain"]["Service"]
Service = { "host1!service1" = { check_command = "dummy" display_name = "my custom name" } "host2!service2" = { check_command = "dummy" display_name = "an other custom name" }[…]
}
You can have the fully detailled objects by using debug_late_checks (2) followed by debug_late_checks___get_details, but i would strongly advise to filter out the result with your needs to be more comfortable since it can be highly verbose.
Where to install custom function
Depending on where you will need it, you can either declare it in /etc/icinga2/conf.d/functions.conf if you need it locally or /etc/icinga2/zones.d/myzone on your master if you want to pin it on satellites in a specific zone.
How to use custom function
You can manually execute it from icinga console, you can also reuse it in your own defined functions thought the interest seems limited to me.
You can access console using this documentation :
https://icinga.com/docs/icinga2/latest/doc/11-cli-commands/#cli-command-console
