DSL: Access the last check output of a host/service retrieved by get_host/get_service?

Icinga 2
1

Hi Icinga team.
I’m looking on a solution for a “scenario” check which actually refers to other checks defined by a custom check attribute. Some of our monitoring probe providers put a “meaning” in the check output and we want to add a capability to define a match() condition so that the calculated scenario can take this intro consideration. Not easy to explain, so let me show some code:

oPtrServiceCurrent = get_host(cSAddressHost)     
if (oPtrServiceCurrent) {
                 /*if a STATE condition is defined in aSetCondMatrixEntry[3], then we compare against oPtrServiceCurrent.state*/
                 /* create closure function that has access to aSetCondMatrixEntry */
                 fFilterFunction = function(x) use(aSetCondMatrixEntry) { x == aSetCondMatrixEntry[3] }
                 if (aStateWord.filter(fFilterFunction).len() > 0) {
                        lIsCondMet=(aStateWord[oPtrServiceCurrent.state] == aSetCondMatrixEntry[3] && oPtrServiceCurrent.last_state_change <  get_time() - number(aSetCondMatrixEntry[4])*60)
                 } else {
                        /*we check match against plugin oPtrServiceCurrent.output*/
                        lIsCondMet=( match(aSetCondMatrixEntry[3],oPtrServiceCurrent.output) && oPtrServiceCurrent.last_state_change <  get_time() - number(aSetCondMatrixEntry[4])*60)
                 }

The code works well for state-compare but looks like as if oPtrServiceCurrent.output is not a valid access:

Exception occurred while checking 'TEST!netm-scenario-SCO.xms_logscan2.check_monitoring': Error: Invalid field access (for value of type 'Service'): 'output'
Location: in /etc/icinga2/conf.d/icinga/servicegroups/common/services.conf: 260:46-260:70
/etc/icinga2/conf.d/icinga/servicegroups/common/services.conf(258):   } else {
/etc/icinga2/conf.d/icinga/servicegroups/common/services.conf(259):    /*we check match against plugin oPtrServiceCurrent.output*/
/etc/icinga2/conf.d/icinga/servicegroups/common/services.conf(260):    lIsCondMet=( match(aSetCondMatrixEntry[3],​oPtrServiceCurrent.output) && oPtrServiceCurrent.last_state_change <  get_time() - number(aSetCondMatrixEntry[4])*60)
                                                                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^

For sure it would be better to have independent checks per each scenario condition but we cannot get this from external check providers in acceptable time/effort so any Icinga DSL solution would be the saviour :slight_smile:
Any idea ?

Best regards
Michael

2

Hi,

where is this code being executed, I’d assume that’s inside a lambda function as custom attribute assignment? If that’s directly at compilation time during startup, this code won’t work reliably.

Data from check results is stored in the last_check_result attribute, similar to what you can see being used in the troubleshooting docs. Test that with the debug console.

Cheers,
Michael

3

Hi @dnsmichi.
Thanks a lot for the helpful hint on last_check_result attribute.
Actually the below code works fine:

  check_command = "dummy"
  vars.dummy_state = {{
....
 /* if a STATE condition is defined in aSetCondMatrixEntry[CME_COL_STATE], then we compare against oPtrServiceCurrent.state*/
         /* create closure function that has access to aSetCondMatrixEntry */
         fFunction = function(x) use(cCurState = aSetCondMatrixEntry[CME_COL_STATE]) { x == cCurState }
         if (aStateWord.filter(fFunction).len() > 0) {
                lIsCondMet=(aStateWord[oPtrServiceCurrent.state] == aSetCondMatrixEntry[CME_COL_STATE] && oPtrServiceCurrent.last_state_change <  get_time() - number(aSetCondMatrixEntry[CME_COL_SINCE])*60)
         } else {
                /*we check match against plugin oPtrServiceCurrent.last_check_result.output*/
                lIsCondMet=( match(aSetCondMatrixEntry[CME_COL_STATE],oPtrServiceCurrent.last_check_result.output) && oPtrServiceCurrent.last_state_change <  get_time() - number(aSetCondMatrixEntry[CME_COL_SINCE])*60)
     }
     ...

The code is evaluated during a check evaluation and hence works pretty well as the plugin output is checked properly.

Thx a lot!
Best regards
Michael

4

Ok, that looks pretty advanced and you know what you’re doing.

Small hint: I’d use a sanity check against last_check_result, the first pending check will render this null and the dot indexer will fail.

  if (oPtrServiceCurrent.last_check_result) {
     lIsCondMet=( match(aSetCondMatrixEntry[CME_COL_STATE],oPtrServiceCurrent.last_check_result.output) ....
  }

Cheers,
Michael

5

Hi Michael.
Thx a lot for the hint on sanitization and insight on the dot-indexer operation, still learning…
I’ve added your proposal slightly modified.
I have another (maybe tricky) question but for the sake of thread separation i’ll create a new issue.

Thanks again for the prompt support, much appreciated.
Best regards
Michael