Cluster check failing with strange message

Icinga 2
1

Using the following configuration to check a cluster status:

template Service "cluster-check-tpl" {
  import "generic-service"
  check_command = "dummy"
  vars.dummy_state = {{
    var ok_count = 0
    var warning_count = 0
    var critical_count = 0
    var unknown_count = 0
    var service = macro("$service$")
    var hosts = macro("$hosts$")
    var num_hosts = len(hosts)
    var critthreshhold = len(hosts) * 0.5
    for (host in hosts) {
      if (get_service(host, service).state == 0 ) {
        ok_count += 1
      } else if (get_service(host, service).state == 1) {
        warning_count += 1
      } else if (get_service(host, service).state == 2) {
        critical_count += 1
      } else if (get_service(host, service).state == 3) {
        unknown_count += 1
      } else if (get_service(host, service).last_check_result == null) {
        unknown_count += 1
      }
    }
    if (ok_count == num_hosts) {                               // OK if all childs are OK
      return 0
    } else if (critical_count + warning_count > critthreshhold || unknown_count > critthreshhold) {   // CRITICAL if all childs are CRITICAL
      return 2
    } else if (unknown_count > 0) {                   // UNKNOWN if all childs are UNKNOWN
      return 1
    } else if ((warning_count + critical_count) > 0 ) {      // WARNING if a problem exists on any child
      return 1
    }
  }}
  vars.dummy_text = {{
    var service = macro("$service$")
    var hosts = macro("$hosts$")
    var num_hosts = len(hosts)
    var output = "Service " + service + " checked on " + num_hosts + " hosts:\n"
    for (host in hosts) {
      output += host + ": " + get_service(host, service).last_check_result.output + "\n"
    }
    return output
  }}
}

I get this (below) message when one of the 2 services is in an “unknown” state. It works well with “critical” and “ok” as it seems. I dont know what to make of it and if more info is needed i will happily provide it.

Exception occurred while checking 'ClusterChecks-ams1!haproxy-extern-ams1': Error: Closing $ not found in macro format string.
(0) icinga2: icinga::MacroProcessor::InternalResolveMacros(icinga::String const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::CheckResult> const&, icinga::String*, std::function<icinga::Value (icinga::Value const&)> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, int) (+0xf32) [0x93d972]
        (1) icinga2: icinga::MacroProcessor::InternalResolveMacros(icinga::String const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::CheckResult> const&, icinga::String*, std::function<icinga::Value (icinga::Value const&)> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, int) (+0xbc0) [0x93d600]
        (2) icinga2: icinga::MacroProcessor::ResolveMacros(icinga::Value const&, std::vector<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> >, std::allocator<std::pair<icinga::String, boost::intrusive_ptr<icinga::Object> > > > const&, boost::intrusive_ptr<icinga::CheckResult> const&, icinga::String*, std::function<icinga::Value (icinga::Value const&)> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool, int) (+0x182) [0x93df12]
        (3) icinga2: icinga::DummyCheckTask::ScriptFunc(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) (+0x335) [0xab0045]
        (4) icinga2: std::_Function_handler<icinga::Value (std::vector<icinga::Value, std::allocator<icinga::Value> > const&), std::enable_if<std::is_function<std::remove_pointer<void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool)>::type>::value&&(!std::is_same<void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool), icinga::Value (*)(std::vector<icinga::Value, std::allocator<icinga::Value> > const&)>::value), std::function<icinga::Value (std::vector<icinga::Value, std::allocator<icinga::Value> > const&)> >::type icinga::WrapFunction<void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool)>(void (*)(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool))::{lambda(std::vector<icinga::Value, std::allocator<icinga::Value> > const&)#1}>::_M_invoke(std::_Any_data const&, std::vector<icinga::Value, std::allocator<icinga::Value> > const&) (+0x1b4) [0x828524]
        (5) icinga2: icinga::Function::Invoke(std::vector<icinga::Value, std::allocator<icinga::Value> > const&) (+0x3f) [0xa07e4f]
        (6) icinga2: icinga::CheckCommand::Execute(boost::intrusive_ptr<icinga::Checkable> const&, boost::intrusive_ptr<icinga::CheckResult> const&, boost::intrusive_ptr<icinga::Dictionary> const&, bool) (+0x172) [0xa08002]
        (7) icinga2: icinga::Checkable::ExecuteCheck() (+0x298) [0xaaba08]
        (8) icinga2: icinga::CheckerComponent::ExecuteCheckHelper(boost::intrusive_ptr<icinga::Checkable> const&) (+0x32) [0xaac952]
        (9) icinga2: icinga::ThreadPool::WorkerThread::ThreadProc(icinga::ThreadPool::Queue&) (+0x77a) [0x8549ba]
        (10) libboost_thread-mt.so.1.53.0: <unknown function> (+0xd27a) [0x2b55cbdb127a]
        (11) libpthread.so.0: <unknown function> (+0x7dd5) [0x2b55cdcc5dd5]
        (12) libc.so.6: clone (+0x6d) [0x2b55cdfd7ead]


        (0) Resolving macros for string 'Service haproxy-http checked on 2 hosts:
server1: Check haproxy OK - checked proxies: haproxy-http
server2: Use of uninitialized value $haproxy in split at /usr/lib64/nagios/plugins/check_haproxy_stats.pl line 197.
Unable to retrieve haproxy stats at /usr/lib64/nagios/plugins/check_haproxy_stats.pl line 199.
'
        (1) Resolving macros for string '$dummy_text$'
        (2) Executing check for object 'ClusterChecks-ams1!haproxy-extern-ams1'
2

Hi,

very advanced example, interesting.

Look at the stack trace, dummy_text is being used in this scope. Your function calculates the output by fetching another service object and its output.

Coming back to the stack trace again, the whole text for dummy_text contains the following string:

'Service haproxy-http checked on 2 hosts:
server1: Check haproxy OK - checked proxies: haproxy-http
server2: Use of uninitialized value $haproxy in split at /usr/lib64/nagios/plugins/check_haproxy_stats.pl line 197.
Unable to retrieve haproxy stats at /usr/lib64/nagios/plugins/check_haproxy_stats.pl line 199.
'

Icinga will detect other runtime macros in that string, and try to recursively resolve those macros. Therefore each dollar sign must be escaped in there and cannot be used standalone.

The Perl warning contains such, $haproxy to be specific.

Therefore you’ll need to sanitize the output before assigning its value to a custom attribute/runtime macro resolver.

Look into the String type which object methods are available and escape or cut the dollar character.

You can use the debug console to test that, but only with --eval and no user input.

icinga2 console --eval '"Use of uninitialized value $haproxy".replace("$", "$$")'

That being said, you code needs something like this

return output.replace('$', '$$')

You probably want to strip newlines and format the output a bit better as well. Or even detect such warnings and change the cluster state too.

Cheers,
Michael

3

Thank you so much for the quick response. That really helped alot. And a great product it is.
kind regards /Carl