How to debug a configuration issue?

ciberandy · May 7, 2020, 10:29pm

Any hints are welcome for debugging the following problem:

I’ve got a custom checkcommand defined like this:

object CheckCommand "uptime1" {
        command = [ PluginDir + "/check_uptime" ]

        arguments = {
        "--warning" = {
            value = "$uptime1_warning$"
            required = true
        }
        "--critical" = {
            value = "$uptime1_critical$"
            required = true
        }
        "--perfparse" = {
            set_if = "$uptime1_perfdata$"
            description = "Add performance data"
        }
    }
    // some defaults
    vars.uptime1_perfdata = true
    vars.uptime1_warning = "15"
    vars.uptime1_critical = "5"
}

Now, when using this CheckCommand in an apply-rule like e.g.
then the check fails.

apply Service "uptime" {
  check_command = "uptime1"
  import "ssh-service"

  vars += host.vars.uptime1

  assign where (host.address || host.address6) && host.vars.os == "Linux"
}

Icinga complains:
Error: Non-optional macro ‘uptime1_critical’ used in argument ‘–critical’ is missing.

(The template “ssh-service” is coming from here: https://blog.netways.de/2016/03/21/check_by_ssh-mit-icinga-2/)

Apparently the variable with the default parameter is not being passed through.
What’s even stranger is that the same setup works for other checkcommands.
I have no clue what’s different here.

How do I go about debugging this problem?
Thanks for any hints!
Andy

aclark6996 · May 8, 2020, 2:13am

Hello Andy,
You are receiving this error because the check command argument “required” field is set to “true” for the critical and warning argument. Set the required field to false to get past this error.

Read more details here.

You have a default value set in the check command object for the critical value, so not need to set the required field to true.

ciberandy · May 8, 2020, 11:40am

True, but - no matter whether required set to true or false - the command option is not being used.
Somehow Icinga doesn’t see the default value which I set.

aclark6996 · May 8, 2020, 2:32pm

If you change your code to this does it work?

object CheckCommand "uptime1" {
        command = [ PluginDir + "/check_uptime" ]

        arguments = {
        "--warning" = {
            value = "$uptime1_warning$"
            required = true
        }
        "--critical" = {
            value = "$uptime1_critical$"
            required = true
        }
        "--perfparse" = {
            set_if = "$uptime1_perfdata$"
            description = "Add performance data"
        }
    }
    // some defaults
    vars.uptime1_perfdata = true
}

apply Service "uptime" {
  check_command = "uptime1"
  import "ssh-service"

  vars.uptime1_warning = "15"
  vars.uptime1_critical = "5"

  assign where (host.address || host.address6) && host.vars.os == "Linux"
}

Regards
Alex

ciberandy · May 8, 2020, 3:12pm

Yes, but the option --perfparse isn’t added to the command line.

ciberandy · May 12, 2020, 10:58pm

If I remove this line:

import "ssh-service"

then the variables get passed through as they should. So it must have to do with the by_ssh-template.

I thought this line takes care of this in the by_ssh-template:
vars.by_ssh_arguments = {{ get_check_command(service.vars.original_check_command).arguments }}

Am I wrong?

ciberandy · May 24, 2020, 10:58pm

I’m still struggling with this.

Anyone else has a hint for me on how to debug that?

anon66228339 · May 25, 2020, 4:03am

Hello,

Maybe you post also the template. Makes it easier to help you.

Regards,
Carsten

ciberandy · May 26, 2020, 4:38pm

Oh, sure.

template Service "ssh-service" {
  import "generic-service"

  // "save" original command name, and replace it
  vars.original_check_command = check_command
  check_command = "by_ssh"
  vars.by_ssh_port = "5522"
  vars.by_ssh_quiet = 1
  vars.by_ssh_timeout = 90

  // these get evaluated at runtime
  vars.by_ssh_command = {{ get_check_command(service.vars.original_check_command).command }}
  vars.by_ssh_arguments = {{ get_check_command(service.vars.original_check_command).arguments }}

  // for integration with graphite
  // https://github.com/Icinga/icingaweb2-module-graphite/blob/master/doc/03-Configuration.md
  // To make the respective graphs work as expected
  // you have to tell the monitored object's "actual" check command
  // by setting its custom variable "check_command"
  vars.check_command = vars.original_check_command
}

anon66228339 · May 26, 2020, 5:48pm

Hello,

i changed the check command arguments = to arguments += and added vars.uptime1_perfdata = true to your service. The trick only handles variables that are defined in the service and not if they are defined in the command.

Final command:

object CheckCommand "uptime1" {
        command = [ PluginDir + "/check_uptime" ]

        arguments += {
        "--warning" = {
            value = "$uptime1_warning$"
            required = true
        }
        "--critical" = {
            value = "$uptime1_critical$"
            required = true
        }
        "--perfparse" = {
            set_if = "$uptime1_perfdata$"
            description = "Add performance data"
        }
    }
    // some defaults
    vars.uptime1_perfdata = true
}

and the service:

apply Service "uptime" {
  check_command = "uptime1"
  import "ssh-service"

  vars.uptime1_warning = "15"
  vars.uptime1_critical = "5"
  vars.uptime1_perfdata = true

  assign where (host.address || host.address6) && host.vars.os == "Linux"
}

Output (i did not install ssh keys etc.)

To debuf such things in the future you can use the log function to write it to logfile or console
Regards,
Carsten

ciberandy · May 26, 2020, 9:36pm

Interesting. What is the reason for the + ?

Yes, but I want to avoid having to set all default values again.
If you don’t add that line, does the service pick up the default value from the command definition and thus add the “–perfparse” parameter?

Oh, too bad. But in other commands it works.
I have the feeling it only works if I use it within a “for” loop.

Here’s an example:

object CheckCommand "lm-sensors" {
    command = [ PluginLocalDir + "/check_lm_sensors" ]

    arguments = {
        "--nodrives" = {
            set_if = "$lm_sensors_nodrives$"
        }
        "--sanitize" = {
            set_if = "$lm_sensors_sanitize$"
        }
        "--high" = {
            value = "$lm_sensors_high$"
            description = "specifies a check for a sensor value which is too high"
        }
        "--low" = {
            value = "$lm_sensors_low$"
            description = "specifies a check for a sensor value which is too low"
        }
        "--range" = {
            value = "$lm_sensors_range$"
            description = "specifies a check for a sensor value which should stay in a given range"
        }
    }

    vars.lm_sensors_nodrives = true
    vars.lm_sensors_sanitize = true
    vars.lm_sensors_high = "Core0=60,90"
}

And the service:

apply Service for (lmsensor => config in host.vars.lmsensors) {
  check_command = "lm-sensors"
  import "ssh-service"
        
  vars += config
}

Why does it pick up the default values in this case?

Thanks,
Andy

ciberandy · May 26, 2020, 9:41pm

Nice! So I would add sth like
log(LogCritical, "Console", vars.uptime1_perfdata)
to my command (or service?) definition?

How exactly did you get this output of the plugin?
Thank you!
Andy

anon66228339 · May 27, 2020, 5:55am

Hello @ciberandy,

the += adds/merges all variables under the variable arguments or vars and does not complete overwrite the arguments or vars. For example if you have a command that imports another command and adds only 1 argument. If you write arguments = ... all the arguments from the import are gone and only the one you defined in the new command is there. If you use arguments += the variables from arguments are merged.

Why it works in a foor loop you have to ask the developers, maybe @theFeu can get the information from one.

Regards,
Carsten

ciberandy · May 27, 2020, 2:25pm

Do you have a hint for me where I can find usage examples of debugging with “log”?
How exactly did you get this explicit output of the plugin?

Thanks
Andy

Al2Klimov · June 8, 2020, 2:19pm

https://icinga.com/docs/icinga2/latest/doc/18-library-reference/#log

ciberandy · June 8, 2020, 3:25pm

Thanks, that’s just the syntax and an example for the console.

I meant to ask “usage examples in the service/command configuration”.
I tried to put log() calls there but don’t see any output anywhere?

Thanks,
Andy

Al2Klimov · June 8, 2020, 3:37pm

You should see it in your standard icinga2.log.

ciberandy · July 2, 2020, 6:00pm

Carsten wrote above: “Why it works in a foor loop you have to ask the developers”
I still did not find a solution to that. Could you point me in the right direction please?

Thanks, Andy

PS: And I still haven’t found out how to use log() in the service/command configuration. Help!

Al2Klimov · July 3, 2020, 10:21am

        "--perfparse" = {
            set_if = () => { var m = macro("$uptime1_perfdata$"); log(m); return m }
            description = "Add performance data"
        }

ciberandy · July 10, 2020, 9:40pm

Thanks. Wow, that’s pretty complicated.
No wonder, I didn’t find any usage examples. Thank you!

Okay, now I see "[2020-07-11 00:12:24 +0200] information/config: " in my icinga2.log
This means that the variable “uptime1_perfdata” is actually “false”
although the CheckCommand defines a default value (see above).

I’m starting to believe I’ve found a bug. What do you think?

Thanks,
Andy

How to debug a configuration issue?

I’ve got a custom checkcommand defined like this:

Now, when using this CheckCommand in an apply-rule like e.g. then the check fails.

Now, when using this CheckCommand in an apply-rule like e.g.
then the check fails.