Memory check "check_mem" not working

Icinga Web Modules Icinga Director
1

What’s the “right” way to get a memory check running? When I first tried it said that check_mem.pl was missing even though all the other plugins were there. I then found that it’s not included in the repository so I got it from here https://github.com/justintime/nagios-plugins/tree/master/check_mem , plonked it in the right directory and it no longer complains that it’s not there.

Now I’m getting this

Plugin Output

check_mem.pl v1.0 - Nagios Plugin

usage: check_mem.pl -<f

If I run this on the box I have the agent on I get this:
[root@centos plugins]# ./check_mem.pl -<f
-bash: f: No such file or directory

So it’s looking for a file on the command line, any ideas what file a memory check would want?

I’m using a service template and it’s got the command ‘mem’ in it, I’m drowning again in the documentation, can someone throw me a lifebelt and a signpost to where I need to look?

I’m also puzzled as to why this doesn’t work out of the box, it seems a pretty important check to do.

2

External check are not provided and maintained by Icinga, only the objects declaration for configuration included in the icinga template library (ITL) is provided (and sometime fixed) by community in the default install.

In the ITL, the vast majority of the long written default nagios checks (check_mem.pl included) are declared, so you’ll need to install the nagios-plugins package corresponding to your system.

You may find some useful informations about the ITL here :
https://icinga.com/docs/icinga2/latest/doc/10-icinga-template-library/

3

The icinga documention refers to the mem command which references the check_mem.pl see here

The icinga documentation then points at a blog post that is 11 years old here

I’m on Ubuntu. The nagios-plugins package doesn’t include the check_mem.pl, it has to be loaded in seperately as per this old post

So I’ve done all that and it’s not working so clearly none of this is the “correct” way. I’m keen to follow the documented way but again I’m struggling to find what I need.

I’ve followed your link to the ITL and that seems to suggest that everything is already there " By default the generic templates are included in the icinga2.conf configuration file:". As I have the those appearing in Director I’m assuming that is the case.

So what am I missing here?

4

Nothing, you did good (unless i’m missing something too), The ITL provides configuration definitions, not the runnables check themselves, you then have no other choice than importing it in the way your prefer on the machines that needs to run it.

If you are interested in digging in the ITL files, they are defined in /usr/share/icinga2/include/.

5

By this I suppose you mean to place the check_mem.pl file in /usr/lib64/nagios/plugins/ on the agent which I’ve done because it was complaining that it wasn’t there.

My preference is for it to work but as the method has yet to be found that achieves this goal I’m not sure which preference or method to use.

6

More fiddling and searching the web results in a new error message:

Plugin Output

*** You must select to monitor either USED or FREE memory!

Any clue as to where I set this? I’ve looked in the service template and it’s not there, I did find the missing fields in there for warning & critical but nothing for used & free. I can’t see where it reads this information from.

7

Based on the itl doc, you need to set either vars.mem_used or vars.mem_free to true in your related service definition, for example :

apply Service "custom-mem" {
check_command = "mem"
vars.mem_used = true
 [...]
 }

https://icinga.com/docs/icinga2/latest/doc/10-icinga-template-library/#mem

If you are curious to see what the called check command object “mem” looks like, it should be defined in/usr/share/icinga2/include/plugins-contrib.d/operating-system.conf
By default, all these vars you need are set to false.

object CheckCommand "mem" {
        command = [ PluginContribDir + "/check_mem.pl" ]

        arguments = {
                "-u" = {
                        set_if = "$mem_used$"
                        description = "Check USED memory"
                }
                "-f" = {
                        set_if = "$mem_free$"
                        description = "Check FREE memory"
                }
                "-C" = {
                        set_if = "$mem_cache$"
                        description = "Count OS caches as FREE memory"
                }
                "-w" = {
                        value = "$mem_warning$"
                        description = "Percent free/used when to warn"
                }
                "-c" = {
                        value = "$mem_critical$"
                        description = "Percent free/used when critical"
                }
        }

        vars.mem_used = false
        vars.mem_free = false
        vars.mem_cache = false
}
1 Like
8

Yup, found that, did that for -w & -c.
That’s how I got the result *** You must select to monitor either USED or FREE memory!

My problem is I can’t see how or where in Director I set this flag. It has to be -f for free or -u for used. I can see how to pass values for -w & -c but I can’t see how to pass a different switch based on some other parameter I’ve yet to discover.

When you say service definition, do you mean the template or an existing service or a new one or a clone or something else?

9

I mean a new service object you create to generate the services using mem CheckCommand.

For the Director part, i couldn’t answer with 100% accuracy, i dont use it, but based on the doc, setting a custom property as described here could solve your problem.
https://icinga.com/docs/director/latest/doc/15-Service-apply-for-example/

10

Nope, sorry that didn’t help at all. check_mem.pl needs either -f or -u you can’t pass both. Everywhere I’ve been in Director I can assign values to pass to parameters but I can’t pass a different set of parameters based on some other parameter even if I could find where to set this other parameter.

Totally lost and confused here.

11

Where I am now: Until I figure out how to programmatically pass variable parameters, I’ve created a command called my-free-mem-check whcih has the command PluginDir + check_mem.pl -f

This produces:
object CheckCommand “my-free-mem-check” {
import “plugin-check-command”
command = [ PluginDir + “check_mem.pl”, “-f” ]
arguments += {
“-c” = {
description = “Critical value”
value = “$criticalmemvalue$”
}
“-w” = {
description = “Warning value”
value = “$warningmemoryvalue$”
}
}
}

This deploys fine and does not throw any validation errors however when it runs the result of this is:

execvpe(/usr/lib64/nagios/pluginscheck_mem.pl) failed: No such file or directory

It’s decided not to put a slash in before the plugin name so it fails. If I add the slash I just get back to where I started:
check_mem.pl v1.0 - Nagios Plugin

usage: check_mem.pl -<f

12

Hi.

I also can’t say much about the director, but here - as an example -
a complete way to create a service and assign it dynamically to specific hosts.

It is supposed to also help other users, so just skip the parts you already know.

A service typically requires 3 parts to get it working:

  1. A plugin-file,
    in this case check_mem.pl.
    Under Centos, typically placed unter /usr/lib64/nagios/plugins/.
    (On Debian based systems under /usr/lib/nagios/plugins).
    Staying at this example: /usr/lib64/nagios/plugins/check_mem.pl
    This can be also executed at the commandline for first tests.
    It is just “something” which can be executed and returns requested information and an exit status.
    0 = OK
    1 = warning
    2 = critical
    3 = unknown
    Based on this status, Icinga2 will interpret the service.

  2. Now we require a CheckCommand,
    which uses the plugin from step 1.
    Fortunately, in this case this has already been done (part of the Icinga Template Library - ITL). The CheckCommand defines how to use the plugin from step 1 (arguments, values).
    In this case:

CheckCommand definition 
# Taken from: /usr/share/icinga2/include/plugins-contrib.d/operating-system.conf
object CheckCommand "mem" {
        command = [ PluginContribDir + "/check_mem.pl" ]

        arguments = {
                "-u" = {
                        set_if = "$mem_used$"
                        description = "Check USED memory"
                }
                "-f" = {
                        set_if = "$mem_free$"
                        description = "Check FREE memory"
                }
                "-C" = {
                        set_if = "$mem_cache$"
                        description = "Count OS caches as FREE memory"
                }
                "-w" = {
                        value = "$mem_warning$"
                        description = "Percent free/used when to warn"
                }
                "-c" = {
                        value = "$mem_critical$"
                        description = "Percent free/used when critical"
                }
        }

        vars.mem_used = false
        vars.mem_free = false
        vars.mem_cache = false
}

So there already is a CheckCommand names “mem” which we can use

  1. Now we need a service, which uses the CheckCommand (“mem”).
    We need to take care, that the check is executed at the agent, not at the master (or satellite).
    So we need to set the “command_endpoint”.
    This also means, the the plugin (in this case: check_mem.pl) has to be placed in the plugin-directory of the target system (where the check is executed) and has to be executable by the user “icinga” (on RedHat based systems) or “nagios” (on Debian based systems)
Example of a Service Defition with apply-rule 
apply Service "memory" {
  import "generic-service"

  check_interval = 1m
  // Using the CheckCommand from step 2
  check_command = "mem"
  // Execute this at the agent-machine, not elsewhere
  command_endpoint = host.name

  vars.mem_used = true
  vars.mem_cache = true
  vars.mem_warning = 80
  vars.mem_critical = 90

  assign where host.vars.os == "Linux"
  ignore where host.vars.noagent
}

What we got until now
So what we got until here:

  • A plugin
  • a CheckCommand definition (“mem”),
  • a Service using the CheckCommand definition (service name: “memory”)
    This service will be applied to all hosts which have set the (host-)variable vars.os = “Linux”,
    hosts without an agent will be ignored (we want the check to be executed by agent on the target system, so an agent is required).
  1. Example output - taken from debug-log
OK - 17.5% (356896 kB) used.|TOTAL=2041320KB;;;; USED=356896KB;1633056;1837188;; FREE=1684424KB;;;; CACHES=1572432KB;;;;

Greetings.

13

If I understand correctly you want to know how you can choose if you want to check the free space or used space based on the variable set in the Director.

Create two datafields, named mem_used and mem_free, set them to type boolean and then assign them to your command.
image ->

image
image622×529 12.8 KB

Then you can create the service template which has a true/false option for those two variables.

A working command line would look like this:
'/usr/lib64/nagios/plugins/check_mem.pl' '-c' '100' '-u' '-w' '95'

2 Likes
14

I thought I’d done that

image
image648×616 12.4 KB

Still doesn’t work:
check_mem.pl v1.0 - Nagios Plugin
usage:
check_mem.pl -<f

Where do I tell it to add -f if the variable it true/false? I can’t see this in your example. Does anyone have this working?

So frustrating.

15

You have to create to datafields inside the Director prior to using them in the command.
image
https://your-monitoring.host/icingaweb2/director#!/icingaweb2/director/data/fields

There you can create a field like shown in my first screenshot which then can be added to the command/service template fields.

The set_if (true/false) variables are not detected by the Director when running the kickstart to import the ITL, so you don’t get them offered by the dropdown automatically like the “normal” variables.
There is already a enhancement request for this, but it hastn’t been updated in quite some time:

1 Like
16

Yup, I’d done that step
image

I still don’t understand how from a data field/variable/whatever you pass -f or -u to the plugin.

17

They appear in the template
image

Which produces this
image

Which results in this on the agent:

object CheckCommand “my-mem-check” {
import “plugin-check-command”
command = [ PluginDir + “/check_mem.pl” ]
arguments += {
“-c” = {
description = “Critical value”
value = “$criticalmemvalue$”
}
“-w” = {
description = “Warning value”
value = “$warningmemoryvalue$”
}
}
}

Still no -f or -u commands, still no clue how these get set based on the data fields.

18

You have two options as I see it:
Either switch the check coammdn of your service template svc_tmpl_check_free_mem to the mem command provided by the ITL, or
configure your my-mem-check command to use the extra arguments (you could copy&past from the ITL command)

1 Like
19

Well I’m getting somewhere, it now does this LOL!
Running command ‘/usr/lib64/nagios/plugins/check_mem.pl’ ‘-c’ ‘5’ ‘-f’ ‘true’ ‘-u’ ‘false’ ‘-w’ ‘10’

20

image
image612×612 76.3 KB