How to write a bash script wrapper (newbie script for ITL check_yum plugin)

Author: @watermelon
Original: https://monitoring-portal.org/t/how-to-write-a-bash-script-wrapper-newbie-script-for-itl-check-yum-plugin/3116


How to write a bash script wrapper

Introduction

Many times on these forums I’ve seen people asking how to handle output from default plugins from the ITL such as the check_snmp plugin. In fact, I was one of those people! I thought it would be useful for people to have some sort of a guideline to writing script wrappers for these types of plugins. Script wrappers, for those who don’t know, are simply a way of handling a plugin’s output, whether that is performance data, exit states (warning, critical, etc.), or the text that it returns. For example, the check_snmp plugin by itself is good in that it is very general, but the default output is not necessarily the most information dense and is begging to be wrapped by another script that will most definitely make it more useful.

This tutorial will cover a script that I made just recently for the check_yum plugin provided by the ITL. I didn’t like how you couldn’t really change the exit state of the plugin so easily and I also wanted to output a list of the packages to be updated rather than just numbers. So, I made the output change from this:

To this:

All while changing the way the plugin handles states as well (I wanted warning instead of unknown for ‘non-security’ updates). Of course, since you’re about to try it for yourself, you can change this behavior to fit your needs.

Author: @watermelon

Revision: v1.0

Tested with:

  • Icinga 2 v2.8.x
  • Icinga Web 2 v2.5.x
  • check_yum v1.1.0

Requirements

  • The default check_yum plugin that comes with the ITL

Configuration

  • Start it off by creating the script file. I like to make it in the default plugin directory so that everything is consistent when calling scripts/plugins
vim /usr/lib64/nagios/plugins/check_yum_wrapper.sh
  • Starting any bash script off, you’ll need the following at the top:
#!/bin/bash
  • This is to indicate that the script is a bash script so that your machine knows how to execute the code.

  • You also will want to define an exit code (or return code) variable to handle how your script will exit. Remember: 0 is OK, 1 is WARNING, 2 is CRITICAL, and 3 is UNKNOWN

EXITCODE="0"
  • At the beginning of my scripts, I like to handle command line input. To do so, you can use the following routine:
while getopts ":t:s:h:" option; do
  case "${option}" in
    t) timeout=${OPTARG};;
    s) security=${OPTARG};;
    h) usage;;
    *) usage;;
  esac
done
  • This snippet is pretty self explanatory, but notice the wildcard (*) calls a method and so does the -h flag. The wildcard means that the loop will default to that case when there is some unknown flag supplied to the command. The method, usage, might display a helpful message similar to when you invoke the --help flag for familiar commands for example yum --help. Therefore, the usage method can be as simple as this:
usage() {
  echo -e "Usage: $0 \n -t <timeout> \tTimeout for plugin execution (OPTIONAL, default 60s)\n -s <security> \tIndicate whether or not to output only security updates (OPTIONAL, default 'off')\n -h <help> \tDisplays this help message\n" 1>&2;
  exit 3;
}
  • The -e flag for echo makes it so that it can interpret backslashes in the string of text to be echo’d. This usage method will produce this output:
[icinga@icinga2master ~]$ ./check_yum_updates.sh -h
Usage: ./test2.sh 
 -t <timeout> 	Timeout for plugin execution (OPTIONAL, default 60s)
 -s <security> 	Indicate whether or not to output only security updates (OPTIONAL, default 'off')
 -h <help> 	Displays this help message
  • After the user input section, it’s time to decide what to do based on what was supplied. Before that though, it’s good practice to reset the position of the command line arguments using the following one-liner:
shift "$((OPTIND-1))"
  • This makes it so that if you need to access the arguments from your initial command for something else in your script (using $1, $0, etc.), it will point to the corresponding argument rather than being offset (more clarification).
  • Now we want to ensure that none of the variables from the user input section are left with no value if the user decides to leave the argument blank. To check if a variable is empty, you can use the following statement:
if [ -z "${timeout}" ]; then
  timeout=60
fi
  • This makes it so that the default timeout of the plugin is 60s.

  • Try something similar for the security flag! See if your solution is similar to mine at the end of this tutorial when I post the entire script. Hint: you probably will want to use some global variable that you can change to fit into the yum check-update command that will be used later.

  • Now we can call the original check_yum plugin to get the number of pending updates for the overall output:

NUMUPDATES="$(/usr/lib64/nagios/plugins/check_yum -t ${timeout} 2>/dev/null | awk '{print $1 ";" $5}')"
  • Notice the various additional flags and notation for the command. Also notice that we are using the timeout variable from before that was obtained from the user. If you didn’t guess already, awk is grabbing the 1st and 5th fields of the output of the check_yum plugin and putting a semicolon inbetween these numbers. These happen to be the fields with the important information, with the 1st field being the number of security updates and the 5th field being the number of non-security updates. Before we do anything, we will want to handle other potential cases. In the case that the plugin times out, the check_yum plugin will output this:
YUM nagios plugin has self terminated after exceeding the timeout (60 seconds)
  • As noted before in the previous command, the 1st and 5th fields are grabbed, so in this string, that would turn out to be YUM and self. It’s slightly inconventional, but it does work and make sense if you know what it actually means. Writing the following check will make sure that you can handle the case that the plugin times out:
if [[ ${NUMUPDATES} == "YUM;self" ]]; then
  echo "Error: Timeout exceeded (${timeout} seconds)"
  exit 3
fi
  • Hint: there are other cases to look for! See if you can figure them out and create similar validation checks.

  • Now we will want to separate those fields from the initial command to get the number of updates (stored in the NUMUPDATES variable). To do so, you can use the cut command:

NUMSECURITYUPDATES="$(echo $NUMUPDATES |cut -d';' -f1)"
NUMOTHERUPDATES="$(echo $NUMUPDATES |cut -d';' -f2)"
  • You can see clearly that you’re simply taking the NUMUPDATES variable and chopping it up based on that semicolon delimiter that was put in initially.

  • Now we can get into handling what the script will actually output. Based on your preferences, you might want to change this up. For me, I put in the following state handling routine:

if [[ ${NUMSECURITYUPDATES} > "0" ]]; then
  EXITCODE="2"
else
  # if there are non-security updates, then exit with warning
  if [[ ${NUMOTHERUPDATES} > "0" ]]; then
    EXITCODE="1"
  else
    EXITCODE="0"
  fi
fi
  • This should be relatively simple to understand; if there are security updates, then exit as CRITICAL (regardless if there are non-security updates), and if there are non-security updates, then exit with WARNING, otherwise exit OK.

  • Then, we’ll want to handle the output of the script. You can do whatever you’d like with this, but I just followed the same format as the original check_yum plugin:

echo "${NUMSECURITYUPDATES} Security Updates Available, ${NUMOTHERUPDATES} Non-Security Updates Available | 'security_updates'=${NUMSECURITYUPDATES};;1;; 'nonsecurity_updates'=${NUMOTHERUPDATES};;1;;"
  • Notice the way that you can ship the output with performance data as well (more information here).

  • Almost there! We want to now add on to the regular functionality of the check_yum plugin and output the package names that are to be updated. I do this using the mapfile command:

mapfile -t updateArr < <( yum check-update --quiet ${CMD})
  • Basically, this puts the command output (of yum check-update) into an array (updateArr). Notice the reference to a CMD variable, which is another hint as to what you can do with the security argument from the beginning of the script.

  • You’ll now want to access the data in this array, so iterate through it with this loop, outputting its values at the same time:

for index in ${!updateArr[*]}
do
  echo "${updateArr[${index}]}"
done
  • Arrays in bash can be pretty funky, you’ll probably want to read up about them if you want to learn more about how they work.

  • Woohoo!! You’re done! Now you’ll just need to exit the script using the EXITCODE variable:

exit ${EXITCODE}

Conclusion

So what do you think? Is it cool knowing that you are so confined to the default plugins that the ITL and 3rd parties provide? You can make your own check scripts now! All you have to do now is to call the script from Icinga.

  • First, make a CheckCommand object:
object CheckCommand "check_yum_updates" {
  import "plugin-check-command"
  command = [ PluginDir + "/check_yum_updates.sh" ]

  arguments = {
  "-t" = 120
  "-s" = "off"
  }
}
  • Then, make a Service to use that CheckCommand:
apply Service "check_yum_updates" {
  import "generic-service"

  check_command = "check_yum_updates"
  display_name = "YUM Updates"

  assign where host.vars.os == "Linux"
  • Now you’re ready to roll.

I hope you enjoyed this and hopefully it has all the information that you need to get started with script wrapping in Icinga2. Happy scripting!

Sincerely,
watermelon

Full script

#!/bin/bash
# Purpose: Script wrapper for the ITL check_yum plugin to add additional output (names of the packages to be updated) as well give the ability to handle states as you please

# Note: This is also meant as kind of a "beginner's tutorial" for script wrappers with Icinga. Feel free to ask me (@Watermelon) for any tips or questions you might have after you read through this script's functions. Enjoy!

# Exit codes: OK(0), WARNING(1), CRITICAL(2), UNKNOWN(3)

# function to help the user out with the usage of this script
usage() {
  echo -e "Usage: $0 \n -t <timeout> \tTimeout for plugin execution (OPTIONAL, default 60s)\n -s <security> \tIndicate whether or not to output only security updates (OPTIONAL, default 'off')\n -h <help> \tDisplays this help message\n" 1>&2;
  exit 3;
}

# check for command line arguments
while getopts ":t:s:h:" option; do
  case "${option}" in
    t) timeout=${OPTARG};;
    s) security=${OPTARG};;
    h) usage;;
    *) usage;;
  esac
done

# Resets position of first non-option argument
shift "$((OPTIND-1))"

# ensures the timeout is set as default 60 seconds if not supplied via command
if [ -z "${timeout}" ]; then
  timeout=60
fi

CMD=""

# handle "s" flag
if [ -z "${security}" ]; then
  security="off"
elif [[ "$security" == "on" ]]; then
  CMD="--security"
elif [[ "$security" == "off" ]]; then
  :
else
  usage
fi

EXITCODE="0"

# gets the number of updates from the original check_yum command
NUMUPDATES="$(/usr/lib64/nagios/plugins/check_yum -t ${timeout} 2>/dev/null | awk '{print $1 ";" $5}')"

# handle check_yum plugin timeout
if [[ ${NUMUPDATES} == "YUM;self" ]]; then
  echo "Error: Timeout exceeded (${timeout} seconds)"
  exit 3
fi

# separates the numbers into variables
NUMSECURITYUPDATES="$(echo $NUMUPDATES |cut -d';' -f1)"
NUMOTHERUPDATES="$(echo $NUMUPDATES |cut -d';' -f2)"

# regex for numbers
numcheck='^[0-9]+$'

# checks to see if the number of updates is actually a number
if ! [[ ${NUMSECURITYUPDATES} =~ ${numcheck} ]]; then
  OUTPUT="$(/usr/lib64/nagios/plugins/check_yum 2>/dev/null)"
  echo "Error - Plugin output: ${OUTPUT}"
  exit 3
fi

# if there are security updates, then exit with critical
if [[ ${NUMSECURITYUPDATES} > "0" ]]; then
  EXITCODE="2"
else
  # if there are non-security updates, then exit with warning
  if [[ ${NUMOTHERUPDATES} > "0" ]]; then
    EXITCODE="1"
  else
    EXITCODE="0"
  fi
fi

# output number of updates along with perf data and thresholds
echo "${NUMSECURITYUPDATES} Security Updates Available, ${NUMOTHERUPDATES} Non-Security Updates Available | 'security_updates'=${NUMSECURITYUPDATES};;1;; 'nonsecurity_updates'=${NUMOTHERUPDATES};;1;;"

# stores output of 'yum check-update' into an array (updateArr) using mapfile
mapfile -t updateArr < <( yum check-update --quiet ${CMD})

# iterate through output array
for index in ${!updateArr[*]}
do
  echo "${updateArr[${index}]}"
done

exit ${EXITCODE}

# Original script by watermelon at monitoring-portal.org
# Please report any bugs that you might find with this so that I can improve this script!

#monitoringlove
2 Likes