Windows PowerShell Checks with Icinga2

Author: @GordonCole

Revision: v0.1

Tested with:

  • Icinga 2 v2.6.3-1
  • Icinga Web 2 v2.4.1
  • Windows Server 2012 R2

Introduction

A vanilla Windows Icinga2 installation provides access to a number of standard server health and performance metrics. For example hard disk space, CPU, free RAM, or accessing the value of a Windows Performance Counter. Each of these is metrics is measured using a “check”. These are programs called by the main Icinga2 service.

Users may write their own “checks”, as long as they return a result in the expected format (status, performance data, text).

PowerShell has established itself as a powerful way of automating tasks and accessing information on a Windows machine. We can use PowerShell to access server metrics and return data to Icinga2 by writing a suitable PowerShell script.

Configuration

Create a Check Command

Here we define a CheckCommand so that Icinga2 knows the path of the executable to call, in this case the powershell.exe interpreter (a PowerShell session).

This should be defined in commands.conf

object CheckCommand "powershell_check" {
  import "plugin-check-command"
  command = [ "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe" ]
  arguments = {
    "-command" = {
    value = "$ps_command$"
    order = -1
    }
    "-warn" = {
    value = "$ps_warn$"
    }
    "-crit" = {
    value = "$ps_crit$"
    }
    ";exit" = {
    value = "$$LastExitCode"
    }
  }
}

This will run the 32-bit version of PowerShell. If you want to use the 64-bit version use the following command instead:

  command = [ "C:\\Windows\\sysnative\\WindowsPowerShell\\v1.0\\powershell.exe" ]

Let’s consider the arguments:

  • -command - this contains a variable which will contain the path to our PowerShell script (our check)

  • -warn - this contains a variable which may be set (elsewhere) to contain a warning threshold value (optional - your script needs to be written to be able to accept arguments)

  • -crit - this contains a variable which may be set (elsewhere) to contain a critical threshold value (optional - your script needs to be written to be able to accept arguments)

  • ;exit - we always pass this argument the value “$LastExitCode”. This is not an Icinga2 variable. This argument tells the powershell.exe session to take the exit code generated by your PowerShell script, and use this for the exit code when the session exits. This exit code is important as the plug-in’s service “status” is taken from the exit code (0=OK, 1=Warning, 2=Critical, 3=Unknown). Note also the “;” in the above code. If you omit this, you powershell.exe will aways exit with code “0” and your check wont have the correct status.

  • “\” is used to escape the “\” so we see “\\” in the path name

  • “$” is used to escape the “$” so we see “$$” in the string “$LastExitCode”

If you find the above confusing, just remember that Icinga2 needs to construct a command that you could yourself enter at the command prompt in order to run a PowerShell script. For example:

C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -command "&'C:\Program Files\ICINGA2\sbin\check_reboot.ps1' ;exit $LastExitCode

This is very useful as you can manually test your scripts (“plug ins”) from the command line without involving Icinga2. You cannot, however, see the exit code.

The above puts in place the underlying ability to call PowerShell from Icinga2.

Define a Service

We can define an Icinga2 “service” that references a PowerShell scripts or “plug-ins”. It relies on the check command we defined in the previous step.

For example:

apply Service "reboot_status_check" {
  import "generic-service"
  display_name = "Reboot Check"
  check_command = "powershell_check"
  vars.ps_command = "& 'C:\\Program Files\\ICINGA2\\sbin\\check_reboot.ps1'"
  command_endpoint = host.address
  assign where host.vars.os == "windows"
}

In this case we are using the variable vars.os in the host object definition to apply this service to all our “windows” hosts.

Write PowerShell script (Plug In)

We now require a suitable PowerShell script to act as our “plug in”. Note that monitoring plug-ins should adhere to guidelines regarding what you can pass them, and what they should return. This example may not be fully compliant in this regard.

You should consult https://www.monitoring-plugins.org/doc/guidelines.html for detailed information on writing plugins.

	# Checks if RebootRequired key exists, if so returns a warning.
	# This key is deleted upon a successful reboot.
	# This may indicate that Windows patching has taken place, without a reboot.
		 
	# Checks if RebootRequired reg path exists
	$value = test-path -path "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\RebootRequired"
	 
	# If path does not exist, return OK status
	if ($value -match "False") {
	echo "OK - No reboot required"
	$returnCode=0
	}
	 
	# Else return WARNING status
	else {
	echo "WARNING - Reboot required"
	$returnCode=1
	}
	 
	exit ($returnCode)

To note:

  • return code is passed as the “exit” code. This in turn is passed back to Icinga2. This is the “status” of the service check
  • This is a simple example and does not include any error handling, which you would normally consider
  • We need to save this script on our Windows server in the correct path, as shown in our service definition. This is best placed in the same location as the other Icinga2 Windows checks:

C:\Program Files\ICINGA2\sbin\

The exit codes from the plug-in relate to Icinga2 service status as follows:

  • 0 = OK
  • 1 = Warning
  • 2 = Critical
  • 3 = Unknown

Permissions

It should be noted that the ability to run PowerShell may be restricted by group policy.

You should consider the user that will be used to run these scripts and what rights they will need. For example, an Icinga2 service run as “Local System” will therefore run the script as the user “local system”. This user can check the registry but likely wont have rights over an SQL database, for example. Care must be taken when assigning rights.

Conclusion

Once you can harness PowerShell, your abilities to check Windows server health and vital statistics is limited only by your scripting knowledge. You may, for example:

  • Check registry keys (some applications store their status, for example “primary” or “stand-by” in the registry)
  • Check files are present and updated in particular folders (useful in file processing systems)
  • Parse application log files for error codes and generate Icinga2 warnings

You may be able to use a combination of Icinga2 and PowerShell and remove the requirement for other agents such as NSClient++.

FAQ

6 Likes

I added the command via director and I get this error when trying to use the command check.

execvpe(C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe) failed: No such file or directory

I am using the Icinga2 Windows Agent.

What could be the problem? The command/path works fine locally on the server.

Hi Colin,

i solved it by extending the command-plugins-windows.conf (on Windows Client configuration!!!) by the given commands.conf code above.
Also the endpoint has to point to the client i learned, so my Service looks:

apply Service "reboot_status_check" {
  import "generic-service"
  display_name = "Reboot Check"
  check_command = "powershell_check"
  vars.ps_command = "& 'C:\\Program Files\\ICINGA2\\sbin\\check_reboot.ps1'"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.os == "Windows" && host.vars.client_endpoint
}

Hi @dnsmichi,

thanks four your howto.
I run into the issue where my Icinga2web says “Check command ‘powershell_check’ does not exist.”
I created the CheckCommand in the commands.conf on the Server (and the client.)
Created the service through the icinga director and set them to run on the agent.

My envirement is a debian server with icinga2 r2.10.4-1
I tried to run a script on a Server 2019

thanks for your efforts
Manuel

After adding commands in any conf file you need to run Kickstart Wizard to inform the director about the changes.

1 Like

hmmm… just keep writing ps-scripts and call them via nrpe…

Hi Manuel,

if you created the objects in director, define the CheckCommand in the zone director-global. this publish the configuration to the endpoints. So no additional work is necessary on endpoints.

regards,
Michael

2 Likes

Is it possible to give additional arguments to the script?

object CheckCommand “check_ms_iis_application_pool” {

import "plugin-check-command"

command = [ "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" ]

arguments = {

    "-command" = {

    value = "& 'C:\\scripts\\check_ms_iis_application_pool.ps1' "

    order = -1

    }

    "-A" = {

    value = "$windows_appPool$"

    }

}

vars.windows_appPool = "$windows_appPool$"

}

Would render:

‘C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe’ ‘-command’ ‘& ‘’‘C:\scripts\check_ms_iis_application_pool.ps1’’’ ’
and vars.windows_appPool=BANANA

But

object CheckCommand “check_ms_iis_application_pool” {

import "plugin-check-command"

command = [ "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" ]

arguments = {

    "-command" = {

    value = "& 'C:\\scripts\\check_ms_iis_application_pool.ps1' "

    order = -1

    }

    "-A" = {

    value = "BANANA"

    }

}

vars.windows_appPool = "$windows_appPool$"

}

Vould render:

‘C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe’ ‘-command’ ‘& ‘’‘C:\scripts\check_ms_iis_application_pool.ps1’’’ ’ ‘-A’ ‘BANANA’

Hi,
i have the problem, that the rendered Commands are not executeable, because the powershell.exe is qouted.

It´s the same as @Coffe described.

Are there any solutions?

Hi,

I tried this out and created a check command with Icinga Director.
It’s all fine, expect one thing. The exit code is not interpreted by Icinga. The result returned by the plugin is critical, but it’s show as Ok. I set the semicolon before the exit as suggested.

The command preview from the director is:

object CheckCommand "check-win-firewall-public-profile-enabled" {
    import "plugin-check-command"
    command = [
        "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe"
    ]
    arguments += {
        "-Profile" = "Public"
        "-command" = {
            order = -1
            value = "Invoke-IcingaCheckFirewall"
        }
        ";exit" = "$$LastExitCode"
    }
}

The executed command:
‘C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe’ ‘-command’ ‘Invoke-IcingaCheckFirewall’ ‘-Profile’ ‘Public’ ‘;exit’ ‘$LastExitCode’

and the result on the dashboard:
image

Any idea what’s wrong?

Hi

Not sure if you got this solved but looks like you had the wrong format at the end, should be something like this instead:

";exit" = {
    value = "$$LastExitCode"
    }

Not like the one you did:

";exit" = "$$LastExitCode"
1 Like

Yes, in the mean time I realized it :slight_smile:
Thank you!

In case someone wants the check to not be in an OK state when the script to be called by the PowerShell is not existing, I came up with this monstrosity :smiley:

object CheckCommand "PowerShell Custom Script" {
    import "plugin-check-command"
    command = [
        "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe",
        "-executionpolicy",
        "bypass",
        "-noprofile"
    ]
    timeout = 3m
    arguments += {
        "-command" = {
            description = "This will contain the path to our PowerShell script"
            order = 0
            required = true
            value = "if(!(Test-Path \"$ps_script_path$\")){ Write-Host \"Script $ps_script_path$ not found.\" ;   exit 1 } else { & $ps_script_path$"
        }
        "};exit" = {
            order = 99
            required = true
            value = "$$LastExitCode"
        }
    }
}

This will first check if the script, given via the variable $ps_schript_path$, exists. If not, it will exit with exit code 1 (WARNING), which is then reflected by the check.

The order = 99 on the exit “argument” is to be able to create other commands that import this one and make arguments for those scripts possible.

Example:

object CheckCommand "mdatp-status_win" {
    import "plugin-check-command"
    import "PowerShell Custom Script"

    arguments += {
        "-crit" = {
            description = "Critical threshold in hours for definition age"
            order = 97
            required = true
            value = "$mdatp_win_definition_crit$"
        }
        "-warn" = {
            description = "Warning threshold in hours for definition age"
            order = 98
            required = true
            value = "$mdatp_win_definition_warn$"
        }
    }
}
1 Like