Windows Event Command example is not working

Icinga 2
1

I’m having an issue with EventCommand and restarting a service on Windows.

I’m trying to have a service restart if it is found to be down and using https://icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#use-event-commands-to-restart-service-daemon-via-command-endpoint-on-windows almost verbatim since it seems it should do exactly what I want.

However, the issue is that the service never restarts when the conditions are met and it looks like the command is issued by Icinga. I confirmed that the powershell script will work properly to restart the service if I run it manually.

I even tried the adjusted eventcommand shown in Event Commands, Windows Services, & Director, but it still didn’t work properly.

Can anyone see where I might have missed something?

eventcommands.conf:

object EventCommand "restart_service-windows" {
  command = [
    "C:\\Windows\\SysWOW64\\WindowsPowerShell\\v1.0\\powershell.exe",
    PluginDir + "/restart_service.ps1"
  ]

  arguments = {
    "-ServiceState" = "$service.state$"
    "-ServiceStateType" = "$service.state_type$"
    "-ServiceAttempt" = "$service.check_attempt$"
    "-Service" = "$restart_service$"
    "; exit" = {
        order = 99
        value = "$$LASTEXITCODE"
    }
  }

  vars.restart_service = "$service_win_service$"
}

Adjusted eventcommands.conf from https://community.icinga.com/t/event-commands-windows-services-director/1309/7:

object EventCommand "restart_service-windows" {
 command = [ "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe" ]

 arguments = {
  "-command" = {
   value = "& ‘C:\\Program Files\\ICINGA2\\sbin\\restart_service.ps1’"
   order = -1
  }

  "-ServiceState" = "$service.state$"
  "-ServiceStateType" = "$service.state_type$"
  "-ServiceAttempt" = "$service.check_attempt$"
  "-Service" = "$restart_service$"
  "; exit" = {
   order = 99
   value = "$$LASTEXITCODE"
  }
 }
  vars.restart_service = "$service_win_service$"
}

Service_apply.conf section for the service:

apply Service "Example Service Check" {
    import "check_service"

    assign where "examples" in host.groups
    vars.service_win_service = "example"

    event_command = "restart_service-windows"
}

Relevant service_templates.conf sections:

template Service "generic-service" {
    max_check_attempts = "5"
    check_interval = 1m
    retry_interval = 30s
    enable_perfdata = true
}

template Service "generic-service-agent" {
    import "generic-service"
    command_endpoint = host.vars.agent_endpoint
}

template Service "check_service" {
    import "generic-service-agent"

    check_command = "service-windows"
}

Debug log sections of the event command running on the server, first for the command as it is in the documentation and second for the adjusted command from the community post:

[2020-03-26 12:06:35 -0700] notice/Process: Running command 'C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe "C:\Program Files\ICINGA2\/sbin/restart_service.ps1" -Service example -ServiceAttempt 3 -ServiceState CRITICAL -ServiceStateType SOFT "; exit" $LASTEXITCODE': PID 7852
[2020-03-26 12:06:35 -0700] notice/ApiListener: Sending message 'event::ExecuteCommand' to 'SATHOSTNAME'
[2020-03-26 12:06:35 -0700] notice/Process: PID 7852 ('C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe "C:\Program Files\ICINGA2\/sbin/restart_service.ps1" -Service example -ServiceAttempt 3 -ServiceState CRITICAL -ServiceStateType SOFT "; exit" $LASTEXITCODE') terminated with exit code 0

[2020-03-26 11:55:07 -0700] notice/Process: Running command 'C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -command "& ‘C:\Program Files\ICINGA2\sbin\restart_service.ps1’" -Service example -ServiceAttempt 3 -ServiceState CRITICAL -ServiceStateType SOFT "; exit" $LASTEXITCODE': PID 1300
[2020-03-26 11:55:07 -0700] notice/ApiListener: Sending message 'event::ExecuteCommand' to 'SATHOSTNAME'
[2020-03-26 11:55:08 -0700] notice/Process: PID 1300 ('C:\Windows\system32\WindowsPowerShell\v1.0\powershell.exe -command "& ‘C:\Program Files\ICINGA2\sbin\restart_service.ps1’" -Service example -ServiceAttempt 3 -ServiceState CRITICAL -ServiceStateType SOFT "; exit" $LASTEXITCODE') terminated with exit code 0
2

HI,

from icinga side everything works. Icinga does fire the script when the state changes. Is the Icinga service user allowed to restart the services? Also can you please post the restart script here, so we can use it for tests.

Regards,
Carsten

3

Here’s the restart_service.ps1 script:

param(
        [string]$Service                  = '',
        [string]$ServiceState             = '',
        [string]$ServiceStateType         = '',
        [int]$ServiceAttempt              = ''
    )

if (!$Service -Or !$ServiceState -Or !$ServiceStateType -Or !$ServiceAttempt) {
    $scriptName = GCI $MyInvocation.PSCommandPath | Select -Expand Name;
    Write-Host "USAGE: $scriptName -ServiceState servicestate -ServiceStateType servicestatetype -ServiceAttempt serviceattempt -Service service" -ForegroundColor red;
    exit 3;
}

# Only restart on the third attempt of a critical event
if ($ServiceState -eq "CRITICAL" -And $ServiceStateType -eq "SOFT" -And $ServiceAttempt -eq 3) {
    Restart-Service $Service;
}

exit 0;

As for the user that Icinga runs as, the process runs as the “NT AUTHORITY\SYSTEM” user, which seems that it should have permission to restart services.

Additional details is that the server is running Windows Server 2012 R2 Standard.

4

Just an update today: I tried having Icinga run as my user(since I can run the script manually via powershell) and it didn’t restart the service(logs show the same entry as the one above, but the service remained offline).

5

I apologize for bumping, but I got really busy and forgot to reply to this topic when I got the issue solved( also apologize to anyone that has come across this topic looking for answers and not found them).

Not too sure exactly what fixes it, but the following is the event command that does work to have Icinga issues the restart:

object EventCommand "restart_service-windows" {
  command = [ "C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe" ]

  arguments = {
    "-command" = {
     value = "C:\\Program` Files\\ICINGA2\\sbin\\restart_service.ps1"
     order = -1
    }
    "-ServiceState" = "$service.state$"
    "-ServiceStateType" = "$service.state_type$"
    "-ServiceAttempt" = "$service.check_attempt$"
    "-Service" = "$restart_service$"
    "-ServiceHandled" = "$service.handled$"
    "; exit" = {
        order = 99
        value = "$$LASTEXITCODE"
    }
  }

  vars.restart_service = "$service_win_service$"
}
1 Like
6

It was the backtick :wink: