Schedule downtime using PowerShell returns a 500 Internal Server Error

I’ve made a PowerShell (5.1) script that is to be used as a PreUpdateScript with Cluster Aware Updating (CAU).
The purpose is to schedule downtime for the Hyper-V cluster node where the script runs.

When I run the script I get the following error:

Invoke-WebRequest : The remote server returned an error: (500) Internal Server Error.
At line:31 char:1

  • Invoke-WebRequest -Headers $headers -Uri “$Uri” -Method Post -Body $b …
  •   + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
      + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
    

The only log file on the icinga2 server that shows anything related to connecting to it is /var/log/icinga2/icinga2.log.
It shows there is a new client connection and that a Request of POST is made. That is it.

[2019-06-11 11:37:57 +0200] information/ApiListener: New client connection from [ip-address]:51552 (no client certificate)
[2019-06-11 11:37:57 +0200] information/HttpServerConnection: Request: POST /v1/actions/schedule-downtime?host=hostname&type=Host (from [ip-address]:51552), user: username)

The script is below, and I really hope someone can shed a light on what is going wrong.
We even enabled debug mode to get more logging, but… nothing.
I don’t know why it’s giving a 500 or where it’s logged (no, it’s not in /var/log/httpd/*)

$hostname = "<hostname>" # ${env:COMPUTERNAME}.ToLower()

# Start and End time in epoch
$start_time = Get-Date -UFormat %s
$end_time = Get-Date (Get-Date).AddDays(1) -UFormat %s

# Authentication
$user = "<username>"
$pass = "<password>"
$pair = "${user}:${pass}"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"

# Icinga stuff
$Uri = "https://<fqdn>:5665/v1/actions/schedule-downtime?host=$hostname&type=Host"
$headers = @{ "Authorization" = "$basicAuthValue"; "Accept" = "application/json" }
$body = @{
author = '<username>'
comment = 'Cluster Aware Updating'
child_options = 2
start_time = $start_time
endtime = $end_time
}
$body = $body | ConvertTo-Json

# Ignore self signed certificates
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

# Schedule the downtime
Invoke-WebRequest -Headers $headers -Uri "$Uri" -Method Post -Body $body

# Don't ignore self signed certificates anymore
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null

Add &verbose=1 to your params, icinga also returns a more informative error message in the json payload.

With &verbose=1 added I still only see the same entries in /var/log/icinga2/icinga2.log.
I’ve checked other logs, but maybe I am looking in the wrong ones.
Can you tell me which log the information should appear?

And just in case I did the wrong, I added &verbose=1 at the end of the $Uri line.

What I’m interested in is the response you are getting from the server. You should get valid json with a more informative error message.

Ah, no. No json at all. Just the same Invoke-WebRequest error with the 500 error.

Hum, I don’t know too much about the powershell curl client. Can you use curl or a different tool to validate you are requesting the correct url with the right headers and credentials?

Something is going wrong with the payload.
Not sure why it talks about the Accept header not being set either.

C:\temp\curl\curl.exe :   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
At line:32 char:1
+ C:\temp\curl\curl.exe -k  -u '$user:$pass' -H $headers -X POST $Uri - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (  % Total    % ...  Time  Current:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   148    0    67  100    81    435    525 --:--:-- --:--:-- --:--:--   961
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (6) Could not resolve host: Aware
curl: (3) unmatched close brace/bracket in URL position 72:
Updating,
    author:  tweustink,
    start_time:  1560268626,86894
}
                                                                       ^
<h1>Accept header is missing or not set to 'application/json'.</h1>

I then typed out the $headers line and removes the double quotes and added single quotes around the whole like this:

&C:\temp\curl\curl.exe -k -u ‘$user:$pass’ -H ‘Authorization = $basicAuthValue; Accept = application/json’ -X POST $Uri -d $body

It still b0rks about the json part.
curl.exe : % Total % Received % Xferd Average Speed Time Time Time Current
At line:32 char:1
+ &C:\temp\curl\curl.exe -k -u ‘$user:$pass’ -H 'Authorization = $basi …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: ( % Total % … Time Current:String) [], RemoteException
+ FullyQualifiedErrorId : NativeCommandError

                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   148    0    67  100    81   1135   1372 --:--:-- --:--:-- --:--:--  2508
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (6) Could not resolve host: Aware
curl: (3) unmatched close brace/bracket in URL position 72:
Updating,
    author:  tweustink,
    start_time:  1560268982,84317
}
                                                                       ^
<h1>Accept header is missing or not set to 'application/json'.</h1>

The $body looks like this:
PS C:\WINDOWS\system32> $body
{
“endtime”: “1560355382,84417”,
“child_options”: 2,
“comment”: “Cluster Aware Updating”,
“author”: “tweustink”,
“start_time”: “1560268982,84317”
}

Looks like you have some mistake in your syntax

The Accept header is missing is because curl.exe can’t handle multiple headers in one go. Need to use -H multiple times.
Need to check if Invoke-WebRequest needs that too.

The json body part still isn’t working alright (reading your post as I type this).
It seems it can’t handle spaces. No clue why.
The single quotes are needed, and the convert to json looks fine too.
Will have to look into that later.

Now I get a 401 Unauthorized, which is odd as that user is used a lot.
But that should be easy to fix.

Removing the Authorization header fixed the 401.
Removing the -u option but leaving the header didn’t work.

Now it complains about the json text. Specifically the endtime.

Variable reads:

PS C:\WINDOWS\system32> $end_time
1560357985,87855

The error:

{“error”:400.0,“status”:“Invalid request body: Error: lexical error: invalid char in json text.\n { endtime: 1560357985,87\n (right here) ------^\n\n”}

Does it have an isue with more then two decimals?

Edit: Removed the decimals by editing the time variable like this:

$start_time = [int][double]::Parse((Get-Date -UFormat %s))
$end_time = [int][double]::Parse((Get-Date (Get-Date).AddDays(1) -UFormat %s))

1 Like

The issue with your decimals is it’s using a comma instead a dot.

1 Like

Ah, yes. That makes sense. As far as non-interchangeability makes sense.

But when I was figuring it out I did notice Icinga, and other programs, don’t use anything behind the decimal/dot. So I guess this is more safe anyway.

Specifying a floating point number allows a more fine granular timestamp down below to micro seconds. Icinga stores that as double internally and you can later see that via API as well. Still, the dot vs comma problem is with using different locales, especially on German based systems.

This topic holds some interesting details, especially with double quoted variables with using the invariant value, and not the culture being set for the Windows system.

You might just specify the culture in that script, and always enforce the English locale for specific date/time operations then.

Cheers,
Michael

Since I don’t need the floating point number, as I just want to simply set downtime, went with the two lines I posted earlier.
This gives the following output:

PS C:> $start_time
1561121994
PS C:> $end_time
1561208395
PS C:>

The final issue is getting the jason payload to work.
For some odd reason it (I assume icinga2 or curl.exe) doesn’t like spaces in the comment text. It tried to resolve Aware. So I removed the spaces for now.

Now I get a 400.0 error complaining about an invalid char in the json text.
I’ve tried several ways of building the variable, with and without convertto-json. No luck.

{“error”:400.0,“status”:“Invalid request body: Error: lexical error: invalid char in json text.\n { end_time: 1561208211, \n (right here) ------^\n\n”}curl: (6) Could not resolve host: Aware
curl: (3) unmatched close brace/bracket in URL position 66:
Updating,
author: weust,
start_time: 1561121811
}

The last json text is this:

PS C:> $body
{
“end_time”: 1561208395,
“child_options”: 2,
“comment”: “ClusterAwareUpdating”,
“author”: “tweustink”,
“start_time”: 1561121994
}

With this I’m really, really stuck.
All the examples regarding json look like what I have in the $body variable.

Together with a colleague we tackled part of the problem.
It works with curl.exe using the following body variable makeup:

$body = @"
{“author”:“tweustink”,
“comment”:“Cluster Aware Updating”,
“child_options”:2,
“start_time”:$start_time,
“end_time”:$end_time,
“pretty”:true
}
"@

But using the same body variable with either invoke-webrequest of invoke-restmethod is fails with a 400.

And I need to fix the time/date to remove two hours as it’s adding +2 hours right now.
Perhaps I do need to work with the link posted earlier.

Update: I fixed the date/time by adding (Get-Date).ToUniversalTime() to the $start_time variable.
It’s may not be super nice, but it works.

$start_time = [int][double]::Parse((Get-Date (Get-Date).ToUniversalTime() -UFormat %s))

Hi @weust , Would you mind sharing your final script block to schedule downtime with Powershell you were able to get working ? I was trying to accomplish something very similar and I stumbled upon this thread.

It’s not working 100% as I wanted too, because it’s using an external program (curl.exe) but it works.
I prefer to run it using either Invoke-WebRequest or Invoke-RestMethod, but I need to fix the $body (json). I think the 's need to be something different.

Also, when using curl.exe the [System.Net.ServicePointManager] lines are not needed.
The -k option of curl.exe takes care of that.

Another also, the Authentication block’s last three lines are mainly needed for the Invoke-WebRequest/RestMethod part.

Other then that, it manages to schedule downtime for 26 hours.
I didn’t bother modifying $end_time as the node won’t be in maintenance that long anyway, and a PostUpdate script will take it out of downtime. This is a script I haven’t started working on.

$hostname = ${env:COMPUTERNAME}.ToLower()

# Start and End time in epoch
$start_time = [int][double]::Parse((Get-Date (Get-Date).ToUniversalTime() -UFormat %s))
$end_time = [int][double]::Parse((Get-Date (Get-Date).AddDays(1) -UFormat %s))

# Authentication
$user = "<username>"
$pass = "<password>"
$pair = "${user}:${pass}"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"

# Icinga stuff
$Uri = "https://icinga.example.net:5665/v1/actions/schedule-downtime?host=$hostname&type=Host"
$headers = @{ "Authorization" = "$basicAuthValue"; "Accept" = "application/json" }
$body = @"
    {\"author\":\"<username>\",
    \"comment\":\"Cluster Aware Updating\",
    \"child_options\":2,
    \"start_time\":$start_time,
    \"end_time\":$end_time,
    \"pretty\":true
}
"@

# Ignore self signed certificates
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

# Schedule the downtime
# Invoke-WebRequest -Headers $headers -ContentType 'application/json' -Uri $Uri -Method Post -Body $body
&C:\temp\curl\curl.exe -k -u $pair -H "Content-Type: application/json" -H "Accept: application/json" -X POST $Uri -d $body
# Invoke-RestMethod -Headers $headers -Uri $Uri -Method Post -ContentType 'application/json' -Body $body

# Don't ignore self signed certificates anymore
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
1 Like

Finally got it working without the external curl.exe!
Not sure why, I think I tried this before, but I redid the body and used Invoke-RestMethod.
This means the whole script is now pure PowerShell.

$hostname = ${env:COMPUTERNAME}.ToLower()

# Start and End time in epoch
$start_time = [int][double]::Parse((Get-Date (Get-Date).ToUniversalTime() -UFormat %s))
$end_time = [int][double]::Parse((Get-Date (Get-Date).AddDays(1) -UFormat %s))

# Authentication
$user = "<username>"
$pass = "<password>"
$pair = "${user}:${pass}"
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
$basicAuthValue = "Basic $base64"

# Icinga stuff
$Uri = "https://icinga.example.net:5665/v1/actions/schedule-downtime?host=$hostname&type=Host"
$headers = @{ "Authorization" = "$basicAuthValue"; "Accept" = "application/json" }
$body = @{
    author          = '<username>'
    comment         = 'Cluster Aware Updating'
    child_options   = 2
    start_time      = $start_time
    end_time        = $end_time
}
$body = $body | ConvertTo-Json

# Ignore self signed certificates
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = { $true }

# Schedule the downtime
Invoke-RestMethod -Headers $headers -Uri $Uri -Method Post -ContentType 'application/json' -Body $body

# Don't ignore self signed certificates anymore
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
2 Likes

Thanks for sharing! It working great in my environment - I just made a little modification to set the downtime for an hour. Glad you were able to get it working with just native PowerShell.

You’re welcome. I’m glad it’s working too.

The one hour maybe too short in a new setup, and since there will be a remove-downtime post-update script someday (when I figure out the API for it, which sucks btw) I think it’s fine.
It’s also in line with a python script used for our OpenStack environment.