We are using the Icinga API via a Python library to add/delete/push-updates whatever. Since some weeks (can not determine exactly) we see Bad Request errors in the Icinga logs. Bad Request usually means a malformed request. In our case they occur only on POSTs. Thing is, the Python library sending some 50 of those requests in parallel via asyncio http requests. All the requests are of same type/structure (for testing simply duplicated them, so definitely no difference) but only few are failing. Issue can be work-arounded by synchronizing the requests and introducing a wait of around 200ms between each of them. Decreasing the delay to 150ms or less makes the Bad Request errors occur again. Seems that when too many requests reaching Icinga at the same time (150ms not really at the same time but…) somewhere requests get corrupted.
Anyone had similar problems and can advise where to look for? Since we are using ssl connection debugging is tricky. From Python I see the requests going out correctly but Icinga complaining the Bad Request. Wireshark is of no help here (due to ssl). Is there a chance to get more info from the Icinga logs. We already turned on the debug log, but there is nothing except Connection Open, Connection Close and the Bad Request.
The failing requests are on process-check-result (what my script basically does). But simply firing 50 of those requests (without any more functionality) makes the Bad Request come up.
Thanks @moreamazingnick , that would be great! I’m sending 50 check results on different hosts via Python asyncio requests. You have some code available?
@moreamazingnick for info. We could narrow it down to some issue in Python library dependencies. I will come back when finished with debugging. Thanks so far!
Expecting to be related to a recently fixed bug?