Bad Request on POST via Icinga2 Api

Hi all -

We are using the Icinga API via a Python library to add/delete/push-updates whatever. Since some weeks (can not determine exactly) we see Bad Request errors in the Icinga logs. Bad Request usually means a malformed request. In our case they occur only on POSTs. Thing is, the Python library sending some 50 of those requests in parallel via asyncio http requests. All the requests are of same type/structure (for testing simply duplicated them, so definitely no difference) but only few are failing. Issue can be work-arounded by synchronizing the requests and introducing a wait of around 200ms between each of them. Decreasing the delay to 150ms or less makes the Bad Request errors occur again. Seems that when too many requests reaching Icinga at the same time (150ms not really at the same time but…) somewhere requests get corrupted.

Anyone had similar problems and can advise where to look for? Since we are using ssl connection debugging is tricky. From Python I see the requests going out correctly but Icinga complaining the Bad Request. Wireshark is of no help here (due to ssl). Is there a chance to get more info from the Icinga logs. We already turned on the debug log, but there is nothing except Connection Open, Connection Close and the Bad Request.

We are using v2.12.2.

Looking forward to hear your ideas.

Cheers,
Oliver

debugging is even trickier no knowing what the requests are or what your script does :slight_smile:
What’s the reason your are using an icinga from 2020-12-01?

Hi and thanks for your answer!

The failing requests are on process-check-result (what my script basically does). But simply firing 50 of those requests (without any more functionality) makes the Bad Request come up.

{'url': 'https://server:5665/v1/actions/process-check-result', 'json': {'host': '703f615f6651f4f28b237c6d9247624c', 'exit_status': 0, 'plugin_output': 'Host UP', 'check_source': 'icinga-sidecar', 'execution_start': 1677832528.548515}}

What’s the reason your are using an icinga from 2020-12-01?

IT not having updated it :frowning: Expecting to be related to a recently fixed bug?

Regards!

Could this be related to a bug fixed in recent versions?

If you want I can try against a 2.13.7 environment, but I need something to execute.

Right now I still don’t know if you send 50 passive check results for one host, or for different hosts at the same time.

Thanks @moreamazingnick , that would be great! I’m sending 50 check results on different hosts via Python asyncio requests. You have some code available?

@moreamazingnick for info. We could narrow it down to some issue in Python library dependencies. I will come back when finished with debugging. Thanks so far!