Remote Icinga instance is not connected

pekohki · April 14, 2023, 11:04am

Hi,
we are running our icinga2 server on a vm. In the past month the server shows a strange behavior:
Some clients are showing “client not connected”. After some time the message disappears and some
other clients are shown to be not connected. A restart of the server is forcing this behavior.

After some checks we so the following error in the log file:
[2023-04-14 10:58:45 +0000] warning/JsonRpcConnection: Error while sending JSON-RPC message for identity ‘xxxx’
Error: Broken pipe

Stacktrace:
0# __cxa_throw in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
1# 0x000056530D3B2CFF in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
2# icinga::NetString::WriteStringToStream(boost::intrusive_ptr<icinga::Sharedicinga::AsioTlsStream > const&, icinga::String const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void ()(), boost::asio::executor> >) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
3# icinga::JsonRpcConnection::WriteOutgoingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void ()(), boost::asio::executor> >) in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
4# 0x000056530D674D66 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
5# 0x000056530D6B1894 in /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2
6# make_fcontext in /lib/x86_64-linux-gnu/libboost_context.so.1.74.0

Our server is running Ubuntu 22.04 LTS
Icinga 2.13.7-1+ubuntu22.04

Any idea what might cause the trouble ?
Bye Peer

rsx · April 14, 2023, 11:12am

Blind guess, you are using unsupported version combination are described here.

pekohki · April 14, 2023, 11:22am

Thanks for the hint. We will check it.
However the master “should” - we have to check - be the newest release.
Bye, Peer

pekohki · April 18, 2023, 6:26am

So I stopped a couple of old clients - without success. For me it looks like this issue

github.com/Icinga/icinga2

Error: JsonRpcConnection: Error while sending JSON-RPC message for identity

opened 02:58PM - 03 Jan 22 UTC

cde0

bug area/api

## Describe the bug I start icinga but i get this Error: JsonRpcConnection: Er…ror while sending JSON-RPC message for identity 'satellite2' Error: Broken pipe And it show that Server is Down. Topology: master -> satellite -> monitor_server ``` [2022-01-03 14:34:59 +0000] information/ApiListener: Finished syncing endpoint 'satellite2' in zone 'satellite2'. [2022-01-03 14:34:59 +0000] warning/JsonRpcConnection: Error while sending JSON-RPC message for identity 'satellite2' Error: Broken pipe Stacktrace: 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2 1# boost::asio::detail::coro_async_result<boost::asio::executor_binder<void (*)(), boost::asio::executor>, unsigned long>::get() in /usr/lib64/icinga2/sbin/icinga2 2# boost::asio::async_result<std::decay<boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >&>::type, void (boost::system::error_code, unsigned long)>::return_type boost::asio::buffered_write _stream<icinga::UnbufferedAsioTlsStream>::async_flush<boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >&>(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::as io::executor> >&) in /usr/lib64/icinga2/sbin/icinga2 3# icinga::JsonRpcConnection::WriteOutgoingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2 4# 0x0000000000AFE697 in /usr/lib64/icinga2/sbin/icinga2 5# 0x0000000000B06D89 in /usr/lib64/icinga2/sbin/icinga2 6# make_fcontext in /lib64/libboost_context.so.1.69.0 [2022-01-03 14:34:59 +0000] warning/JsonRpcConnection: API client disconnected for identity 'satellite2' [2022-01-03 14:34:59 +0000] warning/ApiListener: Removing API client for endpoint 'satellite2'. 0 API clients left. ``` ## To Reproduce IDK :) ## Expected behavior Work, With out a crash ## Screenshots If applicable, add screenshots to help explain your problem. ## Your Environment Include as many relevant details about the environment you experienced the problem in * Version used (`icinga2 --version`): icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.2-1) Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/) License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. System information: Platform: CentOS Linux Platform version: 7 (Core) Kernel: Linux Kernel version: 3.10.0-1160.25.1.el7.x86_64 Architecture: x86_64 Build information: Compiler: GNU 4.8.5 Build host: runner-hh8q3bz2-project-322-concurrent-0 OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017 * Operating System and version: CentOS 7 * Enabled features (`icinga2 feature list`): Enabled features: api checker mainlog * Icinga Web 2 version and modules (System - About): * Icinga Web 2 Version 2.9.5 Git commit 053971c99dc1a4510beb64a888ea695cc14032dc PHP Version 7.2.24-0ubuntu0.18.04.10 Git commit date 2021-11-18 Copyright © 2013-2022 Icinga GmbH * Config validation (`icinga2 daemon -C`): is return with out Error * If you run multiple Icinga 2 instances, the `zones.conf` file (or `icinga2 object list --type Endpoint` and `icinga2 object list --type Zone`) from all affected nodes.

I’m pretty sure that older releases of icinga worked - has anybody tried to downgrade the packages ?
In the moment we have installed:
ii icinga2 2.13.7-1+ubuntu22.04 amd64 host and network monitoring system
ii icinga2-bin 2.13.7-1+ubuntu22.04 amd64 host and network monitoring system - daemon
ii icinga2-common 2.13.7-1+ubuntu22.04 all host and network monitoring system - common files

2:13.3 (or 4) did not show the behavior (pretty sure). Maybe I try it later …