In a client-satellite-mater structure, I have a syslog server lost connection to satellite.
On master can see both satellite and client, but on satellite can’t see client for now, so on master there is warning like: Remote Icinga instance ‘syslog.xxx.yyy’ is not connected to ‘icinga2-satellite1.xxx.yyy’
Base on the following output, is the issue related to CA certificate? or something else, since it is a production environment, I can’t even restart icinga2 service without sufficient evidence. Your help is highly appreciated!
troubleshooting done so far:
-
On satellite #cat /var/log/icinga2/icinga2.log | grep ‘syslog’
show errors as below:
[2020-05-06 17:38:42 -0400] warning/JsonRpcConnection: API client disconnected for identity ‘syslog.xxx.yyy’
[2020-05-06 17:38:52 -0400] information/ApiListener: New client connection for identity ‘syslog.xxx.yyy’ from [IP:X.X.X.X]:60148 (no Endpoint object found for identity)
[2020-05-06 17:38:52 -0400] warning/JsonRpcConnection: Error while processing message for identity ‘syslog.xxx.yyy’ -
On satellite #cat /var/log/icinga2/icinga2.log | grep ‘icinga2*’
show errors as below:
[2020-05-06 17:56:14 -0400] critical/SSL: Error on bio X509 AUX reading pem file ‘/var/lib/icinga2/ca/ca.crt’: 537346050, “error:20074002:BIO routines:FILE_CTRL:system lib”
(2) libbase.so.2.6.3: icinga::GetX509Certificate(icinga::String const&) (+0x3ca) [0x2b4c41a1559a]
(5) libremote.so.2.6.3: icinga::ApiFunction::Invoke(boost::intrusive_ptricinga::MessageOrigin const&, boost::intrusive_ptricinga::Dictionary const&) (+0x34) [0x2b4c42c075e4]
(6) libremote.so.2.6.3: icinga::JsonRpcConnection::MessageHandler(icinga::String const&) (+0x4cf) [0x2b4c42c5b17f]
(7) libremote.so.2.6.3: icinga::JsonRpcConnection::MessageHandlerWrapper(icinga::String const&) (+0x6b) [0x2b4c42c5cc9b]
(8) libbase.so.2.6.3: icinga::WorkQueue::WorkerThreadProc() (+0x529) [0x2b4c41a18a29]
[2020-05-06 17:56:22 -0400] critical/SSL: Could not open CA key file ‘/var/lib/icinga2/ca/ca.key’: 151441516, “error:0906D06C:PEM routines:PEM_read_bio:no start line”