Server unable to perform TLS Handshake

Icinga 2
1

Hello,
we´ve got a problem with the connection between our Icinga server and a remote Icinga server. This problem is fairly new and the connection worked flawless before. The remote server is reachable via ping from our Icinga server and is embedded in the Webapplication(Also reachable there).
But when performing checks the output is "Remote Icinga instance ‘…’ is not connected to ‘icingaserver…’.
While investigating the problem we came to the conclusion the TLS handshake isn´t working.

Remote Server:

[2022-04-29 07:11:00 +0000] critical/ApiListener: Client TLS handshake failed (from [192.168.16.80]:51716): Operation canceled

Icinga Server:

[2022-04-29 09:12:40 +0200] critical/ApiListener: Client TLS handshake failed (to [192.168.19.69]:5665): Operation canceled

[2022-04-29 09:12:40 +0200] information/ApiListener: Finished reconnecting to endpoint 'dataexchange via host ‘192.168.19.69’ and port ‘5665’

I can exclude any certificate problems, OpenSSL versions are updated. There is a 2 hour time difference but thats because of timechange (Normal-/Summertime). The remote server is a CentOS 7.

  • Version used (icinga2 --version)

Version: r2.13.3-1

  • Operating System and version

    System information:
    Platform: Ubuntu
    Platform version: 20.04.4 LTS (Focal Fossa)
    Kernel: Linux
    Kernel version: 5.4.0-109-generic
    Architecture: x86_64
    Build information:
    Compiler: GNU 9.4.0
    Build host: runner-hh8q3bz2-project-298-concurrent-0
    OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020

  • Enabled features (icinga2 feature list)

Disabled features: command compatlog elasticsearch gelf graphite influxdb2 opentsdb syslog
Enabled features: api checker debuglog icingadb ido-mysql influxdb livestatus mainlog notification perfdata
statusdata

  • Config validation (icinga2 daemon -C)

[2022-05-02 12:51:59 +0200] information/cli: Icinga application loader (version: r2.13.3-1)
[2022-05-02 12:51:59 +0200] information/cli: Loading configuration file(s).
[2022-05-02 12:51:59 +0200] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/company-rsdmz’ for unknown zone ‘company-rsdmz’.
[2022-05-02 12:51:59 +0200] warning/config: Ignoring directory ‘/var/lib/icinga2/api/zones/company-rsdmz-satellite’ for unknown zone ‘company-rsdmz-satellite’.
[2022-05-02 12:51:59 +0200] information/ConfigItem: Committing config item(s).
[2022-05-02 12:51:59 +0200] information/ApiListener: My API identity: srv-u-syshealth2.company.private
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 17 HostGroups.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 58 Hosts.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 23 Downtimes.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 77 Notifications.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 35 Zones.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 33 Endpoints.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 3 ApiUsers.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 LivestatusListener.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 300 CheckCommands.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 4 Users.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 2 UserGroups.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 6 TimePeriods.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 900 Services.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 5 ServiceGroups.
[2022-05-02 12:52:00 +0200] information/ConfigItem: Instantiated 23 ScheduledDowntimes.
[2022-05-02 12:52:00 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2022-05-02 12:52:00 +0200] information/cli: Finished validating the configuration file(s).

2

Hello @info-systems!

Sounds like a timeout. Are you sure :5665 is open in your firewall?

Best,
A/K

3

Hello,
Thank you for your response.
I tried a telnet on the port 5665. It connects but closes after just a few seconds.
Netstat also shows the Port is open on the other side.

4

Also with TZ=UTC date?

5

It has worked with the time difference before. Its mandatory for the remote server to run in that timezone. There were no problems in the past. But i already excluded the time difference when i switched the time for a test. Hasn’t worked either.

6

I didn’t ask for a timezone change, just for whether/ how much the command outputs differ.

7

Sorry, could you please explain what you mean?
There is a difference in time because of the different timezones. Otherwise there is no difference…?

8

I’ve just asked for the output of TZ=UTC date per node.

9

Hello, sorry for the delay. Here is the output:

TZ=UTC date

Mon 16 May 08:57:11 UTC 2022

TZ=UTC date
Mon 16 May 2022 08:57:09 AM UTC

So we see no time difference (apart from the two seconds that I needed to repeat the command).
We do see a formatting difference in the date but I do not think this poses a problem!?

10

Hello,
I just wanted to inform you, the problem just solved by itself. It suddenly works flawless again.

Thanks for your help.