No connections from Icinga client to Icinga master

Icinga 2
1

Hello,
I am trying to connect an Icinga agent as client to the master. The client has the IP 192.168.1.70 and is a VM at the host neckar wit IP 192.168.1.1. On the host a shorewall firewall is running with following rules between host (fw) and VMs (loc):

Chain loc-fw (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 dynamic    all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate INVALID,NEW,UNTRACKED
    0     0 tcpflags   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           
    3   252 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22,52066,52070,52075,52076,52084 /* SSH */
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8 /* Ping */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.70         192.168.1.66         tcp dpt:5665 /* ICINGA */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.75         192.168.1.66         tcp dpt:5665 /* ICINGA */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.84         192.168.1.66         tcp dpt:5665 /* ICINGA */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.85         192.168.1.66         tcp dpt:5665 /* ICINGA */
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 6 prefix "loc-fw ACCEPT "
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
Chain fw-loc (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   24  2016 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22,52066,52070,52075,52076,52084 /* SSH */
    5   420 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  *      *       192.168.1.66         192.168.1.70         tcp dpt:5665 /* ICINGA */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.66         192.168.1.75         tcp dpt:5665 /* ICINGA */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.66         192.168.1.84         tcp dpt:5665 /* ICINGA */
    0     0 ACCEPT     tcp  --  *      *       192.168.1.66         192.168.1.85         tcp dpt:5665 /* ICINGA */
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 0 level 6 prefix "fw-loc ACCEPT "
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

My host with Icinga master is running like

    # systemctl status icinga2
    ● icinga2.service - Icinga host/service/network monitoring system
       Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
      Drop-In: /etc/systemd/system/icinga2.service.d
               └─limits.conf
       Active: active (running) since Thu 2019-10-17 14:13:49 CEST; 8min ago
      Process: 109208 ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2 (code=exited, status=0/SUCCESS)
     Main PID: 109215 (icinga2)
        Tasks: 397

and listen like

    # netstat -tlpn | grep icinga2   
    tcp        0      0 0.0.0.0:5665            0.0.0.0:*               LISTEN      109608/icinga2

When I try to save the certificates for communication about API from the host on the client

icinga2 pki save-cert --host 192.168.1.66 --trustedcert trusted-master.crt

I get this error messages on the client:

information/cli: Retrieving X.509 certificate for '192.168.1.66:5665'.
critical/TcpSocket: Invalid socket: Connection refused
critical/pki: Cannot connect to host '192.168.1.66' on port '5665'
critical/cli: Failed to fetch certificate from host.

and in the firewall log I see:

Oct 17 14:26:26 Shorewall:loc-fw:REJECT:IN=vmbr1 OUT= SRC=192.168.1.70 DST=192.168.1.66 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43451 DF PROTO=TCP SPT=48274 DPT=5665 WINDOW=29200 RES=0x00 SYN URGP=0

The maintainer of shorewall told me at his mailing list the firewall is right configured.
Any ideas for that problem?
Best regards
BrotherA

2

Hi,

try that manually with either tcp connects/telnet, or the upper layer with openssl s_client.

openssl s_client -connect 192.168.1.66:5665

If that fails, the remote end has a problem and the firewall is highly likely the problem. Or there are more than one in place (hardware/router, VM firewall).

Cheers,
Michael

3

Hi,
and thanks for your answer. The command

    openssl s_client -connect 192.168.1.66:5665
    139643014423680:error:0200206F:system library:connect:Connection refused:../crypto/bio/b_sock2.c:110:
    139643014423680:error:2008A067:BIO routines:BIO_connect:connect error:../crypto/bio/b_sock2.c:111:
    connect:errno=111

offers this error. On 192.168.1.66 it runs. The same will be when the firewall is stopped and all rules deleted.
So I ask for the rules at iptables to allow this connection. Icinga2 is on the KVM-host installed like the firewall too.
Another idea will be to realized the connections with SSH. Is that possible?

Best regards

BrotherA

4

Stopping the firewall and rule deletion may not fully have flushed existing iptables. I had this behaviour in the past.

In terms of SSH - no, you really need tcp/5665 here, with TLS & certificate validation this is secure.

Cheers,
Michael

5

Okay, that’s not really a solution. What was your way in the past at this behaviour?

shorewall clear
iptables -F INPUT
iptables -F FORWARD
iptables -F OUTPUT

don’t help. The behaviour stay the same. And there are no switches between Host and client. The client is a VM inside of the host. The client is binded on a bridge vmbr1 and the host is connected to internet on a bridge vmbr0 but it has also connections to the clients on vmbr1.
Has Icinga listen to IP address 192.168.1.66 instead of localhost? If yes, where I can configure it?

Why will be the solution with SSH unsure if all observed VMs are on the host in an internal network behind a firewall?

Best regards
BrotherA

6

I didn’t say that. I said that Icinga itself runs its cluster protocol via tcp/5665 which is equally secure to SSH, likely even harder with CN validation from the certificates.

You need tcp/5665 for the Icinga cluster to work, and one direction to connect.

Your network setup sounds complicated with bridges, I don’t understand that to be honest. So you’re saying that the client runs as guest on the host where the monitoring is run on?

Try to lower it down to the simplest possible way - no firewalls, no extra network settings, direct and straight communication between two IPs via port 5665.

Cheers,
Michael

7

Okay, I try to explain the situation. There is a physical root server. On that root server (internal IP 192.168.1.66, external 194.x.x.x) runs KVM as virtualization. The virtualizied VM’s inlcude Web-Server, DNS-Server, Mail-Server and so on.
On the root server exists an internal network which has a connection to internet over the firewall on the root server and NAT. vmbr0 includes the external interfaces and the whole external IPs which are natted over vmbr1 with the IP 192.168.1.66. It’s a firewall with two interfaces which is shown here Shorewall Firewall. The firewall/router in the pictures is vmbr0 on my system.
In my case, I can’t shutdown the firewall because it’s needed by the Mail server, DNS, Web server and so on.

But I have rebuild Icinga Master and Icinga client on testing VM’s in the same network 192.168.200.0 where the connection runs.
I will try to find the solution for the right rules.

Best regards
BrotherA

8

My rules for that looks like here

pkts bytes target     prot opt in     out     source               destination 
0     0 ACCEPT     tcp  --  *      *       192.168.1.66         192.168.1.70         tcp dpt:5665
0     0 ACCEPT     tcp  --  *      *       192.168.1.70         192.168.1.66         tcp dpt:5665