Firewall setup at a Icinga2 cluster with iptables & ipset

Tutorials and Guides
1

Issue

The last production environment I deployed was running at DigitalOcean. Their cloud firewall does not support VRRP (yet!) which was an issue as this was required to achieve high availability and a proper handling of failed nodes.

Solution? A local firewall!

ipset is a real nice tool for well-structured iptables and firewall configurations. It looks like ipset is not that common - so I decided to share it here. :slight_smile:

At the end you will know how to setup a firewall with the required iptables rules and ipsets to have a working icinga2 cluster with SSH, HTTP and HTTPS access from your office.

It will be ease to add new environments or drop old ones.

Keep in mind that this article just uses a very small subset of the capabilities of iptables and ipset. See the respective man pages for further information.

Environment

This was tested on

  • Ubuntu 16.04
  • Ubuntu 18.04.3
  • CentOS 7.6

Please let me know if you experience any issues on other distributions. I’m happy to help and extend this how-to.

Setup

Installation - Ubuntu / Debian

The required packages are:

  • ipset
  • iptables
  • netfilter-persistent

Install them via

sudo apt install ipset iptables netfilter-persistent

Note: iptables is probably already installed.

The installer will ask you if the currently applied iptables rules should be written to the /etc/iptables/rules.v4 and /etc/iptables/rules.v6. Answer with yes if you applied your own rules!

Installation - CentOS / RedHat

The required packages are:

  • ipset
  • ipset-service
  • iptables
  • iptables-services

Install them via

sudo apt install ipset iptables iptables-services

Configuration

The configuration files will be stored in the following location:

Debian / Ubuntu

Content Location
IPv4 rules /etc/iptables/rules.v4
IPv6 rules /etc/iptables/rules.v6
IPv4 ipsets /etc/iptables/ipset.v4
IPv6 ipsets /etc/iptables/ipset.v6

CentOS

Content Location
IPv4 rules /etc/sysconfig/iptables
IPv6 rules /etc/sysconfig/iptables6
IPv4 ipsets /etc/iptables/ipset
IPv6 ipsets /etc/iptables/ipset6

None of the files requires permissions greater than 644.

The examples will show the configuration for IPv4. It works for IPv6, too, but I do not have as much experience with IPv6 as I have with IPv4 so I will not post any examples. The reason is obvious, I guess. :wink:

ipset

We will create the following ipsets:

Set name Purpose
COMPANY-OPS-OFFICE-NETWORKS SRC networks for SSH, HTTP and HTTPS requests
UPSTREAM-DNS-SERVERS IPs of used DNS servers
MONITORING-MASTERS IPs of both Icinga2 nodes
COMPANY-SATELLITES IPs of all satellites hosting services for the company itself
CUSTOMER1-SATELLITES IPs of all satellites in the environment of CUSTOMER1
ALL-SATELLITES A list that groups all *-SATELLITES lists into a single list

The names above are just examples. Feel free to change them or at least replace COMPANY by the name of your project or company and CUSTOMERx by the actual name (or the actual JIRA project key for example) of the customer.

Example ipset / ipset.v4 file

-N -! COMPANY-OPS-OFFICE-NETWORKS hash:net 
-A -! COMPANY-OPS-OFFICE-NETWORKS 134.124.144.192/28
-A -! COMPANY-OPS-OFFICE-NETWORKS 234.214.24.72/29

-N -! UPSTREAM-DNS-SERVERS hash:net
-A -! UPSTREAM-DNS-SERVERS 8.8.8.8/32
-A -! UPSTREAM-DNS-SERVERS 8.8.4.4/32
-A -! UPSTREAM-DNS-SERVERS 1.1.1.1/32

-N -! MONITORING-MASTERS hash:net
-A -! MONITORING-MASTERS 10.135.135.71/32
-A -! MONITORING-MASTERS 10.135.135.96/32

-N -! COMPANY-SATELLITES hash:net
-A -! COMPANY-SATELLITES 12.13.14.15/32
-A -! COMPANY-SATELLITES 33.44.55.160/32
-A -! COMPANY-SATELLITES 11.12.33.160/27

-N -! CUSTOMER1-SATELLITES hash:net
-A -! CUSTOMER1-SATELLITES 66.77.88.99/32
-A -! CUSTOMER1-SATELLITES 66.79.87.96/32
-A -! CUSTOMER1-SATELLITES 66.80.88.196/32

-N -! ALL-SATELLITES list:set
-A -! ALL-SATELLITES MONITORING-MASTERS
-A -! ALL-SATELLITES COMPANY-SATELLITES
-A -! ALL-SATELLITES CUSTOMER1-SATELLITES

NOTES:

  • -N adds a new set
  • -A adds an object to an existing set
  • hash:net & list:set define the type set type
  • -! is a flag to ignore errors if the applied entry / set does already exists
  • MONITORING-MASTERS are added to ALL-SATELLITES to reduce the need of redundant iptables rules (see below)

Example iptables rules

Be careful and proceed only if you know what you’re doing! Make sure to just add the missing rules to your iptables rule file if you have your additional rules in place already!

The rules

  • grant HTTP, HTTPS, SSH access to the server from your own network(s)
  • allow connections from satellites to the local icinga2 nodes
  • allow both icinga2 nodes to communicate with each other
  • allow icinga2 to write to the local carbon-relay daemon and haproxy (or MySQL)
  • allow icingaweb2 to connect to the local icinga2 daemon
  • allow DNS queries to systemd-resolved and accept DNS responses from the upstream DNS servers
# Generated by iptables-save v1.6.1 on Thu Sep  5 21:42:45 2019
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -m comment --comment "already established Sessions are always allowed" -j ACCEPT
-A INPUT -s 127.0.0.1/32 -p udp -m udp --dport 53 -m comment --comment "DNS queries to the local systemd-resolved" -j ACCEPT
-A INPUT -m set --match-set UPSTREAM-DNS-SERVERS src -p udp -m udp --sport 53 -m comment --comment "Allow DNS responses from upstream servers" -j ACCEPT
-A INPUT -p tcp -i lo --dport 2013 -m comment --comment "Allow connections from icinga2 to carbon-relay" -j ACCEPT
-A INPUT -p tcp -i lo --dport 3306 -m comment --comment "Allow MySQL from localhost (icinga ido)" -j ACCEPT
-A INPUT -p tcp -i lo --dport 3307 -m comment --comment "Allow MySQL from localhost (director)" -j ACCEPT
-A INPUT -p tcp -i lo --dport 3308 -m comment --comment "Allow MySQL from localhost (icingaweb)" -j ACCEPT
-A INPUT -p tcp -i lo --dport 5665 -m comment --comment "Allow connections from icingaweb2 to the local backend" -j ACCEPT
-A INPUT -m set --match-set COMPANY-OPS-OFFICE-NETWORKS src -p tcp -m multiport --dports 22,80,443 -m comment --comment "SSH, HTTP, HTTPS access from company networks" -j ACCEPT
-A INPUT -m set --match-set COMPANY-OPS-OFFICE-NETWORKS src -p icmp -m icmp --icmp-type 8 -m comment --comment "ICMP Echo Requests - COMPANY Office" -j ACCEPT
-A INPUT -m set --match-set COMPANY-OPS-OFFICE-NETWORKS src -p icmp -m icmp --icmp-type 0 -m comment --comment "ICMP Echo Replies - COMPANY Offcie" -j ACCEPT
-A INPUT -m set --match-set COMPANY-OPS-OFFICE-NETWORKS src -p icmp -m icmp --icmp-type 3 -m comment --comment "ICMP Destination Unreachable - COMPANY Office" -j ACCEPT
-A INPUT -m set --match-set COMPANY-OPS-OFFICE-NETWORKS src -p icmp -m icmp --icmp-type 11 -m comment --comment "ICMP Time exceeded - COMPANY Office" -j ACCEPT
-A INPUT -m set --match-set MONITORING-MASTERS src -p vrrp -m comment --comment "Allow VRRP between cluster nodes" -j ACCEPT
-A INPUT -m set --match-set ALL-SATELLITES src -p icmp -m icmp --icmp-type 8 -m comment --comment "ICMP Echo Requests - COMPANY Satellites" -j ACCEPT
-A INPUT -m set --match-set ALL-SATELLITES src -p icmp -m icmp --icmp-type 0 -m comment --comment "ICMP Echo Replies - COMPANY Satellites" -j ACCEPT
-A INPUT -m set --match-set ALL-SATELLITES src -p icmp -m icmp --icmp-type 3 -m comment --comment "ICMP Destination Unreachable - COMPANY Satellites" -j ACCEPT
-A INPUT -m set --match-set ALL-SATELLITES src -p icmp -m icmp --icmp-type 11 -m comment --comment "ICMP Time exceeded - COMPANY Satellites" -j ACCEPT
-A INPUT -m set --match-set ALL-SATELLITES src -p tcp --dport 5665 -m comment --comment "Allow Icinga2 connections from all satellites" -j ACCEPT
COMMIT

NOTE 1: The MySQL rules are only required if a local haproxy as described in this awesome how-to from @anon66228339 Galera MySQL cluster with VIPs and HAProxy for IDO-Mysql and more
Note2: list:set sets can not be added to list:set sets. So we have the ICMP rules twice.

Manage services

CentOS

# enable services 
sudo systemctl enable ipset
sudo systemctl enable iptables

# start the services
sudo service ipset start
sudo service iptables start

Ubuntu

The ipset service file needs to be created manually. Create /etc/systemd/system/ipset.service with the following content:

[Unit]
Description=ipset persistent configuration
DefaultDependencies=no
Before=network.target
# ipset sets should be loaded before iptables
# Because creating iptables rules with names of non-existent sets is not possible and the iptables command will fail
Before=netfilter-persistent.service
ConditionFileNotEmpty=/etc/iptables/ipset.v4

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/ipset restore -file /etc/iptables/ipset
ExecStop=/sbin/ipset flush
ExecStopPost=/sbin/ipset destroy

[Install]
WantedBy=multi-user.target
RequiredBy=netfilter-persistent.service

Enable and start the services:

sudo chmod 644 /etc/systemd/system/ipset.service

# enable services (netfilter-persistent should already be enabled)
sudo systemctl enable ipset.service
sudo systemctl enable netfilter-persistent.service

# start services
sudo service ipset start
sudo service netfilter-persistent force-reload

Add iptables rules or ipsets

Additional satellites for the existing sets can be added by adding another line starting with ´-A -! <set-name> <ip>/<subnet>`.

Create additional ipsets for new environments with icinga2 satellites and don’t forget to add the set name to ALL-SATELLITES.

Reload / restart the ipset and iptables services to apply changes on the ipset or iptables rules files.

Check applied configuration

Finally, you can check the applied configuration by running sudo iptables-save or sudo iptables -L and sudo ipset -L.

Summary

Congratulations, now you have a very clean structured firewall setup. Use comments on iptables rules and choose the ipset names wisely. A clean structure allows you to modify the existing configuration easily and minimizes the risk for failures. Another advantage is that it’s easy to modify the configuration with your configuration management tools. :slight_smile:

6 Likes