Icingaweb2: Distributed HA-Cluster without Config-Management (I know, I know ... )

Situation

The Standard way of installing Icingaweb2 on two servers is to save the configuration-files on a Samba-Share and link to that share. This wasn’t possible in my environment as the only way to share Files via a “Share-Directory” is a mounted MooseFS-Share. My Tests showd that simply adapting the Samba-Principle to my MooseFS-Solution didn’t work out so I had to forge a another way of keeping the servers in sync.

Also: Setting up Puppet and do it the right way ( :wink: ) was out of scope of the project.

Prerequisites

  • Webservers are behind HA-Proxy that distributes with standard leastconn-Algorythm (looking on the final Build, that’s just an Information that basically will not matter)
  • Icingaweb will be served from two webservers which are running on nginx
  • icinga2 HA-Cluster is set up
  • both servers have the MooseFS share mounted (not part of this howto)

Conclusion what we will achieve

“Master-1” will serve Icingaweb and syncs its files constantly to the MooseFS-Share. “Master-2” constantly watches the webserver functionality of “Master-1” and syncs its config-files from the MooseFS-Share. The nginx-service on Master-2 is switched of.
If Master-1 fails, Master-2 recognizes and starts its webserver for as long as Master-1 is not reachable.

Let’s do it

Im using debian 9, keep that in mind

  • Install lsyncd on both machines
    apt-get install lsyncd
  • create file and folder structure
    mkdir -p /var/log/lsyncd/
    touch /var/log/lsyncd/{lsyncd.log,lsyncd.status}
    mkdir -p /etc/lsyncd
    touch /etc/lsyncd/lsyncd.conf.lua
  • write your configs:
    ++ lsyncd on master-1
    master-1$: vi /etc/lsyncd/lsyncd.conf.lua
settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync {
  default.rsync,
  source = "/usr/share/icingaweb2",
  target = "/path/to/moosefs/mount/usr/icingaweb2",
  rsync = {
    archive = true,
    owner = true,
    perms = true,
    group = true,
    compress = false,
    whole_file = true,
    temp_dir = "/path/to/moosefs/mount/master-1_temp",
  }
}
sync {
  default.rsync,
  source = "/etc/icingaweb2",
  target = "/path/to/moosefs/mount/etc/icingaweb2",
  rsync = {
   archive = true,
   owner = true,
   perms = true,
   group = true,
   compress = false,
   whole_file = true,
   temp_dir = "/path/to/moosefs/mount/master-1_temp",
  }
}
sync {
  default.rsync,
  source = "/usr/share/php/Icinga",
  target = "/path/to/moosefs/mount/usr/share/php/Icinga",
  rsync = {
    archive = true,
    owner = true,
    perms = true,
    group = true,
    compress = false,
    whole_file = true,
    temp_dir = "/path/to/moosefs/mount/master-1_temp",
  }
}
  • lsyncd-Config on master-2
    master-2:$ vi /etc/lsyncd/lsyncd.conf.lua
settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  statusFile = "/var/log/lsyncd/lsyncd.status"
}
sync {
  default.rsync,
  source = "/path/to/moosefs/mount/usr/icingaweb2",
  target = "/usr/share/icingaweb2",
  rsync = {
    archive = true,
    owner = true,
    perms = true,
    group = true,
    compress = false,
    whole_file = true,
    temp_dir = "/path/to/moosefs/mount/master-2_temp",
  }
}
sync {
  default.rsync,
  source = "/path/to/moosefs/mount/etc/icingaweb2",
  target = "/etc/icingaweb2",
  rsync = {
    archive = true,
    owner = true,
    perms = true,
    group = true,
    compress = false,
    whole_file = true,
    temp_dir = "/path/to/moosefs/mount/master-2_temp",
  }
}
sync {
  default.rsync,
  source = "/path/to/moosefs/mount/usr/share/php/Icinga",
  target = "/usr/share/php/Icinga",
  rsync = {
    archive = true,
    owner = true,
    perms = true,
    group = true,
    compress = false,
    whole_file = true,
    temp_dir = "/path/to/moosefs/mount/master-2_temp",
  }
}

Implementing the Server-Check on master-2

  • What we need:
    ++ a little bash-script that does the magic
    ++ a systemd-service
    ++ a systemd.timer

The bash-Script

#!/bin/bash
# File: /usr/local/bin/icinga-failover.sh 

curl -kIs https://10.10.10.42 | head -n 1 | grep -q Found
if [ $? -eq 0 ] ; then
   systemctl is-active --quiet nginx
   if [ $? -eq 0 ] ; then
   	systemctl stop nginx
   fi
else
   systemctl start nginx
fi

The Systemd-Service

# File/etc/systemd/system/icinga-failover.service
[Unit]
Description=Icinga Failover Check via curl

[Service]
ExecStart=/usr/local/bin/icinga-failover.sh

[Install]
WantedBy=multi-user.target

The systemd timer

# File /etc/systemd/system/icinga-failover.timer
[Unit]
Description=Icinga Failover Timer

[Timer]
OnBootSec=10s
OnUnitInactiveSec=1s

[Install]
WantedBy=basic.target

That’s basically it. Enable the service and the timer with the systemctl command and you should be good to go.

You can add /etc/icinga2/ to t he synced folders, too. Then you don’t have to manually sync notification-scripts (as an example).

That’s my first #howto, I hope it helps people and you liked it :smile:

1 Like