Multiple Icinga Agents on same

Tas-sos · September 15, 2021, 9:31pm

Hello Icinga community o/

Could I run multiple Icinga2 agent environments ?
Actually I want to run on the same node, two different icinga2 agent configurations. Is such a thing possible ?

I have two completely different Icigna 2 master setups ( eg: : two different organizations/companies ) and each one wants to do their own agent checks.

I tried the following…

Create CSR and Certificate for the node.

icinga2 pki new-cert --cn icinga-agent.example \
--key /var/lib/icinga2/certs/icinga-agent.example.key \
--cert /var/lib/icinga2/certs/icinga-agent.example.crt

Request the master certificate from the master host (icinga2-master1.example.com ) and store it as icinga2-master1.example.com

icinga2 pki save-cert \
--trustedcert /var/lib/icinga2/certs/icinga2-master1.example.com \
--host icinga2-master1.example.com

Node setup for master1

icinga2 node setup --ticket 954....3315 \
--cn icinga-agent.example \
--endpoint icinga2-master1.example.com \
--zone icinga-agent.example \
--parent_zone master \
--parent_host icinga2-master1.example.com \
--trustedcert /var/lib/icinga2/certs/icinga2-master1.example.com.crt \
--accept-commands --accept-config \
--disable-confd

systemctl restart icinga2

Now everything works perfect for master1.
But I also want another one Icinga2 master to monitor this box, so I do the following:

icinga2 pki save-cert \
--trustedcert /var/lib/icinga2/certs/icinga2-master2.example.com \
--host icinga2-master2.example.com

icinga2 node setup --ticket 123....456 \
--cn icinga-agent.example \
--endpoint icinga2-master2.example.com \
--zone icinga-agent.example \
--parent_zone master \
--parent_host icinga2-master2.example.com \
--trustedcert /var/lib/icinga2/certs/icinga2-master2.example.com.crt \
--accept-commands --accept-config \
--disable-confd

After that, this node only connected to icinga2-master2.example.com.
I want to send agent checks to both : icinga2-master1.example.com & icinga2-master2.example.com.

ls -alh /var/lib/icinga2/certs/
total 40K
drwxr-x--- 2 icinga icinga 4.0K Sep 16 00:25 .
drwxr-x--- 5 icinga icinga 4.0K Sep 16 00:26 ..
-rw-r--r-- 1 icinga icinga 1.8K Sep 15 22:56 icinga2-master1.example.com.crt
-rw-r--r-- 1 icinga icinga 1.7K Sep 16 00:25 ca.crt
-rw-r--r-- 1 icinga icinga 1.8K Sep 16 00:25 icinga-agent.example.crt
-rw-r--r-- 1 icinga icinga 1.8K Sep 15 22:55 icinga-agent.example.crt.orig
-rw------- 1 icinga icinga 3.2K Sep 16 00:25 icinga-agent.example.key
-rw------- 1 icinga icinga 3.2K Sep 15 22:55 icinga-agent.example.key.orig
-rw-r--r-- 1 icinga icinga 1.8K Sep 15 23:00 icinga2-master2.example.com.crt
-rw------- 1 icinga icinga   40 Sep 16 00:25 ticket

[2022-09-26 20:22:05 +0200] information/cli: Icinga application loader (version: v2.13.5)
[2022-09-26 20:22:05 +0200] information/cli: Loading configuration file(s).
[2022-09-26 20:22:05 +0200] critical/cli: Could not compile config files: Error: Function call ‘std::ifstream::open’ for file ‘c:\tmp/icinga2.conf’ failed with error code 2, ‘No such file or directory’

https://github.com/Icinga/icinga2/issues/6993

As soon as your are in a different icinga2.conf you are a different Node

But in this highly experimental scenario your api listener for different agents/ masters must be on different ports.
You must edit your startup service definition.

I didn’t go the distance but if someone wants to try I think this information will get you started.

davekempe · October 17, 2022, 4:52am

I had a crack at this and got it working fine. It requires a bunch of directories to be made, for the sake of the argument I have called my second instance icinga2-satellite.
Here is the systemd config file for it:

root@icinga-01:~# cat /etc/systemd/system/icinga2-satellite.service
[Unit]
Description=Icinga host/service/network monitoring system - Satellite
After=syslog.target network-online.target postgresql.service mariadb.service carbon-cache.service carbon-relay.service

[Service]
Type=notify
Environment="ICINGA2_ERROR_LOG=/var/log/icinga2-satellite/error.log"
EnvironmentFile=/etc/default/icinga2-satellite
ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2-satellite
ExecStart=/usr/sbin/icinga2 daemon --close-stdio -e ${ICINGA2_ERROR_LOG} -D ConfigDir=/etc/icinga2-satellite -D LogDir=/var/log/icinga2-satellite -D DataDir=/var/lib/icinga2-satellite -D CacheDir=/var/cache/icinga2-satellite -D SpoolDir=/var/spool/icinga2-satellite -D InitRunDir=/var/run/icinga2-satellite -D ZonesDir=/etc/icinga2-satellite/zones.d
PIDFile=/run/icinga2-satellite/icinga2.pid
ExecReload=/usr/lib/icinga2/safe-reload /etc/default/icinga2-satellite
TimeoutStartSec=30m

# Systemd >228 enforces a lower process number for services.
# Depending on the distribution and Systemd version, this must
# be explicitly raised. Packages will set the needed values
# into /etc/systemd/system/icinga2.service.d/limits.conf
#
# Please check the troubleshooting documentation for further details.
# The values below can be used as examples for customized service files.

#TasksMax=infinity
#LimitNPROC=62883

[Install]
WantedBy=multi-user.target

I needed to create a bunch of directories which are all mentioned in the ExecStart line, with the relevant permissions (mostly nagios:nagios, but also nagios:www-data).

Then it worked fine. I also use a bash alias to make the commands work better, you can put this at the end of your bashrc:

icinga2-satellitefn() {
/usr/sbin/icinga2 $@ -D ConfigDir=/etc/icinga2-satellite -D LogDir=/var/log/icinga2-satellite -D DataDir=/var/lib/icinga2-satellite -D CacheDir=/var/cache/icinga2-satellite -D SpoolDir=/var/spool/icinga2-satellite -D InitRunDir=/var/run/icinga2-satellite -D ZonesDir=/etc/icinga2-satellite/zones.d
}
alias icinga2-satellite=icinga2-satellitefn

Then I learnt what the minimum config for icinga to work correctly was, but adding the config files back in one by one:

root@icinga-01:~# find /etc/icinga2-satellite/
/etc/icinga2-satellite/
/etc/icinga2-satellite/constants.conf
/etc/icinga2-satellite/features-enabled
/etc/icinga2-satellite/features-enabled/mainlog.conf
/etc/icinga2-satellite/features-enabled/checker.conf
/etc/icinga2-satellite/features-enabled/api.conf
/etc/icinga2-satellite/icinga2.conf
/etc/icinga2-satellite/conf.d
/etc/icinga2-satellite/zones.conf

You need to adjust constants.conf to have a different hostname and you need to manually join the cluster:

 icinga2 pki new-cert --cn "icinga2-satellite" --key /var/lib/icinga2-satellite/certs/icinga2-satellite.key --cert /var/lib/icinga2-satellite/certs/icinga2-satellite.crt

icinga2 pki save-cert --trustedcert /var/lib/icinga2-satellite/certs/icinga-master.local.crt --host icinga-master.local --port 5665

Your zones.conf and things needs to be setup right, so I assume you know how to do all that.

and I had to adjust bind_port in the api feature as well of couse.
But once you do all that, it works just fine, the second instance sits in the separate zone, and does its thing.
In my case I needed to run a new set of checks and things against an old set of machines but from the new cluster, as changing firewall rules to help would have been too hard.

Its not something I would do every day, and I’m sure you could make this nicer with ansible, but it does work.
@lewgegg @otl

Multiple Icinga Agents on same

Related