Master supervised by another master

max13 · June 5, 2024, 10:08am

Hello,

I have 2 separated network (call them A and B)
On each one I have an icinga infrastructure for check each network’s hosts.
No problem with that. Here the complicated things, I want to check the master B with the master A …
I know that on one icinga instance it’s not possible but is there a way to run 2 icinga instance on master B (one for check his hosts, the other to be checked itself by master A) ?
It can be fully separated service on the linux OS listen on a different port but I don’t know how to tell master A to use this custom port for this specific host …
Here a draw to explain better :

Thanks for advance

rsx · June 5, 2024, 11:11am

Yes, you could add a second icinga instance as agent by adapting many settings (haven’t being trying though).

What about using icinga’s Rest API instead? Create checks on masters A quering icinga on master B?

max13 · June 5, 2024, 11:58am

Why not, I did not think about it.
My first intention is to keep the same checks I already use for all of my hosts, and it’s work with agent.
The installation of the second agent instance does not worry me but i realy no idea where I can tell master A to check master B with the custom port of the second instance

jeanm · June 5, 2024, 12:12pm

Hello,

How “separated” are the two networks? Do you mean that you have one or more network firewalls between them, or that you have no connectivity at all between them?

Cheers,

Jean

max13 · June 5, 2024, 12:18pm

I have one or more firewall between them

rsx · June 5, 2024, 1:45pm

You could let the agent on master B to initiate the connection to master A, there will then be nothing special with different ports etc.

jeanm · June 5, 2024, 1:51pm

I would consider moving the masters up one level (leaving them in network A, or moving them to network M, your choice), and transforming the 3 current masters in A and B to satellites.

Hosts in A would belong to zone A and have as parents satellites in zone A
Hosts in B would belong to zone B and have as parents satellites in zone B
Satellites in all zones would have as parents the new masters in zone M
Master servers in zone M would be in master zone and poll the satellites

Perhaps this is a bit over the top in your setup if you don’t have many hosts, but it is quite clear and scalable.

max13 · June 5, 2024, 2:01pm

unless I’m mistaken I can’t configure icinga2 to be master for his child AND send check result to an other master. In my understanding the master is at the top of the organisation, i’m I wrong ?

max13 · June 5, 2024, 2:02pm

unfortunately I can’t do that for organisation reason I have to keep the two network separated

rsx · June 5, 2024, 2:12pm

You will have two instances, one is master B and the other is an agent. Both don’t about each other. You will have a parent for the agent which is master A.

Correct.

max13 · June 5, 2024, 2:26pm

oh yes, sorry i miss understand what you say. That’s true, I can juste let the passive check enable for this host on master A, it’s probably the easier way.
For now I try to make the second instance awake (not so easy finally^^)

rsx · June 5, 2024, 3:00pm

Don’t mind.

That’s not necessary since the agent on master B acts as any agent in network A.

max13 · June 6, 2024, 10:16am

So I successfully create the second instance witch is running on master B with agent configuration to point on master A.
On master A side, I put host and services only on passive check and I waited…
Some service became OK but some other don’t … I don’t understand why …

max13 · June 7, 2024, 10:24am

It was a configuration problem on the second instance. It’s now fully functional

Here the step to reproduce if someone else want to do the same thing :

Build mandatory directories

cp -rp /etc/icinga2 /etc/icinga2-corp
rm features-enabled/icingadb.conf
rm features-available/icingadb.conf
cp -p /etc/default/icinga2 /etc/default/icinga2-corp
mkdir -p /var/log/icinga2-corp /var/lib/icinga2-corp/certs /var/run/icinga2-corp /var/cache/icinga2-corp /var/spool/icinga2-corp
chown -R nagios:nagios /var/lib/icinga2-corp /var/log/icinga2-corp /etc/icinga2-corp /var/cache/icinga2-corp /var/spool/icinga2-corp
chown nagios:www-data /var/run/icinga2-corp

Generate new certificates

sudo -u nagios icinga2 pki new-cert --cn [[host_fqdn]] \
--key /var/lib/icinga2-corp/certs/[[host_fqdn]].key \
--cert /var/lib/icinga2-corp/certs/[[host_fqdn]].crt

sudo -u nagios icinga2 pki save-cert \
--trustedcert /var/lib/icinga2-corp/certs/master.crt \
--host [[master_fqdn]] \
--port 5665 \
--key local.key \
--cert local.crt

Second daemon configuration

sudo -u nagios icinga2 node wizard -D LogDir=/var/log/icinga2-corp -D DataDir=/var/lib/icinga2-corp -D CacheDir=/var/cache/icinga2-corp -D SpoolDirDir=/var/run/icinga2-corp -D ZonesDir=/etc/icinga2-corp/zones.d -D ConfigDir=/etc/icinga2-corp
Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]:

Starting the Agent/Satellite setup routine...

Please specify the common name (CN) [[[host_fqdn]]]:

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): [[master1_fqdn]]

Do you want to establish a connection to the parent node from this node? [Y/n]:
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): [[master1_fqdn]]
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]: y
Master/Satellite Common Name (CN from your master/satellite node): [[master2_fqdn]]

Do you want to establish a connection to the parent node from this node? [Y/n]: y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): [[master2_fqdn]]
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]: n
Parent certificate information:

 Version:             3
 Subject:             CN = [[master2_fqdn]]
 Issuer:              CN = Icinga CA
 Valid From:          Nov 15 14:17:16 2023 GMT
 Valid Until:         Dec 16 14:17:16 2024 GMT
 Serial:              

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   [[master2_fqdn]]
 Fingerprint:         

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn '[[host_fqdn]]'):

No ticket was specified. Please approve the certificate signing request manually
on the master (see 'icinga2 ca list' and 'icinga2 ca sign --help' for details).
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []: 5664

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.

Local zone name [[[host_fqdn]]]:
Parent zone name [master]:

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]:

Do you want to disable the inclusion of the conf.d directory [Y/n]:
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

Sign CSR on master

root@master2(~) : icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
[[fingerprint]]                                                  | Jun  6 13:21:23 2024 GMT |        | CN = [[host_fqdn]]
root@master2(~) : icinga2 ca sign [[fingerprint]]
information/cli: Signed certificate for 'CN = [[host_fqdn]]'.

Create systemd service
Create the file /lib/systemd/system/icinga2-corp.service and paste

[Unit]
Description=Icinga host/service/network monitoring system
Requires=network-online.target
After=syslog.target network-online.target icingadb-redis.service postgresql.service mariadb.service carbon-cache.service carbon-relay.service

[Service]
Type=notify
NotifyAccess=all
Environment="ICINGA2_ERROR_LOG=/var/log/icinga2-corp/error.log"
EnvironmentFile=/etc/default/icinga2-corp
ExecStartPre=/usr/lib/icinga2/prepare-dirs /etc/default/icinga2-corp
ExecStart=/usr/sbin/icinga2 daemon -c /etc/icinga2-corp/icinga2.conf --close-stdio -e ${ICINGA2_ERROR_LOG} -D LogDir=/var/log/icinga2-corp -D DataDir=/var/lib/icinga2-corp -D CacheDir=/var/cache/icinga2-corp -D SpoolDir=/var/spool/icinga2-corp -D InitRunDir=/var/run/icinga2-corp -D ZonesDir=/etc/icinga2-corp/zones.d
PIDFile=/run/icinga2-corp/icinga2.pid
ExecReload=/usr/lib/icinga2/safe-reload /etc/default/icinga2-corp
TimeoutStartSec=30m
KillMode=mixed

# Systemd >228 enforces a lower process number for services.
# Depending on the distribution and Systemd version, this must
# be explicitly raised. Packages will set the needed values
# into /etc/systemd/system/icinga2.service.d/limits.conf
#
# Please check the troubleshooting documentation for further details.
# The values below can be used as examples for customized service files.

#TasksMax=infinity
#LimitNPROC=62883

[Install]
WantedBy=multi-user.target

Start service

systemctl start icinga2-corp.service

lorenz · June 10, 2024, 3:59pm

Thank you for the extented description, I might need that at some point

NiclasE · June 11, 2024, 11:18am

I’m in a similar scenario, but with just Master A and one Master B and would like Master B have an agent to send to Master A (one direction only). Is it just then to run the icinga node setup on Master B as I would on a normal agent and point to the Master A ?

max13 · June 11, 2024, 12:04pm

yes it’s just this BUT on the second instance of icinga service because if you run it on the main instance it will erase the master configuration and make your supervision down

NiclasE · June 11, 2024, 12:06pm

Not sure how you mean by second instance? Should I create a second instance on the same server as Master B, and if so, how?

max13 · June 11, 2024, 12:10pm

do you see here Master supervised by another master - #14 by max13

NiclasE · June 11, 2024, 12:16pm

Aha… ok I’ll give it a try and see… Thanks