Clustering issues and apparently random assignment of Recurring Downtimes

rsturm · November 20, 2019, 1:21pm

Moin,
when I set a recurring downtime for multiple hosts, the downtime objects are only created for a part of them.
Downtime Configuration:

  author = "rsturm"
  comment = "Test-Wartungsfenster 2"
  ranges = {
    monday = "02:00-05:00"
    tuesday = "02:00-05:00"
    wednesday = "02:00-05:00"
    thursday = "02:00-05:00"
    friday = "02:00-05:00"
    saturday = "02:00-05:00"
    sunday = "02:00-05:00"
  }
  assign where match ("termsmxenwk*",host.name)
  //assign where "citrix-worker" in host.groups
}

Both rules above should match 60 Hosts and when I run icinga2 daemon --validate, 60 Downtimes are instantiated (no matter which assign rule I use)

However, only 29 of the machines get a downtime object:

icinga2 object list --type downtime|grep ^Object|wc -l
29

The same happens when I set the scheduled Downtime via Icinga2 Director.

The same applies for Downtimes on Services, Icinga seems to pick about 50% of the services and create a downtime object for it, again completely random, on the same host there are some services in downtime, others are not, resulting in full mailboxes and grumpy co-workers.

About my setup:
Master-Nodes are on Icinga2 2.10-8 because of clustering Issues in 2.11
Affected Clients are Windows Server 2012R2/2016 with Icinga2 2.10.7-1

Any ideas what is wrong here?

(If this is a known bug in 2.10, please tell me and I will patiently wait for 2.12)

Thanks in advance

anon66228339 · November 20, 2019, 1:26pm

Which clustering issues? The only thing is you have to correct your zone configuration, see upgrade docs about that.

rsturm · November 20, 2019, 1:37pm

The communication between satellite servers and the masters broke down when I upgraded, checkresults from the satellite zone didn’t get through if the satellite sent them to node #2 because node #2 didn’t know any of the clients in these zones.

At the moment I simply refuse to create zones and endpoints for each client in the config files, as this should be done by the director (in my opinion)
As far as I understood Thomas Gelf’s comments on github, this should be fixed in 2.12, so I am waiting.

anon66228339 · November 20, 2019, 1:44pm

Its not an issue/bug. You only need to define the zones for master and satellites in configuration files. Endpoints can still be done via director. In my opinion, zones for satellites should never be created by director.

For your problem, since its a recurring downtime, you need to check them with icinga2 object list --type ScheduledDowntime

rsturm · November 20, 2019, 2:00pm

Okay, then I misunderstood that… I’ll update my configs and try an upgrade to 2.12

Regarding the scheduled Downtimes: icinga2 object list --type ScheduledDowntime does return 60 Objects.

rsturm · November 21, 2019, 9:02am

Ok, I did the upgrade to 2.11-2 with the help of Thomas Gelf’s post (https://github.com/Icinga/icinga2/issues/7542#issuecomment-535976245)
and ran into the same cluster-related issues as last time.

All host- and servicechecks from the satellite zones are ‘late’, including the checks on the satellites.

zones.conf of masters and satellites are here: https://pastebin.com/iqaKrAH6

Once I stop icinga2 on sysmon002, most checks work again, only the satellite servers stay in the ‘late’ state.

sysmon002 does not know anything about the hosts in the satellite zones:

[root@mgmtslsysmon001 ~]# icinga2 object list --type host|grep ^Object|wc -l
209
[root@mgmtslsysmon002 ~]# icinga2 object list --type host|grep ^Object|wc -l
187
[root@mgmtslsysmon002 ~]# icinga2 object list --type host --name "mgmtslsysmon*"|grep ^Object
Object 'mgmtslsysmon002.domain.tld' of type 'Host':
Object 'mgmtslsysmon001.domain.tld' of type 'Host':
[root@mgmtslsysmon002 ~]# icinga2 object list --type endpoint --name "mgmtslsysmon*"|grep ^Object
Object 'mgmtslsysmon002.domain.tld' of type 'Endpoint':
Object 'mgmtslsysmon001.domain.tld' of type 'Endpoint':
Object 'mgmtslsysmon003.domain.tld' of type 'Endpoint':
Object 'mgmtslsysmon004.domain.tld' of type 'Endpoint':
[root@mgmtslsysmon002 ~]# icinga2 object list --type zone --name "site-*"|grep ^Object
Object 'site-hal' of type 'Zone':
Object 'site-ba' of type 'Zone':
Object 'site-kln' of type 'Zone':

All of this worked in older Icinga versions.

anon66228339 · November 21, 2019, 9:32am

Zone looks good so far, you should only change the connection settings for satellites. Choose either your masters (site-kln i think) connecto to the satellites or the satellites connect to the masters.
I always let the satellites initiate connect to the masters, because masters never know if they are up an running.
Just remove the host = from the satellite endpoint objects in your masters zone.conf.

rsturm · November 21, 2019, 9:49am

I tried a top-down version before (masters connecting to satellites) which didn’t work as well.
I removed the host directive again from the parent endpoints on the satellites which made it even worse.

current status: sysmon001 running, sysmon002 off, all checks in satellite zones late.
tailf-ing the icinga2.log on the satellite servers tells me that the checks are executed.

Gonna try a bottom-up version…

rsturm · November 21, 2019, 10:04am

Update: Bottom up helped for a moment. a minute or two.
Checkresults came in right after a restart of icinga2 on all systems (ordered by name), only the checks on the satellites were still late.
After 2 minutes it stopped working again.
(both masters active at the moment)

My guess is, that the direction of communication doesn’t matter here. The Problem seems to be that sysmon002 (master2) ignores all host and service configuration of the satellite zones as if they weren’t his children.
Like described in a previous post, icinga2 object list shows no object from ‘site-ba’ and ‘site-hal’

anon66228339 · November 21, 2019, 10:15am

Where is your zones.conf on the masters?
If it is inside the master zone directory move it to /etc/icinga please.
Also provide information which OS you use on masters/satellites and how you installed icinga2 (from official repo or other).

rsturm · November 21, 2019, 10:25am

zones.conf are in /etc/icinga2 on all four systems.
Operating system is CentOS 7.7.1908, patchlevel of last week
Icinga packages are from http://packages.icinga.com/epel/7/release/, mirrored by our local redhat satellite server, synced last sunday morning (2019-11-17 03:00)

anon66228339 · November 21, 2019, 10:52am

Just to clarify, your second master does not receive any configuration from master1?

rsturm · November 21, 2019, 11:34am

It does receive and stage them but doesn’t apply them, I just found this line in the icinga2.log:
[2019-11-21 11:41:41 +0100] information/ApiListener: Received configuration updates (5) from endpoint 'mgmtslsysmon001.domain.tld' do not qualify for production, not triggering reload.

https://pastebin.com/DixJCzf3

anon66228339 · November 21, 2019, 11:40am

Hmm there is the problem. Did you recently delete a zone?
Maybe it helps if you stop icinga2 on master2 and delete everything out of the api dirs.

rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*

Also check New configuration does not trigger a reload, espacially the director part.

rsturm · November 21, 2019, 11:47am

I’ll check it in a moment.

I just did a deployment with changes in the master zone (site-kln), the secondary applies the configuration and after a reload the deleted hosts are gone from the object list, just as expected.
It seems that the secondary only ignores the satellite zones.

I removed everything from the api dirs, unfortunately it didn’t help. The late checkresults pile up again and master2 doesn’t know about the hosts in the satellite zones.

rsturm · November 21, 2019, 12:37pm

digging through the debuglogs right now.
It seems that the satellites don’t send any check results to master1 as long as master2 is alive.
On master one i can only see messages regarding event::heartbeat and event::SetLogPosition coming from the satellites
On master2 I see event::SetNextCheck and event::CheckResult coming from master1 and the satellites.
the debuglog on the satellite matches this behaviour, SetNextCheck and CheckResult are only sent to master2

Another line that might be interesting:

[2019-11-21 13:16:04 +0100] debug/ApiListener: Not connecting to Zone 'site-ba' because it's not in the same zone, a parent or a child zone.
[2019-11-21 13:16:04 +0100] debug/ApiListener: Not connecting to Zone 'site-hal' because it's not in the same zone, a parent or a child zone.

This happens on both masters.

Certificates and initial communication seem to be okay:
debug.log master2:

2019-11-21 13:01:18 +0100] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'mgmtslsysmon001.domain.tld'.
[2019-11-21 13:01:20 +0100] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'mgmtslsysmon004.domain.tld'.
[2019-11-21 13:01:20 +0100] notice/JsonRpcConnection: Received 'pki::RequestCertificate' message from identity 'mgmtslsysmon004.domain.tld'.
[2019-11-21 13:01:20 +0100] information/JsonRpcConnection: Received certificate request for CN 'mgmtslsysmon004.domain.tld' signed by our CA.
[2019-11-21 13:01:20 +0100] information/JsonRpcConnection: The certificate for CN 'mgmtslsysmon004.domain.tld' is valid and uptodate. Skipping automated renewal.
[2019-11-21 13:01:23 +0100] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'mgmtslsysmon003.domain.tld'.
[2019-11-21 13:01:23 +0100] notice/JsonRpcConnection: Received 'pki::RequestCertificate' message from identity 'mgmtslsysmon003.domain.tld'.
[2019-11-21 13:01:23 +0100] information/JsonRpcConnection: Received certificate request for CN 'mgmtslsysmon003.domain.tld' signed by our CA.
[2019-11-21 13:01:23 +0100] information/JsonRpcConnection: The certificate for CN 'mgmtslsysmon003.domain.tld' is valid and uptodate. Skipping automated renewal.
[2019-11-21 13:02:07 +0100] notice/JsonRpcConnection: Received 'icinga::Hello' message from identity 'mgmtslsysmon004.domain.tld'.
[2019-11-21 13:02:07 +0100] notice/JsonRpcConnection: Received 'pki::RequestCertificate' message from identity 'mgmtslsysmon004.domain.tld'.
[2019-11-21 13:02:07 +0100] information/JsonRpcConnection: Received certificate request for CN 'mgmtslsysmon004.domain.tld' signed by our CA.
[2019-11-21 13:02:07 +0100] information/JsonRpcConnection: The certificate for CN 'mgmtslsysmon004.domain.tld' is valid and uptodate. Skipping automated renewal.

rsturm · November 22, 2019, 3:06pm

Here we go again…
I found out that even the satellite didn’t get any new config for some time, the master didn’t talk to him at all. After rage-rm-rf’ing the satellite I reinstalled it, without result.

So I reinstalled the whole setup to test this in a clean environment. No director this time, no real clients, just plain icinga2 with dummy-hosts plus ido-pgsql and icingaweb2 for convenience

Systems:
mgmtslsysmon001 => master1, configuration master, ca
mgmtslsysmon002 => master2
mgmtslsysmon004 => satellite for zone site-hal
(sysmon003 is needed for other stuff, so no testing with him)

Operating System: CentOs 7.7.1908 from DVD, no patches yet.
Icinga2: 2.11.2-1 from packages.icinga.org

zones:
site-kln => master zone
site-hal => satellite zone

zones.conf’s in pastebin: https://pastebin.com/seRdAMBj

Master1 was created with the cli command icinga2 node setup --master --zone site-kln --accept-commands --disable-confd
Master2 was created with the node wizard, both master’s zones.conf edited
Satellite was created with the node wizard as well.

api accepts commands on every node, config on master2 and satellite

To test this, i placed 100 dummy hosts into each zone `object Host "dummy-$site-$i {check_command = “dummy” }

With this initial setup, none of the masters connected to the satellite

=> switched the connection direction, satellite connects to master and logs the connection including his try to update master1’s config

But as yesterday:
satellite and master2 receive no config.
satellite: no host objects in icinga object list, /var/lib/icinga2/api/zones empty, zones-state empty too, no config apply in the icinga2.log
master2: /var/lib/icinga2/api/zones contains only site-kln (it’s local zone) The satellite zone is missing. Same for zones-stage.

In the webinterface, all the dummy-kln go green, the dummy-hal stay pending, no matter how I set the connection direction.

=> wtf is (not) going on here? which step of the cluster config did I miss? It’s not my first cluster setup but the first which acts so strangely…

anon66228339 · November 22, 2019, 3:48pm

Selinux?
But iam just setting up my new develop environment and can tell you more tomorrow.

rsturm · November 22, 2019, 5:07pm

good point. Selinux is set to permissive, I didn’t reboot the masters yet, as it takes a while for them to come back. Port 5665 is enabled in firewalld, I even stopped firewalld for good measure.

One of the really weird things is, master2 receives, updates and applies the configuration for site-kln and participates in executing checks, but it ignores the child zone site-hal completely.

Status after reboots: still the same.
Status after updating all systems: still the same

I stop for today… gonna build the setup in my homelab tomorrow.

anon66228339 · November 22, 2019, 6:35pm

Quick test with 2 masters and a single satellite.
Did on icingama01 icinga2 node wizard.
then i created the tickets on icingama01 for icingama02 and satellite01 with

icinga2 pki ticket --cn 'icingama02'
icinga2 pki ticket --cn 'satellite01'

Run icinga2 node wizard on icingama02 and satellite .

Final /etc/icinga2/zones.conf on icingama0[1,2]

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-11-22 18:40:57 +0100
 */

object Endpoint "icingama01" {
        host = "192.168.178.233"
}

object Endpoint "icingama02" {
        host = "192.168.178.234"
}

object Zone "master" {
        endpoints = [ "icingama01", "icingama02" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}


#
## Zonen
#
object Endpoint "satellite01" {
}

object Zone "internet" {
        endpoints = [ "satellite01" ]
        parent = "master"
}

Restarted icinga2 on both master -> connection works

Final /etc/icinga2/zones.conf on satellite01

/*
 * Generated by Icinga 2 node setup commands
 * on 2019-11-22 18:48:51 +0100
 */

object Endpoint "icingama01" {
        host = "192.168.178.233"
        port = "5665"
}

object Endpoint "icingama02" {
        host = "192.168.178.234"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "icingama01", "icingama02" ]
}

object Endpoint "satellite01" {
}

object Zone "internet" {
        endpoints = [ "satellite01" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Connection works to both masters

Created zone directories on icingama01:

mkdir -p /etc/icinga2/zones.d/master/hosts 
mkdir -p /etc/icinga2/zones.d/internet/hosts
mkdir -p /etc/icinga2/zones.d/global-templates

Added test hosts to master zone

cat <<MASTER > /etc/icinga2/zones.d/master/hosts/test-hosts.conf
var globals.numhosts1 = 1

while (numhosts1 < 100) {
    object Host "dummy-" + numhosts1 use(numhosts1) {check_command = "dummy" }
    globals.numhosts1 += 1
}
MASTER

and for the internet zone:

cat <<INTERNET > /etc/icinga2/zones.d/internet/hosts/test-internet-hosts.conf
var globals.numhosts1 = 1

while (numhosts1 < 100) {
    object Host "dummy-internet-" + numhosts1 use(numhosts1) {check_command = "dummy" }
    globals.numhosts1 += 1
}
INTERNET

Triggered reload on icingama01 and after a few seconds both master have 198 host objects and the satellite01 has 99 host objects.

No matter how often i restart or trigger reload on icingama01, the configuration and number of host objects are on all masters and satellite like they should be.

From the logs i can see that icingama02 and satellite01 got a new configuration and then restarted. After restart they compare the configuration again, but this time the say “no change, no reload”

Log from satellite01 during configuration update

[2019-11-22 19:05:57 +0100] information/ApiListener: Reconnecting to endpoint 'icingama01' via host '192.168.178.233' and port '5665'
[2019-11-22 19:05:57 +0100] information/ApiListener: Reconnecting to endpoint 'icingama02' via host '192.168.178.234' and port '5665'
[2019-11-22 19:05:57 +0100] information/ApiListener: New client connection for identity 'icingama01' to [192.168.178.233]:5665
[2019-11-22 19:05:57 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icingama01'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Sending config updates for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icingama01'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icingama01'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Sending replay log for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished sending replay log for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished syncing endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished reconnecting to endpoint 'icingama01' via host '192.168.178.233' and port '5665'
[2019-11-22 19:05:57 +0100] information/ApiListener: Applying config update from endpoint 'icingama01' of zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Received configuration for zone 'internet' from endpoint 'icingama01'. Comparing the timestamp and checksums.
[2019-11-22 19:05:57 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/internet//_etc/hosts/hosts-internet-test.conf' for zone 'internet'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/internet' (295 Bytes).
[2019-11-22 19:05:57 +0100] information/ApiListener: Received configuration updates (1) from endpoint 'icingama01' are different to production, triggering validation and reload.
[2019-11-22 19:05:57 +0100] information/ApiListener: New client connection for identity 'icingama02' to [192.168.178.234]:5665
[2019-11-22 19:05:57 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icingama02'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Sending config updates for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icingama02'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icingama02'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Sending replay log for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished sending replay log for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished syncing endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Finished reconnecting to endpoint 'icingama02' via host '192.168.178.234' and port '5665'
[2019-11-22 19:05:57 +0100] information/ApiListener: Applying config update from endpoint 'icingama02' of zone 'master'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Received configuration for zone 'internet' from endpoint 'icingama02'. Comparing the timestamp and checksums.
[2019-11-22 19:05:57 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/internet//_etc/hosts/hosts-internet-test.conf' for zone 'internet'.
[2019-11-22 19:05:57 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/internet' (295 Bytes).
[2019-11-22 19:05:57 +0100] information/ApiListener: Received configuration updates (1) from endpoint 'icingama02' are different to production, triggering validation and reload.
[2019-11-22 19:05:57 +0100] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2019-11-22 19:05:57 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/internet' (295 Bytes).
[2019-11-22 19:05:57 +0100] information/ApiListener: Received configuration updates (1) from endpoint 'icingama02' are different to production, triggering validation and reload.
[2019-11-22 19:05:57 +0100] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2019-11-22 19:05:57 +0100] information/ApiListener: Copying file 'internet//.checksums' from config sync staging to production zones directory.
[2019-11-22 19:05:57 +0100] information/ApiListener: Copying file 'internet//.timestamp' from config sync staging to production zones directory.
[2019-11-22 19:05:57 +0100] information/ApiListener: Copying file 'internet//_etc/hosts/hosts-internet-test.conf' from config sync staging to production zones directory.
[2019-11-22 19:05:57 +0100] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2019-11-22 19:05:57 +0100] information/ApiListener: Copying file 'internet//.checksums' from config sync staging to production zones directory.
[2019-11-22 19:05:57 +0100] information/ApiListener: Copying file 'internet//.timestamp' from config sync staging to production zones directory.
[2019-11-22 19:05:57 +0100] information/ApiListener: Copying file 'internet//_etc/hosts/hosts-internet-test.conf' from config sync staging to production zones directory.
[2019-11-22 19:06:00 +0100] information/Application: Received request to shut down.
[2019-11-22 19:06:00 +0100] information/Application: Shutting down...
[2019-11-22 19:06:00 +0100] information/CheckerComponent: 'checker' stopped.
[2019-11-22 19:06:00 +0100] information/ApiListener: 'api' stopped.
[2019-11-22 19:06:00 +0100] information/FileLogger: 'main-log' started.
[2019-11-22 19:06:00 +0100] information/ApiListener: 'api' started.
[2019-11-22 19:06:00 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2019-11-22 19:06:00 +0100] information/CheckerComponent: 'checker' started.
[2019-11-22 19:06:00 +0100] information/ConfigItem: Activated all objects.
[2019-11-22 19:06:00 +0100] information/ApiListener: Reconnecting to endpoint 'icingama01' via host '192.168.178.233' and port '5665'
[2019-11-22 19:06:00 +0100] information/ApiListener: Reconnecting to endpoint 'icingama02' via host '192.168.178.234' and port '5665'
[2019-11-22 19:06:00 +0100] information/ApiListener: New client connection for identity 'icingama01' to [192.168.178.233]:5665
[2019-11-22 19:06:00 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icingama01'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Sending config updates for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icingama01'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icingama01'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Sending replay log for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Replayed 99 messages.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished sending replay log for endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished syncing endpoint 'icingama01' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished reconnecting to endpoint 'icingama01' via host '192.168.178.233' and port '5665'
[2019-11-22 19:06:00 +0100] information/ApiListener: Applying config update from endpoint 'icingama01' of zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Received configuration for zone 'internet' from endpoint 'icingama01'. Comparing the timestamp and checksums.
[2019-11-22 19:06:00 +0100] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/internet'. Current timestamp '2019-11-22 19:05:51 +0100' (1574445951.740264) >= received timestamp '2019-11-22 19:05:51 +0100' (1574445951.740264).
[2019-11-22 19:06:00 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/internet//_etc/hosts/hosts-internet-test.conf' for zone 'internet'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/internet' (295 Bytes).
[2019-11-22 19:06:00 +0100] information/ApiListener: Received configuration updates (1) from endpoint 'icingama01' do not qualify for production, not triggering reload.
[2019-11-22 19:06:00 +0100] information/ApiListener: New client connection for identity 'icingama02' to [192.168.178.234]:5665
[2019-11-22 19:06:00 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icingama02'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Sending config updates for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icingama02'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icingama02'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Sending replay log for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Replayed 99 messages.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished sending replay log for endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished syncing endpoint 'icingama02' in zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Finished reconnecting to endpoint 'icingama02' via host '192.168.178.234' and port '5665'
[2019-11-22 19:06:00 +0100] information/ApiListener: Applying config update from endpoint 'icingama02' of zone 'master'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Received configuration for zone 'internet' from endpoint 'icingama02'. Comparing the timestamp and checksums.
[2019-11-22 19:06:00 +0100] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/internet'. Current timestamp '2019-11-22 19:05:51 +0100' (1574445951.740264) >= received timestamp '2019-11-22 19:05:51 +0100' (1574445951.740264).
[2019-11-22 19:06:00 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/internet//_etc/hosts/hosts-internet-test.conf' for zone 'internet'.
[2019-11-22 19:06:00 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/internet' (295 Bytes).
[2019-11-22 19:06:00 +0100] information/ApiListener: Received configuration updates (1) from endpoint 'icingama02' do not qualify for production, not triggering reload.

Tomorrow i will add another zone with 2 satellites.