New zones not showing on icingaweb2

[playfullwlr]

Hi everyone,

We finished our icinga2 setup (on docker, HA, 2 satellites per zones). I began ramping up the zone installation, after installing 1 zone alone (that worked fine, showing up in icingaweb2 without issues). I installed 6 zones at once, with the same configuration (everything has been puppeted, from install to zone deployment, not using the puppet module though) (all futures zones/satellite endpoints in the master zones.conf has been written on the file ahead of time so I only have to install icinga on the targetted satellites, I can provide the file after anonymization)

EDIT : I forgot to add, that we are using a topdown configuration here (2 masters, 2 sats per zones executing tasks and probes)

My problem is : Although eveyrthing looks normal on the icinga2 log (and icingadb too), I CANNOT see my newly added hosts on icingaweb2 (except for the zone I installed alone)

Log from master-1, while setting up icinga2 on my satellite with the command icinga2 node setup --ticket $ticket --cn ``infra-1.zone2.example.com`` --endpoint ``icinga-master-1.example.com`` --zone ``infra-1.zone2.example.com`` --parent_zone master --parent_host ``icinga-master-1.example.com`` --trustedcert /var/lib/icinga2/certs/trusted-parent.crt --accept-commands --accept-config:

[2025-12-04 13:32:24 +0000] information/ApiListener: New client connection from [X.X.X.X]:32898 (no client certificate)

[2025-12-04 13:32:24 +0000] information/ApiListener: No data received on new API connection from [X.X.X.X]:32898: End of file. Ensure that the remote endpoints are properly configured in a cluster setup.

[2025-12-04 13:32:25 +0000] information/ApiListener: New client connection for identity ‘infra-1.zone2.example.com’ from [X.X.X.X]:32908 (certificate validation failed: code 18: self-signed certificate)

[2025-12-04 13:32:25 +0000] information/JsonRpcConnection: Received certificate request for CN ‘infra-1.zone2.example.com’ which couldn’t be verified: self-signed certificate (code 18)

[2025-12-04 13:32:25 +0000] information/JsonRpcConnection: Sending certificate response for CN ‘infra-1.zone2.example.com’ to endpoint ‘infra-1.zone2.example.com’ (auto-signing ticket).

[2025-12-04 13:32:25 +0000] information/JsonRpcConnection: API client disconnected for identity ‘infra-1.zone2.example.com’

[2025-12-04 13:32:28 +0000] information/ApiListener: New client connection from [Y.Y.Y.Y]:yyyyy (no client certificate)

[2025-12-04 13:32:28 +0000] information/ApiListener: No data received on new API connection from [Y.Y.Y.Y]:yyyyy: End of file. Ensure that the remote endpoints are properly configured in a cluster setup.

[2025-12-04 13:32:31 +0000] information/ApiListener: New client connection for identity ‘infra-2.example.com’ from [Y.Y.Y.Y]:yyyyy (certificate validation failed: code 18: self-signed certificate)

[2025-12-04 13:32:31 +0000] information/JsonRpcConnection: Received certificate request for CN ‘infra-2.zone2.example.com’ which couldn’t be verified: self-signed certificate (code 18)

[2025-12-04 13:32:31 +0000] information/JsonRpcConnection: Sending certificate response for CN ‘infra-2.zone2.example.com’ to endpoint ‘infra-2.zone2.example.com’ (auto-signing ticket).

[2025-12-04 13:32:31 +0000] information/JsonRpcConnection: API client disconnected for identity ‘infra-2.zone2.example.com’

Here is the log from one of the sats, while booting the application:

[2025-12-04 14:47:48 +0000] information/FileLogger: ‘main-log’ started.

[2025-12-04 14:47:48 +0000] information/ApiListener: ‘api’ started.

[2025-12-04 14:47:48 +0000] information/ApiListener: Started new listener on ‘[54.38.42.191]:5665’

[2025-12-04 14:47:48 +0000] information/ApiListener: Reconnecting to endpoint ‘icinga-master-2.example.com’ via host ‘X.X.X.X’ and port ‘5665’

[2025-12-04 14:47:48 +0000] information/ApiListener: Reconnecting to endpoint ‘icinga-master-1.example.com’ via host ‘X.X.X.X’ and port ‘5665’

[2025-12-04 14:47:48 +0000] information/IcingaDB: ‘icingadb’ started.

[2025-12-04 14:47:48 +0000] information/CheckerComponent: ‘checker’ started.

[2025-12-04 14:47:48 +0000] information/IcingaDB: Trying to connect to Redis server (async) on host ‘redis:6379’

[2025-12-04 14:47:48 +0000] information/ConfigItem: Activated all objects.

[2025-12-04 14:47:48 +0000] information/IcingaDB: Connected to Redis server

[2025-12-04 14:47:48 +0000] information/IcingaDB: Starting initial config/status dump

[2025-12-04 14:47:48 +0000] information/ApiListener: New client connection for identity ‘icinga-master-1.example.com’ to [X.X.X.X]:5665

[2025-12-04 14:47:48 +0000] information/JsonRpcConnection: Requesting new certificate for this Icinga instance from endpoint ‘icinga-master-1.example.com’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Finished reconnecting to endpoint ‘icinga-master-1.example.com’ via host ‘X.X.X.X’ and port ‘5665’

[2025-12-04 14:47:48 +0000] information/ApiListener: Sending config updates for endpoint ‘icinga-master-1.example.com’ in zone ‘master’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Finished sending config file updates for endpoint ‘icinga-master-1.example.com’ in zone ‘master’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Syncing runtime objects to endpoint ‘icinga-master-1.example.com’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Finished syncing runtime objects to endpoint ‘icinga-master-1.example.com’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Finished sending runtime config updates for endpoint ‘icinga-master-1.example.com’ in zone ‘master’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Sending replay log for endpoint ‘icinga-master-1.example.com’ in zone ‘master’.

[2025-12-04 14:47:48 +0000] information/ApiListener: Replayed 146 messages.

This does not end here but idk if it is useful. I can fiund the same log on the satellites from the zone that is showing up rpoperly on icingaweb2

I did not checked the other new satellites, but i think we have the same outcome there.

How can I troubleshoot this? it seems that the zones and their sats are correctly configured and can correctly communicate with the master-1, but nothing (of these new zones) is showing up on icingaweb2.

on web2, when forcing the check using the url https://mon.example.com/icingaweb2/dashboard#!/icingaweb2/icingadb/service?name=ping4&host.name=new-sat.example.com, I have a “service not found” :

Dec 04 14:59:05 ``icinga-master-1.example.com`` docker-compose[3710246]: icingaweb_1 | [Thu Dec 04 14:59:05.456367 2025] [php:notice] [pid 48:tid 48] [client 127.0.0.1:60470] icingaweb2: ERROR - Icinga\\Exception\\NotFoundError in /usr/share/icingaweb2/modules/icingadb/application/controllers/ServiceController.php:76 with message: Service not found\n#0 /usr/share/icingaweb2/library/Icinga/Web/Controller/ActionController.php(181): Icinga\\Module\\Icingadb\\Controllers\\ServiceController->init()\n#1 /usr/share/icingaweb2/library/Icinga/Web/Controller/Dispatcher.php(59): Icinga\\Web\\Controller\\ActionController->__construct()\n#2 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Front.php(954): Icinga\\Web\\Controller\\Dispatcher->dispatch()\n#3 /usr/share/icingaweb2/library/Icinga/Application/Web.php(294): Zend_Controller_Front->dispatch()\n#4 /usr/share/icingaweb2/library/Icinga/Application/webrouter.php(105): Icinga\\Application\\Web->dispatch()\n#5 /usr/share/icingaweb2/public/index.php(4): require_once(String)\n#6 {main}, referer: ``https://mon.example.com/icingaweb2/dashboard

If you have an idea (or if i missed something trivial)…
Feel free to tell me if you need some more logs/configs.

Have a nice day!

• Version used (icinga2 --version) : 2.15.1
• Operating System and version : Debian 12 - Docker 28.5.1
• Enabled features (icinga2 feature list) : api checker icingadb mainlog notification opsgenie syslog
• Icinga Web 2 version and modules (System - About) 2.16.6
• Config validation (icinga2 daemon -C)
  [2025-12-04 14:30:29 +0000] information/cli: Icinga application loader (version: v2.15.1)
[2025-12-04 14:30:29 +0000] information/cli: Loading configuration file(s).
[2025-12-04 14:30:29 +0000] information/ConfigItem: Committing config item(s).
[2025-12-04 14:30:29 +0000] information/ApiListener: My API identity: ``icinga-master-1.exemple.com
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 SyslogLogger.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 User.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 ServiceGroup.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 2981 Services.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 2 ScheduledDowntimes.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 136 Zones.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 12 Notifications.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1452 Hosts.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 135 HostGroups.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 2 Downtimes.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 266 Endpoints.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 2 ApiUsers.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 252 CheckCommands.
[2025-12-04 14:30:29 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2025-12-04 14:30:29 +0000] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2025-12-04 14:30:29 +0000] information/cli: Finished validating the configuration file(s).

[lorenz (LorenzK)]

Hi @playfullwlr ,
First things first, could you revisit your post above and maybe fix the formatting? That would make it easier to it

But, regarding your problem: From the daemon sanity check ( icinga2 daemon -C ) I can read, that there are indeed a lot of Zone s, that’s good. But maybe the configuration of the satellite Zone s is slightly of.

Could you perhaps show us the configuration of the satellite Zone objects on the master nodes? It should look similar to:

object Zone “satellite” {
  endpoints = [ “sat1” ]
  parent = “master”
}

parent in this case should be the top level Zone (the one the master nodes are in).
I am guessing right now, that this attribute might be missing in your configuration.

[playfullwlr]

Post has been re-formatted, let me know if it’s still not good
sure i can show you part of my zones.conf on the master (since there is more than 100 zones)
Everything has been anonymised obviously. Zone1 represents the zone that i installed solo (and that works), zone2 represents the new zone that’s not showing on web2

icinga-master-1.example.com ~ # cat /data/icinga/data/etc/icinga2/zones.conf
/*

Generated by puppet

on 2025-12-04 14:36:44 +0000
*/

object Endpoint “icinga-master-1.example.com” {
}

object Endpoint “icinga-master-2.example.com” {
}

[…]

object Endpoint “infra-1.zone1.example.com” {
}

object Endpoint “infra-2.zone1.example.com” {
}

[…]

object Endpoint “infra-1.zone2.example.com” {
}

object Endpoint “infra-2.zone2.example.com” {
}

[…]

object Zone “zone1.example.com” {
  endpoints = [ “infra-1.zone1.example.com”, “infra-2.zone1.example.com” ]
  parent = “master”
}

[…]

object Zone “zone2.example.com” {
  endpoints = [ “infra-1.zone2.example.com”, “infra-2.zone2.example.com” ]
  parent = “master”
}

[…]

object Zone “global-templates” {
global = true
}

object Zone “global-commands” {
global = true
}

object Zone “director-global” {
global = true
}

[playfullwlr]

I have checked the icingadb db (psql), only my first zone (installed solo) is showed under the table “zone”, i don’t see the other zones i bulk-installed:

icingadb=> select name, name_ci, is_global, depth from zone;

        name         |       name_ci       | is_global | depth

---------------------+---------------------+-----------+-------

 global-commands     | global-commands     | y         |     0

 global-templates    | global-templates    | y         |     0

 director-global     | director-global     | y         |     0

 master              | master              | n         |     0

 zone1.example.com   | zone1.example.com   | n         |     1

[lorenz (LorenzK)]

hm, it does look right.
Could you also add the zone configuration from one of the “failing” sats? /etc/icinga2/constants.conf and /etc/icinga2/zones.conf should do it.

Additionaly: Is there alreaday at least one Host in one the “failing” Zones? If not, they might not show up after all, if there are no objects in it.
The Host object for the satellite should actually be in the master Zone (to control the monitoring there from further up in the hierarchy).

[playfullwlr]

root@infra-1:~# cat /data/icinga/data/etc/icinga2/zones.conf
/*

Generated by puppet

on 2025-12-04 14:37:09 +0000
*/

object Endpoint “icinga-master-1.example.com” {
  host = “x.x.x.x”
}

object Endpoint “icinga-master-2.example.com” {
  host = “y.y.y.y”
} 

object Endpoint “infra-1.zone2.example.com” {
}

object Endpoint “infra-2.zone2.example.com” {
}

object Zone “master” {
  endpoints = [ “icinga-master-1.example.com”, “icinga-master-2.example.com” ]
}

object Zone “zone2.example.com” {
  endpoints = [ “infra-1.zone2.example.com”, “infra-2.zone2.example.com” ]
  parent = “master”
}

object Zone “global-templates” {
  global = true
}

object Zone “global-commands” {
  global = true
}

object Zone “director-global” {
  global = true
}

root@infra-1:~# cat /data/icinga/data/etc/icinga2/constants.conf
/**

This file defines global constants which can be used in

the other configuration files.
*/

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = “/usr/lib/nagios/plugins”

/* The directory which contains the Manubulon plugins.

Check the documentation, chapter “SNMP Manubulon Plugin Check Commands”, for details.
*/
const ManubulonPluginDir = “/usr/lib/nagios/plugins”

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.

Check the documentation, chapter “Plugins Contribution”, for details.
*/
const PluginContribDir = “/usr/lib/nagios/plugins”

/* Our local instance name. By default this is the server’s hostname as returned by hostname --fqdn.

This should be the common name from the API certificate.
*/
const NodeName = “infra-1.zone2.example.com”

/* Our local zone name. */
const ZoneName = “infra-1.zone2.example.com”

/* Secret key for remote node tickets */
const TicketSalt = “$some_tickets”

icinga@infra-1:/opt$ cat /var/lib/icinga2/api/zones/zone2.example.com/_etc/thot_hosts_zone2.conf
/*

Generated by puppet

on 2025-12-05 15:11:53 +0000
*/

object Host “infra-1.example.com” {
  import “Prod Template”
  display_name = “infra-1.example.com”
  address = “x.x.x.x”
  groups = [ “zone2.example.com” ]
}

object Host “infra-2.example.com” {
  import “Prod Template”
  display_name = “infra-2.example.com”
  address = “x.x.x.x”
  groups = [ “zone2.example.com” ]
}

object Host “master.example.com” {
  import “Prod Template”
  display_name = “master.example.com”
  address = “x.x.x.x”
  groups = [ “zone2.example.com” ]
}

icinga@icinga-master-1:/opt$ cat /etc/icinga2/zones.d/zone2.example.com/thot_hosts_zone2.conf
/*

Generated by puppet

on 2025-12-05 14:39:51 +0000
*/

object Host “infra-1.zone2.example.com” {
  import “Prod Template”
  display_name = “infra-1.zone2.example.com”
  address = “x.x.x.x”
  groups = [ “zone2.example.com” ]
}

object Host “infra-2.zone2.example.com” {
  import “Prod Template”
  display_name = “infra-2.zone2.example.com”
  address = “x.x.x.x”
  groups = [ “zone2.example.com” ]
}

object Host “master.zone2.example.com” {
  import “Prod Template”
  display_name = “master.zone2.example.com”
  address = “x.x.x.x”
  groups = [ “zone2.example.com” ]
}

I added the host file from the sat (from the topdown config synchronization, to show that the icinga-master-1 is giving the correct sat configuration)

[playfullwlr]

EDIT: I deleted the other post as it was irrelevant in our case
Ok I found the problem : after stopping all satellites, i saw on the db that my zones were back again.
After some other investigation, I was able to reproduce the issue, by stopping one master (master-1), letting the other takeover (master-2), start the first one (master-1, that will not takeover since master-2 is active), then stop the second one (master-2). Then i lost my zones in the db. I saw the health status on icingaweb with a random satellite as the main icinga2 endpoint, which is super weird, they’re all satellites in my zones.conf not masters…
So somehow, it goes deeper than just some network and psql issues. the “HA“ is electing a random satellite that only has its configuration (through top/down config sync)
Do you have any idea how we can prevent this? let me know if you need logs.

[rivad (Dominik)]

If your zones.conf on the master and satellites don’t contain any weird stuff, I would open a issue on GitHub.

[lorenz (LorenzK)]

At least one problem I found:

/* Our local zone name. */
const ZoneName = “infra-1.zone2.example.com”

in constants.conf should be master

[playfullwlr]

I will try this, but can you explain why it should be master instead of “zone2.example.com” please?

[rivad (Dominik)]

Why? If I read thinks right, this constants.conf is on an agent not on a master or satellite.

[lorenz (LorenzK)]

@rivad Ha, possibly I was confused here. I guess I thought infra-1 would be one of the master nodes.

[playfullwlr]

I looked at the master-2, found that zonename was empty, fixed it. I will try to replicate the issue today

[playfullwlr]

well fixing the zonename in constants.conf does seem to fix the random election issue, it’s working now. Shutting down the 2 masters threw an error (icingadb is not running and writing to the database) in icingaweb as expected (a random satellite still become the active endpoint but the whole configuration stays, as expected)
I will do some more testing before confirming this was the issue (let’s be honest I think it was, since HA is enabled but no other node was assigned to the master zonename), i will get back to you asap

thank you very much folks!

[playfullwlr]

Well I forgot to relaunch the satellites, the behavior is still the same with all satellites running. Everytime i shutdown a master (icingadb container), another random infra takes the lead. I will open a bug on github i think. please keep the issue opened so i can update the situation here for everyone to see

thanks

[playfullwlr]

The github issue was created => Ha not working properly - Satellite elected as master instead of the real master · Issue #10680 · Icinga/icinga2 · GitHub

[Pooh (Antony Stone)]

I’m puzzled at the occurrence of “icinga-master-1.exemple.com” (note the
spelling) in the output of “icinga2 daemon -C”.

Is this a strange copy/paste error, or does it indicate a genuine typo
somewhere in your configuration?

Antony.

[playfullwlr]

yes it’s a typo i made (exemple is example in french, kinda use to write this typo in english sorry)

• i had to anonymize everything so sorry for the typos (tho there should not be much of them, i tried to be as accurate as possible so everyone can understand the case

[playfullwlr]

Hi all,

Final solution (was definitely logical, but not super evident to think about)
Satellites don’t need icingadb (so they have no way of overwriting the db thus screwing up the HA)

Ha not working properly - Satellite elected as master instead of the real master