Service cannot be put in global zone

Forgot to mention to the Director config preview in the “resolved mode”.
Please also show the imported host template :slight_smile:

I’m not sure of what you asked me :sweat_smile: should i share templates.conf?

In the Director webinterface you have te option to show a resolved view of the configuration of a host/service.
This expands the configuration and resolves all imported templates and the variables they contain:
image

normal view:
image

resolved view:
image

immagine

This is the conf of the pending host

I cannot see the host i created manually here in the director, i can only see it in tactical overview
Maybe is an API problem?

That is correct as well.
The only config objects created outside the Director that are visible inside the Director are commands (and only if you ran the Kickstart after reloading your icinga2 service). They are then displayed (but not editable) here: /director/commands?type=external_object

Templates created outside the Director can be used inside the Director, but you can see them.
And Hosts and Services created outside of the Director are also not visible.

Regarding you pending host: have you checked that your satellite is acutally connected to your master and has the most recent config running?

Running the icinga check on the satellite will tell you if a config reload has failed.
Running the cluster-zone check on the master checking for your “satellite-bd” zone, will tell you if it is connected correctly.
Also check the /var/log/icinga2/icinga2.log or /var/lib/icinga2/api/zones-stage/startup.log(if this file is not present, it’s a good sign) for errors.

I’m pretty sure they are connected: i use the same satellite to check another device and the only difference between the two is:

the first (which works) is stored in /etc/icinga2/zones.d/satellite-bd/satellite-bd.conf
the second (which does not work) is stored in the default director writing folder

Please check :wink:

I don’t see why the host configured via the config files should work and the one via the Director doesn’t.
Both configs are basically the same.
Check if the satellite actually has the host object in /var/lib/icinga2/api/zones/... configs (most likely /var/lib/icinga2/api/zones/satellite-bd/director/hosts.conf. There you will find the current running config of the satellite.

This is what i have in the master at
/var/lib/icinga2/api/zones/satellite-bd/director/hosts.conf

object Host "provanic.blueday.it" {

          import "sat-ping"
          display_name = "provanic.blueday.it"
          address = "192.168.0.70"
}

And this is what i have in the satellite

object Host “niccolo-bd.blueday.it” {

  check_command = “hostalive”
  address = “192.168.0.69”
}

So this host is not in the hosts.conf file on the satellite in the /var/lib/icinga2/api/zones/... folder?
If that is so, then there is a problem with the config sync or config validation on the satellite.

Check, if you have a zones-stage folder on the satellite, containing a startup log file. That should show you errors during the config reload.
If there is no such folder/file, check the icinga2.log on both master and satellite for problem messages during a reload (deployment via Director or service restart via cli on the master)

Exactly, there is not object Host "provanic.blueday.it" but only object Host "niccolo-bd.blueday.it"
(which i wrote manually in the master /etc/icinga2/zones.d/satellite-bd/satellite-bd.conf)

i found /var/lib/icinga2/api/zones-stage/ but it only contains :
director-global global-templates satellite-bd

then I looked for /var/log/icinga2/icinga2.log in the master:

and this is the content:

[2023-02-20 13:57:50 +0100] critical/ApiListener: Cannot connect to host 'XXXXXXXX' on port '5665': No route to host
[2023-02-20 13:57:53 +0100] warning/PluginCheckTask: Check command for object '02710-ths-firewall-fortigate!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' (PID: 14630, argum$
[2023-02-20 13:57:54 +0100] warning/PluginCheckTask: Check command for object '01408-considi-firewall-fortigate!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' (PID: 14637, a$
[2023-02-20 13:57:57 +0100] information/ApiListener: Reconnecting to endpoint 'satellite-bd.blueday.it' via host '195.39.202.134' and port '5665'
[2023-02-20 13:57:59 +0100] warning/PluginCheckTask: Check command for object '00064-basso_viaggi-firewall-fortigate!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' (PID: 146$
[2023-02-20 13:57:59 +0100] warning/PluginCheckTask: Check command for object '00920-rifa-firewall-fortigate-schio!fortigate-snmp-interfaccia_rete-FTTC_VOIPVOICE-wan2' (PID: 14649$
[2023-02-20 13:58:00 +0100] critical/ApiListener: Cannot connect to host ''XXXXXXXX'' on port '5665': No route to host
[2023-02-20 13:58:02 +0100] warning/PluginCheckTask: Check command for object '02090-vepack-firewall-fortigate-grisignano!fortigate-snmp-interfaccia_rete-ADSL_TELECOM-wan1' (PID: $
[2023-02-20 13:58:03 +0100] warning/PluginCheckTask: Check command for object '01339-lucas-firewall-fortigate-chiuppano!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' (PID: $
[2023-02-20 13:58:05 +0100] warning/PluginCheckTask: Check command for object '01541-p63-firewall-fortigate!fortigate-snmp-interfaccia_rete-ADSL_WELCOME-wan' (PID: 14675, argument$
[2023-02-20 13:58:05 +0100] warning/PluginCheckTask: Check command for object '02592-jfd-firewall-fortigate-cavazzale!fortigate-snmp-interfaccia_rete-LAN_DMZ_EX_SAID-internal4' (P$
[2023-02-20 13:58:06 +0100] warning/PluginCheckTask: Check command for object '01749-cogeass-firewall-fortigate!fortigate-snmp-interfaccia_rete-FIBRA_WIND-wan' (PID: 14678, argume$
[2023-02-20 13:58:07 +0100] information/ApiListener: Reconnecting to endpoint 'satellite-bd.blueday.it' via host 'XXXXXXXX' and port '5665'
[2023-02-20 13:58:08 +0100] warning/PluginCheckTask: Check command for object '02710-ths-firewall-fortigate!fortigate-snmp-interfaccia_rete-ADSL_FASTWEB-wan1' (PID: 14679, argumen$
[2023-02-20 13:58:10 +0100] warning/PluginCheckTask: Check command for object '02780-tre_esse-firewall-fortigate!fortigate-snmp-interfaccia_rete-FIBRA_TIM-wan2' (PID: 14686, argum$
[2023-02-20 13:58:10 +0100] critical/ApiListener: Cannot connect to host 'XXXXXXXX' on port '5665': No route to host
[2023-02-20 13:58:12 +0100] warning/PluginCheckTask: Check command for object '01339-lucas-firewall-fortigate-chiuppano!fortigate-snmp-interfaccia_rete-FIBRA_TIM-wan2' (PID: 14693$
[2023-02-20 13:58:14 +0100] warning/PluginCheckTask: Check command for object '02371-laprimaplastic-firewall-fortigate!fortigate-snmp-interfaccia_rete-ADSL_TELECOM-wan' (PID: 1469$
[2023-02-20 13:58:15 +0100] warning/PluginCheckTask: Check command for object '01376-comal-firewall-fortigate-tripoli!fortigate-snmp-interfaccia_rete-FIBRA_TELECOM-wan1' (PID: 147$
[2023-02-20 13:58:16 +0100] warning/PluginCheckTask: Check command for object '01534-tecnopaper-firewall-fortigate-rosa!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' (PID: $
[2023-02-20 13:58:16 +0100] warning/PluginCheckTask: Check command for object '01713-tickets_maker-firewall-fortigate!fortigate-snmp-interfaccia_rete-ADSL_TIM-wan' (PID: 14769, ar$
[2023-02-20 13:58:16 +0100] warning/PluginCheckTask: Check command for object '02090-vepack-firewall-fortigate-padova!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' (PID: 14$
[2023-02-20 13:58:16 +0100] warning/PluginCheckTask: Check command for object '00254-frescura-firewall-fortigate-thiene!fortigate-snmp-interfaccia_rete-CASA_VINCENZO-internal6' (P$
[2023-02-20 13:58:16 +0100] warning/PluginCheckTask: Check command for object '00013-tna_cargo-firewall-fortigate-brendola!fortigate-snmp-interfaccia_rete-FIBRA_IPCOMPANY-port7' ($
[2023-02-20 13:58:17 +0100] information/ApiListener: Reconnecting to endpoint 'satellite-bd.blueday.it' via host '195.39.202.134' and port '5665'
[2023-02-20 13:58:17 +0100] warning/PluginCheckTask: Check command for object '00013-tna_cargo-firewall-fortigate-brendola!fortigate-snmp-interfaccia_rete-LAN_MAGAZZINO-port5' (PI$
[2023-02-20 13:58:19 +0100] warning/PluginCheckTask: Check command for object '01133-studio_rebecca-firewall-fortigate-schio!fortigate-snmp-interfaccia_rete-LAN_UFFICI-internal' ($
[2023-02-20 13:58:19 +0100] warning/Process: Terminating process 14837 ('/usr/lib/nagios/plugins/check_ping' '-H' '81.21.25.1' '-c' '5000,100%' '-w' '3000,80%') after timeout of 3$
[2023-02-20 13:58:19 +0100] warning/PluginCheckTask: Check command for object '01534-tecnopaper-linea_internet-vdsl-voipvoice' (PID: 14837, arguments: '/usr/lib/nagios/plugins/che$
[2023-02-20 13:58:20 +0100] warning/PluginCheckTask: Check command for object '00005-valle_autotrasporti-firewall-fortigate!fortigate-snmp-interfaccia_rete-FIBRA_INTERPLANET-wan1'$
[2023-02-20 13:58:20 +0100] critical/ApiListener: Cannot connect to host 'XXXXXXXX' on port '5665': No route to host
[2023-02-20 13:58:26 +0100] warning/PluginCheckTask: Check command for object '02090-vepack-firewall-fortigate-padova!fortigate-snmp-interfaccia_rete-ABITAZIONE_01-internal4' (PID$
[2023-02-20 13:58:27 +0100] warning/PluginCheckTask: Check command for object '00254-frescura-firewall-fortigate-thiene!fortigate-snmp-interfaccia_rete-ADSL_MAXFONE-internal5' (PI$
[2023-02-20 13:58:27 +0100] information/IdoMysqlConnection: Pending queries: 9 (Input: 8/s; Output: 8/s)
[2023-02-20 13:58:27 +0100] information/ApiListener: Reconnecting to endpoint 'satellite-bd.blueday.it' via host 'XXXXXXXX' and port '5665'
[2023-02-20 13:58:28 +0100] warning/PluginCheckTask: Check command for object '02172-zanuso-firewall-fortigate!fortigate-snmp-interfaccia_rete-FIBRA_TELECOM-wan1' (PID: 14866, arg$
[2023-02-20 13:58:30 +0100] critical/ApiListener: Cannot connect to host ''XXXXXXXX'' on port '5665': No route to host
[2023-02-20 13:58:31 +0100] warning/PluginCheckTask: Check command for object '01195-tvr-firewall-fortigate-schio!fortigate-snmp-interfaccia_rete-LAN_PROD-internal4' (PID: 14874, $
[2023-02-20 13:58:35 +0100] warning/PluginCheckTask: Check command for object '01339-lucas-firewall-fortigate-chiuppano!fortigate-snmp-interfaccia_rete-ADSL_TELEMAR-wan1' (PID: 14$
[2023-02-20 13:58:37 +0100] information/ApiListener: Reconnecting to endpoint 'satellite-bd.blueday.it' via host ''XXXXXXXX'' and port '5665'
[2023-02-20 13:58:40 +0100] critical/ApiListener: Cannot connect to host 'XXXXXXXX' on port '5665': No route to host
[2023-02-20 13:58:41 +0100] warning/PluginCheckTask: Check command for object '02816-bpk-firewall-fortigate-tezze_sul_brenta!fortigate-snmp-interfaccia_rete-ADSL_TRIVENET_WAN2' (P$
[2023-02-20 13:58:45 +0100] warning/PluginCheckTask: Check command for object '00280-sassi_editore-firewall-fortigate-san_vito_di_leguzzano!f

Looking at that critical error I think there is a problem with API, but I don’t know how to resolve it :frowning:
(i obfuscated ip addresses)

Exactly, there is not
object Host "provanic.blueday.it" but only
object Host "provanic.blueday.it"

That must be a typo. Please can you correct so we know what does or does not
exist?

Antony.

Ok now i corrected it, sorry

Looks like you have a network-related problem. “no route to host” could indicate various things:

  • the Master does not know how to reach the IP address of the satellite (maybe the master needs another network interface which is inside the VLAN of the satellite)
  • the icinga service on the satellite isn’t running
  • software firewall (like iptables or firewalld) blocking the communication (on either the satellite or the master)
  • hardware firewall blocking communication

Ok thank you, I’m going to check this problem, I’ll write you as soon as possible to keep you updated

I tried to resolve the issue:

I checked every tip you gave me with telnet i can obtain communication between master and satellite, i tried the connection
master → satellite
and satellite → master
Then i checked some more tips in the satellite and it showed this:

[2023-02-21 08:37:28 +0000] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2>
[2023-02-21 08:37:45 +0000] critical/IdoMysqlConnection: Error "Lost connection to MySQL server during query" when executing query "COMMIT;BEGIN"
[2023-02-21 08:37:45 +0000] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2023-02-21 08:37:50 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.15.1')
[2023-02-21 08:37:50 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga2' in 0.027385 second(s).
[2023-02-21 08:42:20 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 2/s; Output: 2/s)
[2023-02-21 08:42:20 +0000] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 4/5min 8/15min);
[2023-02-21 08:42:20 +0000] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2023-02-21 08:42:20 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2023-02-21 08:45:42 +0000] critical/ApiListener: Client TLS handshake failed (from [::ffff:192.168.0.254]:50758): wrong version number

Maybe could TLS be the problem?

and the content of zones-stage-startup-last-failed.log is:

[2023-02-21 14:21:07 +0000] information/cli: Icinga application loader (version: r2.13.5-1)
[2023-02-21 14:21:07 +0000] information/cli: Loading configuration file(s).
[2023-02-21 14:21:07 +0000] critical/config: Error: Object 'esva-snmp-disco' of type 'Service' re-defined: in /var/lib/icinga2/api/zones-stage//global-templates/director/service_templates.conf: 1:0-1:33; previous definition: in /var/lib/icinga2/api/zones-stage//director-global/director/service_templates.conf: 31:1-31:34
Location: in /var/lib/icinga2/api/zones-stage//global-templates/director/service_templates.conf: 1:0-1:33
/var/lib/icinga2/api/zones-stage//global-templates/director/service_templates.conf(1): template Service "esva-snmp-disco" {
                                                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//global-templates/director/service_templates.conf(2):     check_command = "bd_check_esva"
/var/lib/icinga2/api/zones-stage//global-templates/director/service_templates.conf(3):     max_check_attempts = "5"
[2023-02-21 14:21:07 +0000] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

I tried to manually connect with
openssl s_client -connect XXX.XXX.XXX.XXX:5665

and it seems accepting the connection

Have you purposely activated the ido-mysql feature on the satellite?
In general you only want the master(s) to write into the database.

Seems like, at some point you create the template esva-snmp-disco via the Director and pinned it to the zone director-global and then you changed that at some point to global-templates.
I would suggest you leave the zone option of the Director set to nothing in all cases, except when you create a host template with which you want to pin hosts to your satellite zone.

Try clearing the /var/lib/icinga2/api/[zones/zones-stage] directory and the restart the icinga2 service on the satellite. That should make it correctly start again and then receive the config from the master.

rm -rf /var/lib/icinga2/api/zones/*
rm -rf /var/lib/icinga2/api/zones-stage/*
systemctl restart icinga2

After this check the logs again for errors and the config files if your host is present.

No i did not, should I disable it?

I had to delete the wrongly redefined global services but now it works! thanks a lot!

Yes, I would say so. I wonder why it is enabled though.
Do you use the icinga2 node wizard to set up the satellite? That should normally disable not needed features when choosing a satellite setup.

Yes I used it, i dont know what to say :sweat_smile: