Problem with service visibility with multiple zones

Icinga 2
1

The setup is as follows :

i have a master node, it has “master” zone.

There are two satellites in “ABS” zone, that’s child of the “master” zone.

There is a host in “ABS” zone, under which i register services via api. If i use API on master node, the service appears in icingaweb2, but the satellites do not notify about it.

If i do it on nodes in “ABS” zone, the service is not visible in icingaweb2, but i get notifications. What would be the ideal way to resolve this ?

Also, the service does not list in icinga2 object list, on master node or on satellites. It can only be seen on one of those.

I assume that maybe i have invalid feature list on either node. But that would not explain why certain services are not visible on either end.

Current situation is :

# on satellite in "ABS": 
icinga:/var/log/icinga2/notification# icinga2 object list -t service -n "prometheus*" | grep -i object
Object 'prometheus!Minio_kub04_disk_space_pct_6631fe2a5d60a192' of type 'Service':
Object 'prometheus!FailedReload_a7c9fddf5ebb4bf8' of type 'Service':
Object 'prometheus!Minio_kub04_disk_space_percent_40be853ad3fa0bf8' of type 'Service':
Object 'prometheus!heartbeat' of type 'Service':
Object 'prometheus!check-snmp-connectivity' of type 'Service':
Object 'prometheus!ping4' of type 'Service':
Object 'prometheus!ssh' of type 'Service':
Object 'prometheus!check_ntp_diff' of type 'Service':
Object 'prometheus!check-snmp-memory' of type 'Service':
Object 'prometheus!check-unix-disks-snmp' of type 'Service':
Object 'prometheus!check-unix-disk' of type 'Service':

On the master:

icinga2:/var/log/icinga2# icinga2 object list -t service -n "prometheus*" | grep -i object
Object 'prometheus!check_ntp_diff' of type 'Service':
Object 'prometheus!check-snmp-connectivity' of type 'Service':
Object 'prometheus!check-snmp-memory' of type 'Service':
Object 'prometheus!check-unix-disk' of type 'Service':
Object 'prometheus!check-unix-disks-snmp' of type 'Service':
Object 'prometheus!heartbeat' of type 'Service':
Object 'prometheus!ping4' of type 'Service':
Object 'prometheus!ssh' of type 'Service':
2

For clarification, i am using icingaweb2 that connects to master node.

All nodes have accept_commands and accept_config enabled.

Satellites have notification feature, master does not.

3

Service and notification are two different topics. They need to be defined separately.

In your case you need to register hosts and services at your master, having hosts assigned to your satellite zone and services in any global zone. Config sync will then send according object to your satellite.

Notification is a different story. You need to have apply rules that defines notifications objects where they are needed. For example you want your satellites to notify for their hosts, you need a dedicated apply rule for it. I use assign where host.zone == ZoneName in my apply rules and this creates notification rules automatically for zones to which a particular host is assigned.

4

How are you adding services using the API on the master? Do you set the zone attribute of the services you create? If you set this to ABS, they should be synced to that zone.

5

I use service called signalilo that submits checks via icinga2 api based on prometheus alerts , and when inspecting them they indeed have the zone “ABS”.

The host they are appearing under is in that zone, as well.

For now i have switched the master to send notifications, instead of “ABS” zone, and things seem ok. It’s still a puzzling issue. For a while i thought it was an issue with database configuration, but that would not explain the discrepancy in the output of icinga2 object list.