Trying to force notification to be send on the same zone checks are being made

Hello Icinga community.

I have an icinga HA setup as descdibed below:

         +--------- HA ---------+
         |                      |
         v                      v
 +---------------+      +---------------+
 |    Master1    |<---->|    Master2    |
 +---------------+      +---------------+
         \                  /
          \                /
           \              /
            \            /
             \          /
              \        /
               \      /
                \    /
                 \  /
                  \/
         +---------------+
         | Azure Satellite|
         +---------------+

Everything works. But I haven’t achieved to make it so that my hosts in azure zone get their notifications send by the satellite but rather notifications are being sent by the master zone hosts. So currently master zone send all notifications. Also duplicate alerts happen sometimes thats when azure zone and master zone both send a sms for the same issue.

I’ve defined my notification in global-templates zone which is being synced to both master and azure zone.

This is how it’s currently defined:

/\* SMS ALERTS \*/
apply Notification "sms-host" to Host {
import "normal-notification-template"
import "host-notification-template"
command = "sms-notification"
vars.notif_type = "host"
user_groups = host.vars.notification.sms.groups
zone = host.zone
assign where host.vars.notification.sms && typeof(host.vars.notification.sms) == Dictionary
}

apply Notification "sms-service" to Service {
import "normal-notification-template"
import "service-notification-template"
command = "sms-notification"
vars.notif_type = "service"
user_groups = service.vars.notification.sms.groups
zone = host.zone
assign where service.vars.notification.sms && typeof(service.vars.notification.sms) == Dictionary
}

So essentially the question is. How do i force the notifications for objects in azure zone to be sent out only by the satellite?

Info on setup below.

icinga2 - The Icinga 2 network monitoring daemon (version: r2.16.0-1)
Platform: Red Hat Enterprise Linux
Platform version: 9.8 (Plow)

Master1
Enabled features: api checker command ido-mysql mainlog notification perfdata syslog
Master2
Enabled features: api checker command ido-mysql mainlog notification perfdata syslog
Satellite
Enabled features: api checker command mainlog notification perfdata syslog

icinga2 object list --type Zone
Object 'master' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 16:1-16:20

* \__name = "master"
* endpoints = \[ "master1.example.com", "master2.example.com" \]
  % = modified in '/etc/icinga2/zones.conf', lines 17:2-17:69
* global = false
* name = "master"
* package = "\_etc"
* parent = ""
* source_location
  * first_column = 1
  * first_line = 16
  * last_column = 20
  * last_line = 16
  * path = "/etc/icinga2/zones.conf"
* templates = \[ "master" \]
  % = modified in '/etc/icinga2/zones.conf', lines 16:1-16:20
* type = "Zone"
* zone = ""

Object 'azure' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 23:1-23:19

* \__name = "azure"
* endpoints = \[ "satellite.example.com" \]
  % = modified in '/etc/icinga2/zones.conf', lines 24:2-24:42
* global = false
* name = "azure"
* package = "\_etc"
* parent = "master"
  % = modified in '/etc/icinga2/zones.conf', lines 25:2-25:18
* source_location
  * first_column = 1
  * first_line = 23
  * last_column = 19
  * last_line = 23
  * path = "/etc/icinga2/zones.conf"
* templates = \[ "azure" \]
  % = modified in '/etc/icinga2/zones.conf', lines 23:1-23:19
* type = "Zone"
* zone = ""

Object 'global-templates' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 28:1-28:30

* \__name = "global-templates"
* endpoints = null
* global = true
  % = modified in '/etc/icinga2/zones.conf', lines 29:2-29:14
* name = "global-templates"
* package = "\_etc"
* parent = ""
* source_location
  * first_column = 1
  * first_line = 28
  * last_column = 30
  * last_line = 28
  * path = "/etc/icinga2/zones.conf"
* templates = \[ "global-templates" \]
  % = modified in '/etc/icinga2/zones.conf', lines 28:1-28:30
* type = "Zone"
* zone = ""

Any help is appreciated. Thanks in advance.

Do you have the notification feature enabled on the satellite?
Use icinga2 object list to inspect a notification object and try to find out where it lives (if the satellite has access to it or only the masters).

Yes the notification feature is enabled as presented in previous post.

This is the notification. It exists both on the master and on the satellite and it’s identical

Object 'example.target.com!https!sms-service' of type 'Notification':
  % declared in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 211:1-211:43
  * __name = "example.target.com!https!sms-service"
  * command = "sms-notification"
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 214:5-214:32
  * command_endpoint = ""
  * host_name = "example.target.com"
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 211:1-211:43
  * interval = 86400
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 95:5-95:18
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 95:5-95:18
  * name = "sms-service"
  * package = "_cluster"
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 211:1-211:43
  * period = "24x7"
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 94:5-94:19
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 94:5-94:19
  * service_name = "https"
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 211:1-211:43
  * source_location
    * first_column = 1
    * first_line = 211
    * last_column = 43
    * last_line = 211
    * path = "/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf"
  * states = [ "OK", "Warning", "Critical" ]
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 91:5-91:43
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 118:5-118:38
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 91:5-91:43
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 106:5-106:38
  * templates = [ "sms-service", "normal-notification-template", "generic-notification-template", "service-notification-template", "generic-notification-template" ]
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 211:1-211:43
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 116:1-116:52
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 90:1-90:53
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 104:1-104:53
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 90:1-90:53
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 92:5-93:72
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 119:5-119:33
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 92:5-93:72
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 107:5-107:33
  * user_groups = [ "test" ]
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 216:5-216:54
  * users = null
  * vars
    * notif_message_originator = "SMS-SENDER"
      % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 120:5-120:48
    * notif_title = "TITLE"
      % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/templates.conf', lines 121:5-121:47
    * notif_type = "service"
      % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 215:5-215:31
  * zone = "azure"
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 211:1-211:43
    % = modified in '/var/lib/icinga2/api/zones/global-templates/_etc/conf.d/notifications.conf', lines 217:5-217:20

Looking closer to this: command_endpoint = " "
Should the command endpoint be specified to be the satellite here?

I know i can force the command_endpoint for checks but is this the same for notification?

Like this

template Host "generic-host" {
    max_check_attempts = 5
    check_interval = 5m
    retry_interval = 1m
    check_command = "hostalive"
    enable_perfdata = true
    enable_event_handler = false
    command_endpoint = "master1.example.com"
}

Or is it the same variable here?

Also the satellite zone has only 1 endpoint so the endpoint is by default satellite1 which i can check from the check source. So why would i have to hard code that into the configs?

I wouldn’t touch the command_endpoint on the host.

Maybe a second template in notifications.conf for satelite1 notifications with the command_endpoint set there could be the solution?

So why would i have to hard code that into the configs?

I was hoping the first node with notifications enabled from the client to the master would send the notification automatically bit from your description it looks like this isn’t the case.

Yeah i’m just trying to avoid making duplicate templates and making my config more complex than is necessary. I’ll try to create a new notification template for my satellite zone to see if that works.