Restrict Notifications/Metrics to Master and Satellite

Icinga 2
, ,
1

Hello community,

We have a setup with a Master HA and 3 zones Satellites HA.

We wanted to ease the load on the Masters and let every Satellite to handle the shipping of their own notifications/metrics.

In addition we wanted to also monitor the Icinga cluster itself, and ofc it should be the Masters to handle this part (monitoring of the zones, ido, etc).

Though, if we also activate the notification and graphite feature on the Masters they will also send (as a duplicate) all the satellite’s ones.

Is there a way to avoid for the master to see what the satellite are doing, in this matter?

1 Like
2

Hi & welcome,

We use host.zone == ZoneName in our assign rules to apply notification objects only for hosts and services which belongs to master or a satellite zone.

For monitoring of icinga staff we use from ITL: icinga, cluster, cluster-zone and ido.

3

Thanks for the answer

Yeah thank you, I know the ITL, I just mentioned this to explain why the need to “segregate” only certain notification type to a specific master/satellite.

I see what you mean about assigning the notification object to a specific zone, but the Master would still need to monitoring and notify any cluster issue (through ITL), therefor still have the notification object assigned to itself as well.

My understanding was that by placing certain host objects in a specific zone.d folder, only that satellite would perform the needed check and send the notification, but it looks like that the Master still see them as well, and still perform any possible check/notification as well.

As I said, I still need the notification and checker objects on the Master as well to monitor itself.

4

Sending notification depends on how you define notification objects. If you apply all of them at your master, then the master will send notification no matter which of the satellites was performing the check.

And?

5

Our notification object simple call a script and is assigned to every host that has the “notification” setting enabled,

object NotificationCommand "notify-alert" {
  import "plugin-notification-command"
  command = [ SysconfDir + "/icinga2/scripts/script.sh" ]
}
apply Notification "notify-alert" to Host {
 ....
  assign where host.notifications == true
}

Both Master and Satellites have the Notification feature active in order to send their “own notifications”, according to what they are monitoring.

  • The Master should monitor only itself and send notification/graphite metrics for these
  • The satellites monitor what is made available in their own zones.d, and send notification/graphite metrics for these

What the master doesn’t need to do, is to “interact” with the Host/service objects defined under each zones.d.

6

As I said, this could be achieved by using such assign definition:

assign where host.notifications == true && host.zone == ZoneName

7

I have to admit that then I am a bit lost on how exactly the host.zone == ZoneName works.

I thought it was a variable to use, and you were telling me to assign the notification only to host that were part of a specific satellite:
assign where host.zone == satellite-02 || host.zone == satellite-01

Therefor I should assign this condition to all the services as well, in order for each zone to perform the check of the host objects of its own zone?

8

ZoneName is a constant which is defined individually on each icinga machine (master and satellites). The limitation host.zone == ZoneName will only create notification objects for hosts and services that belong to a particular icinga machine.

This has nothing to do with where a check is performed.

1 Like
9

I’m coming back to this as your answer was only for the Notification, but not the metrics (graphiteWriter). Can I achieve the same somehow?

10

Hmm, not really. You could disable graphite feature at your masters (makes sense only if you have no host objects defined there)

or (no idea if it will work)

Add zone info to your perfdata and create “retention” in graphite to delete immediately.

11

nope the master has to send metrics.

What we need is:

  • Satellite → send metrics/notification for the hosts it monitors
  • Master → send metrics/notification for the cluster check. And as only the master has the idodb feature (as suggested for best practice), it’s the only one that can properly do the cluster check and return the idodb metrics
12

Hi,
I am not aware of a good way to restrict which Perfdata are written by any Icinga2 instance. Therefore I would not write metrics on the satellites at all and move that to the master.