Notifications are mean to me

tamudo · November 23, 2020, 8:23am

Such a novel topic I know but I have two small issues with mail notifications (Conf. done through Director).

First issue is I’m trying to get service notifications sent out only on “OK/Recovery” or “Critical/Problem” states, just ignore warnings. This kind of works if the service goes the full cycle (Crit-[Warn]-OK)but when a service check goes from “Critical” to “Warning” and then back to “Critical” no notification is sent and I’d like to get one.It seems it will refuse to notify a critical again until it has reached OK status before at some point.
I was going to check if this is an issue related to disabling re-notifications. Still would like to get a notification every time a check that was not in critical reaches it no matter where it comes from.

Second issue is that sometimes I’d swear that a notification should have been sent but Icingaweb displays “Notification not sent out to any contacts”. One of us is wrong (probably me) but in the log file I can see some info when a notification is sent but nothing really to help me debug why Icinga decided no contact should be notified. How can I find out why Icinga thought that no contact needed to be notified?

Configuration info and an example of notification object:

Icinga 2 2.11-3
Icinga Web 2 2.8.0
Git Commit 642ec11228c3be8d2abbdff6ef31da77e34f6c70
PHP Version 7.2.11
MODULE VERSION
cube 1.1.0
director 1.7.2
fileshipper 1.1.0
graphite 1.1.0
incubator 0.5.0
ipl v0.5.0
monitoring 2.8.0
reactbundle 0.7.0
vspheredb 1.1.0

Object 'CN-PCCS-1030.FAKE.DOMAIN!Service AnnoyingProcess!Services - Annoying Servers' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 125:1-125:59
  * __name = "CN-PCCS-1030.FAKE.DOMAIN!Service AnnoyingProcess!Services - Annoying Servers"
  * command = "mail-service-notification"
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_templates.conf', lines 15:5-15:41
  * command_endpoint = ""
  * host_name = "CN-PCCS-1030.FAKE.DOMAIN"
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 125:1-125:59
  * interval = 28800
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 128:5-128:17
  * name = "Services - Annoying Servers"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 125:1-125:59
  * period = "workhours"
% = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_templates.conf', lines 16:5-16:24
  * service_name = "Service AnnoyingProcess"
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 125:1-125:59
  * source_location
    * first_column = 1
    * first_line = 125
    * last_column = 59
    * last_line = 125
    * path = "/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf"
  * states = [ "Critical", "OK" ]
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_templates.conf', lines 17:5-17:29
  * templates = [ "Services - Annoying Servers", "Email-Service-RegularUsers" ]
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 125:1-125:59
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_templates.conf', lines 14:1-14:50
  * times = null
 * type = "Notification"
  * types = [ "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_templates.conf', lines 18:5-18:33
  * user_groups = [ "eurovision", "nato" ]
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 130:5-130:39
  * users = [ "commanderkeen" ]
    % = modified in '/var/lib/icinga2/api/packages/director/87ffb85e-1b50-469d-bb73-08f9bb56d4cf/zones.d/CN-PCCS-9876.FAKE.DOMAIN/notification_apply.conf', lines 131:5-131:24
  * vars = null
  * zone = "CN-PCCS-9876.FAKE.DOMAIN"

dgoetz · November 23, 2020, 9:10am

The first issue is by design. Icinga will only send a notification for hard transition between ok and non-ok. You can make it volatile to make every output change being notified, but this could easily be to much.

The second one is much harder to debug, but I also look for the state transition, the notification applied, its filter and timeperiod and then the user’s notification filter and timeperiod.

tamudo · November 23, 2020, 9:23am

@dgoetz thanks for the information, I imagine that if I were to mark these services as “Volatile” the warning notifications would still be filtered out because of my config states (“OK”, “Critical”) but what could happen is that any Critical recheck would send out notifications ignoring the interval set in the template.
So trading incoveniences but it’s good to know why this is happening.