Service Notifications not working

I am trying to use the apply notifications rule for services. Probably a lack of understanding but I do not get any service notifications. Here are my conf files for clarification.

services.conf

apply Service "ssh" {
  import "generic-service"

  check_command = "ssh"

  assign where (host.address || host.address6) && host.vars.os == "Linux"
}



apply Service for (http_vhost => config in host.vars.http_vhosts) {
  import "generic-service"

  check_command = "http"

  vars += config
}

apply Service for (disk => config in host.vars.disks) {
  import "generic-service"

  check_command = "disk"

  vars += config
}

apply Service "icinga" {
  import "generic-service"

  check_command = "icinga"

  assign where host.name == NodeName
}

apply Service "load" {
  import "generic-service"

  check_command = "load"

  /* Used by the ScheduledDowntime apply rule in `downtimes.conf`. */
  vars.backup_downtime = "02:00-03:00"

  assign where host.name == NodeName
}

apply Service "procs" {
  import "generic-service"

  check_command = "procs"

  assign where host.name == NodeName
}

apply Service "swap" {
  import "generic-service"

  check_command = "swap"

  assign where host.name == NodeName
}

apply Service "users" {
  import "generic-service"

  check_command = "users"

  assign where host.name == NodeName
}

notifications.conf

apply Notification "mail-icingaadmin" to Host {
  import "mail-host-notification"
  user_groups = [ "sysadmins" ]
  assign where host.address !=" "
}

apply Notification "mail-icingaadmin" to Service {
  import "mail-service-notification"
  user_groups = [ "sysadmins" ]
  assign where host.address !=""
}

template.conf

template Host "generic-host" {
  max_check_attempts = 3
  check_interval = 1m
  retry_interval = 30s
  vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
  vars.notification["telegram"] = {
    users = [ "telegram-bot" ]
  }

  check_command = "hostalive"
}

template Service "generic-service" {
  max_check_attempts = 5
  check_interval = 1m
  retry_interval = 30s
vars.notification["mail"] = {
    groups = [ "icingaadmins" ]
  }
  vars.notification["telegram"] = {
   users = [ "telegram-bot" ]
 }

}

template User "generic-user" {

}

##############################
########  EMAIL ##############
##############################

##### HOST #####

template Notification "mail-host-notification" {
  command = "mail-host-notification"

  states = [ Up, Down ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_logtosyslog = false
    notification_icingaweb2url = "https://mydomain.com/icingaweb2"
    notification_from = "infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
  }

  period = "24x7"
}

##### SERVICE #####

template Notification "mail-service-notification" {
  command = "mail-service-notification"

  states = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  vars += {
    notification_icingaweb2url = "https://mydomain.com/icingaweb2"
    notification_from = "Infrastructure Icinga2 Monitoring Server <inframon@mydomain.com>"
    notification_logtosyslog = false
  }

  period = "24x7"
}

Hi,

please pick a category when creating a new topic - #community isn’t where this belongs, moved to #icinga-2-core

Pick one of these services, and send in a critical check result. Once it changes to a soft state, repeat the check result sending in to reach a hard state (typically 2/3 and then). Once the hard state is reached, a notification is sent.

Read more about states and notifications here.

Cheers,
Michael

Hi,
Sorry about the choice of category.

This apply even when I select the faulting service type for example memory warning and select to send a custom notification? Because when I do this it still doesn’t send an email.

Hmmm Custom is a specific type which is currently allowed in your notification template. It just doesn’t work the same way as when triggering a critical state and sending a notification.

What does the icinga2.log say about the case when such a notification email should happen?

Cheers,
Michael

aha this is something significant I guess:

'/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: Required parameter 'SERVICENAME' is missing.

Where should servicename be?

That looks like as if the host would send a notification, but actually uses the service notification scripts. In this scope, the host doesn’t have a service name.

Compare the timestamps from the logs with your current attempt, make sure this matches. The above is likely something from your previous tests - extract the current test timestamp and post the full line here.

Cheers,
Michael

It seems to the be the same:

[2019-04-30 12:49:45 +0200] warning/PluginNotificationTask: Notification command for object 'hostlinuxserver1!Memory' (PID: 4078321, arguments: '/etc/icinga2/scripts/mail-service-notification.sh') terminated with exit code 1, output: Required parameter 'SERVICENAME' is missing.

Required parameters:
-d LONGDATETIME ($icinga.long_date_time$)
-e SERVICENAME ($service.name$)
-l HOSTNAME ($host.name$)
-n HOSTDISPLAYNAME ($host.display_name$)
-o SERVICEOUTPUT ($service.output$)
-r USEREMAIL ($user.email$)
-s SERVICESTATE ($service.state$)
-t NOTIFICATIONTYPE ($notification.type$)
-u SERVICEDISPLAYNAME ($service.display_name$)

Optional parameters:
-4 HOSTADDRESS ($address$)
-6 HOSTADDRESS6 ($address6$)
-b NOTIFICATIONAUTHORNAME ($notification.author$)
-c NOTIFICATIONCOMMENT ($notification.comment$)
-i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
-f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
-v ($notification_sendtosyslog$, Default: false)

I found a workaround to change SERVICENAME to SERVICEDESC in /etc/icinga2/scripts/mail-service-notification.sh

So this part works I need to test again if the service notifications are working or not.

Seems you are using a very old NotificationCommand definition - I would rather fix SERVICENAME in there, as the other definition is old and stale.

Where could I get or compile an up to date NotificationCommand definition?

The example configuration is provided as part of the package, which sources from our Git repository and specific tags. This one is the version for 2.10.4 for example: https://github.com/Icinga/icinga2/blob/v2.10.4/etc/icinga2/conf.d/commands.conf

ok after that workaround everything is working as desired. Thank you so much for you help!

1 Like