How are you handling notification rules (DSL only)?

Icinga 2
1

Hi all,
as things grow, I have more user using my cluster, which want different notifications. So I started to re-factor my notification rules and I am curious, how you are handling it.
There are different notification channels (mail, sms, ms teams) and different teams (which in my case are just “single user” in icinga2). Example given for mails: Basically, my apply rule was

users = host.vars.notification.mail.users
…
assign where host.vars.notification.mail

which means, that every user gets the same kind of email (based on the states and types) I defined and whether he is in host.vars.notification.mail.users or not.
But what is, if some user wants other states and types? Of course, I could write another rule and assign it like

assign where “user1” in host.vars.notification.mail.users

But then I also have to ignore this user from the “default mail” apply rule to avoid double notifications. Would a

ignore where “user1” in host.vars.notification.mail.users

work in the default mail rule or would this ignore the rule for all users in this dictionary?
I don’t want to write an apply rule for every user, so a “default” mail rule with the option to change it for specific users would be fine.

On the other way, of course, I could just config the notification rules with all states and types and filter them in the user object with states and types. But this also wouldn’t work if e.g. the user wants warnings and criticals as mail and only criticals as sms.

So I am curious, how you guys are handling it.

And by the way: I saw this about a NoMa2 a long time ago. https://www.youtube.com/watch?v=bXoakb9PE-c
What happened with it? If it is not released yet, please make it independent from the director. DSL is love :smiley:

Cheers,
Marcus

2

Hello @MarcusCaepio!

Then he gets only the notifications for his states and types:

Best,
AK

3

Okay, but what if he wants e.g.
Warning, Critical, Unknown as mail
and only Critical as sms? ^^

By the way, I can’t remember the states, types and period filters on user objects. Were they added recently?

4 
➜  icinga2 git:(master) cat example.conf
object CheckCommand "cc" {
 command = [ "true" ]
}

object Host "h" {
 check_command = "cc"
}

apply Service "s" {
 check_command = "cc"
 vars.users = [ "u" ]
 assign where true
}

object User "u" {
 vars.mail = [ Warning, Critical, Unknown ]
 vars.sms = [ Critical ]
}

object NotificationCommand "mail" {
 command = [ "true" ]
}

apply Notification "mail-" for (user in service.vars.users) to Service {
 command = "mail"
 users = [ user ]
 states = get_object(User, user).vars.mail
}

object NotificationCommand "sms" {
 command = [ "true" ]
}

apply Notification "sms-" for (user in service.vars.users) to Service {
 command = "sms"
 users = [ user ]
 states = get_object(User, user).vars.sms
}
➜  icinga2 git:(master) prefix/sbin/icinga2 daemon -c example.conf -C
[2020-08-12 16:03:12 +0200] information/cli: Icinga application loader (version: v2.12.0)
[2020-08-12 16:03:12 +0200] information/cli: Loading configuration file(s).
[2020-08-12 16:03:12 +0200] information/ConfigItem: Committing config item(s).
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 1 Host.
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 2 Notifications.
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 1 Service.
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 1 User.
[2020-08-12 16:03:12 +0200] information/ConfigItem: Instantiated 1 CheckCommand.
[2020-08-12 16:03:12 +0200] information/ScriptGlobal: Dumping variables to file '/Users/aklimov/NET/WS/icinga2/prefix/var/cache/icinga2/icinga2.vars'
[2020-08-12 16:03:12 +0200] information/cli: Finished validating the configuration file(s).
➜  icinga2 git:(master) prefix/sbin/icinga2 object list
Object 'app' of type 'IcingaApplication':
  % declared in '', lines 0:0-0:0
  * __name = "app"
  * enable_event_handlers = true
  * enable_flapping = true
  * enable_host_checks = true
  * enable_notifications = true
  * enable_perfdata = true
  * enable_service_checks = true
  * environment = ""
  * name = "app"
  * package = ""
  * source_location
    * first_column = 0
    * first_line = 0
    * last_column = 0
    * last_line = 0
    * path = ""
  * templates = [ "app" ]
    % = modified in '', lines 0:0-0:0
  * type = "IcingaApplication"
  * vars = null
  * zone = ""

Object 'mail' of type 'NotificationCommand':
  % declared in 'example.conf', lines 20:1-20:33
  * __name = "mail"
  * arguments = null
  * command = [ "true" ]
    % = modified in 'example.conf', lines 21:2-21:21
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "mail"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 20
    * last_column = 33
    * last_line = 20
    * path = "example.conf"
  * templates = [ "mail", "plugin-notification-command" ]
    % = modified in 'example.conf', lines 20:1-20:33
    % = modified in 'methods-itl.conf', lines 22:2-22:122
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = ""

Object 'sms' of type 'NotificationCommand':
  % declared in 'example.conf', lines 30:1-30:32
  * __name = "sms"
  * arguments = null
  * command = [ "true" ]
    % = modified in 'example.conf', lines 31:2-31:21
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "sms"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 30
    * last_column = 32
    * last_line = 30
    * path = "example.conf"
  * templates = [ "sms", "plugin-notification-command" ]
    % = modified in 'example.conf', lines 30:1-30:32
    % = modified in 'methods-itl.conf', lines 22:2-22:122
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = ""

Object 'u' of type 'User':
  % declared in 'example.conf', lines 15:1-15:15
  * __name = "u"
  * display_name = "u"
  * email = ""
  * enable_notifications = true
  * groups = [ ]
  * name = "u"
  * package = "_etc"
  * pager = ""
  * period = ""
  * source_location
    * first_column = 1
    * first_line = 15
    * last_column = 15
    * last_line = 15
    * path = "example.conf"
  * states = null
  * templates = [ "u" ]
    % = modified in 'example.conf', lines 15:1-15:15
  * type = "User"
  * types = null
  * vars
    * mail = [ "Warning", "Critical", "Unknown" ]
      % = modified in 'example.conf', lines 16:2-16:43
    * sms = [ "Critical" ]
      % = modified in 'example.conf', lines 17:2-17:24
  * zone = ""

Object 'cc' of type 'CheckCommand':
  % declared in 'example.conf', lines 1:0-1:23
  * __name = "cc"
  * arguments = null
  * command = [ "true" ]
    % = modified in 'example.conf', lines 2:2-2:21
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 19:3-19:23
    * arguments = [ "checkable", "cr", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginCheck"
    * side_effect_free = false
    * type = "Function"
  * name = "cc"
  * package = "_etc"
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 23
    * last_line = 1
    * path = "example.conf"
  * templates = [ "cc", "plugin-check-command" ]
    % = modified in 'example.conf', lines 1:0-1:23
    % = modified in 'methods-itl.conf', lines 18:2-18:94
  * timeout = 60
  * type = "CheckCommand"
  * vars = null
  * zone = ""

Object 'h' of type 'Host':
  % declared in 'example.conf', lines 5:1-5:15
  * __name = "h"
  * action_url = ""
  * address = ""
  * address6 = ""
  * check_command = "cc"
    % = modified in 'example.conf', lines 6:2-6:21
  * check_interval = 300
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "h"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 3
  * name = "h"
  * notes = ""
  * notes_url = ""
  * package = "_etc"
  * retry_interval = 60
  * source_location
    * first_column = 1
    * first_line = 5
    * last_column = 15
    * last_line = 5
    * path = "example.conf"
  * templates = [ "h" ]
    % = modified in 'example.conf', lines 5:1-5:15
  * type = "Host"
  * vars = null
  * volatile = false
  * zone = ""

Object 'h!s' of type 'Service':
  % declared in 'example.conf', lines 9:1-9:17
  * __name = "h!s"
  * action_url = ""
  * check_command = "cc"
    % = modified in 'example.conf', lines 10:2-10:21
  * check_interval = 300
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "s"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "h"
    % = modified in 'example.conf', lines 9:1-9:17
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 3
  * name = "s"
    % = modified in 'example.conf', lines 9:1-9:17
  * notes = ""
  * notes_url = ""
  * package = "_etc"
    % = modified in 'example.conf', lines 9:1-9:17
  * retry_interval = 60
  * source_location
    * first_column = 1
    * first_line = 9
    * last_column = 17
    * last_line = 9
    * path = "example.conf"
  * templates = [ "s" ]
    % = modified in 'example.conf', lines 9:1-9:17
  * type = "Service"
  * vars
    * users = [ "u" ]
      % = modified in 'example.conf', lines 11:2-11:21
  * volatile = false
  * zone = ""

Object 'h!s!sms-u' of type 'Notification':
  % declared in 'example.conf', lines 34:1-34:69
  * __name = "h!s!sms-u"
  * command = "sms"
    % = modified in 'example.conf', lines 35:2-35:16
  * command_endpoint = ""
  * host_name = "h"
    % = modified in 'example.conf', lines 34:1-34:69
  * interval = 1800
  * name = "sms-u"
  * package = "_etc"
    % = modified in 'example.conf', lines 34:1-34:69
  * period = ""
  * service_name = "s"
    % = modified in 'example.conf', lines 34:1-34:69
  * source_location
    * first_column = 1
    * first_line = 34
    * last_column = 69
    * last_line = 34
    * path = "example.conf"
  * states = [ "Critical" ]
    % = modified in 'example.conf', lines 37:2-37:41
  * templates = [ "sms-u" ]
    % = modified in 'example.conf', lines 34:1-34:69
  * times = null
  * type = "Notification"
  * types = null
  * user_groups = null
  * users = [ "u" ]
    % = modified in 'example.conf', lines 36:2-36:17
  * vars = null
  * zone = ""

Object 'h!s!mail-u' of type 'Notification':
  % declared in 'example.conf', lines 24:1-24:70
  * __name = "h!s!mail-u"
  * command = "mail"
    % = modified in 'example.conf', lines 25:2-25:17
  * command_endpoint = ""
  * host_name = "h"
    % = modified in 'example.conf', lines 24:1-24:70
  * interval = 1800
  * name = "mail-u"
  * package = "_etc"
    % = modified in 'example.conf', lines 24:1-24:70
  * period = ""
  * service_name = "s"
    % = modified in 'example.conf', lines 24:1-24:70
  * source_location
    * first_column = 1
    * first_line = 24
    * last_column = 70
    * last_line = 24
    * path = "example.conf"
  * states = [ "Warning", "Critical", "Unknown" ]
    % = modified in 'example.conf', lines 27:2-27:42
  * templates = [ "mail-u" ]
    % = modified in 'example.conf', lines 24:1-24:70
  * times = null
  * type = "Notification"
  * types = null
  * user_groups = null
  * users = [ "u" ]
    % = modified in 'example.conf', lines 26:2-26:17
  * vars = null
  * zone = ""
➜  icinga2 git:(master)
2 Likes
5

Oh brilliant!
Thanks a lot. Will help me so much while re-factoring!

6

@Al2Klimov one more question. If a user has no specific types/states defined and a default setup should match, would sth. like this work?

apply Notification "mail-" for (user in service.vars.users) to Service {
 command = "mail"
 users = [ user ]
 if (get_object(User, user).vars.mail) {
 states = get_object(User, user).vars.mail
 } else {
states = [ Ok, Warning, Critical, Unknown ] }
}

Or how has the condition be written?

7

You can shorten it if desired:

states = get_object(User, user).vars.mail || [ Ok, Warning, Critical, Unknown ]

1 Like
8

Nice! Thanks again @Al2Klimov

1 Like
9

Hi again,
sooooo, I have these Notifications rules, based on the suggest of @Al2Klimov (thanks again!):

apply Notification "teams-" for (user in host.vars.notification.teams.users) to Host {
  command = "teams-host-notification"
  period = "24x7"
  interval = 0
  users = [ user ]
  states = get_object(User, user).vars.teams_host_states || [ Up, Down ]
  types = get_object(User, user).vars.teams_host_types || [ DowntimeStart, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart, FlappingEnd ]
}

apply Notification "teams-" for (user in host.vars.notification.teams.users) to Service {
  command = "teams-service-notification"
  period = "24x7"
  interval = 0
  users = [ user ]
  states = get_object(User, user).vars.teams_service_states  || [ OK, Warning, Critical, Unknown ]
  types = get_object(User, user).vars.teams_service_types || [ DowntimeStart, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart, FlappingEnd ]
ignore where service.vars.no_teams
ignore where service.name in host.vars.exclude_teams
}

apply Notification "teams-" for (user in service.vars.notification.teams.users) to Service {
  command = "teams-service-notification"
  period = "24x7"
  interval = 0
  users = [ user ]
  states = get_object(User, user).vars.teams_service_states  || [ OK, Warning, Critical, Unknown ]
  types = get_object(User, user).vars.teams_service_types || [ DowntimeStart, DowntimeEnd, DowntimeRemoved, Custom, Acknowledgement, Problem, Recovery, FlappingStart, FlappingEnd ]
ignore where service.vars.no_teams
ignore where service.name in host.vars.exclude_teams

3 Rules because of

  • Both Host and Service Notifications for user X
  • No Host but Service Notifications for user X
    and so on…

Now I have one user, where the second rule should not apply. It is an “announcement” user, where only host notifications should be send, but no service notifications (except it is explicitly defined via the service.vars.notif…) . I have added the user to my generic-host template, so every host will be announced ( vars.notification.teams.users += [ “announcement” ]

I tried to ignore the second rule for this user by e.g.:
ignore where host.vars.notification.teams.users.contains("announcement") or
ignore where "announcement" in host.vars.notification.teams.users or
ignore where host.vars.notification.teams.users == "announcement"

and so on… All three rules disable not only the notifications for the announcement user, but the notification rules for all users in the array of host.vars.notification.teams.users. So can sb. give me a hint, how to ignore only the announcement user, when the host config is e.g.:
vars.notification.teams.users = [ “A”,“B”,“announcement”]

Cheers,
Marcus

10

I guess for (user in service.vars.notification.teams.users.filter(user => user != "announcement")).

11

You guess right! Thanks a lot!