Can't stop Icinga to send notifications very often

FQDN · February 14, 2019, 10:03am

Hello guys.

I can’t solve my problem for 2 days already so i came here for an advice. I have read almost everything but I have no idea where is my misconfiguration.

My problem is that I have no idea how to stop Icinga to notify me with 800 emails per night about single problem. I would say that I have tried almost everything and the result was same or it worked just for a while. Of course i know that some people faced same issue but “old” Monitoring portal is not working right now and I didn’t find the same issue before when it worked.

I am using director so I can’t put 0 to the interval. Director writes it as “0s” to the conf file. Strange is fact that some services sometimes works and sometimes sends so many emails without any change in status or configuration - this could be caused by wrong settings of intervals in my opinion.

Here is an example of service that is sending emails (when Jenkins is stopped)

zones.d/director-global/notification_templates.conf

template Notification "notification-template-service" {
    times = {
        begin = 1s
        end = 3s
    }
    command = "notify-command-service"
    interval = 2h
    period = "workday1"
    states = [ Critical, OK, Warning ]
    types = [
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        FlappingEnd,
        FlappingStart,
        Problem,
        Recovery
    ]
    users = [ "sdavid" ]
}

zones.d/director-global/notification_apply.conf

apply Notification "notification-service" to Service {
    import "notification-template-service"

    assign where host.zone == "master"
    states = [ Critical, OK, Warning ]
    types = [
        Custom,
        DowntimeEnd,
        DowntimeRemoved,
        DowntimeStart,
        FlappingEnd,
        FlappingStart,
        Problem,
        Recovery
    ]
    users = [ "sdavid"]
    vars.notification_interval = "120"
}

zones.d/director-global/command_templates.conf

template NotificationCommand "notification-command-template-service" {
    import "plugin-notification-command"
    command = [ "/etc/icinga2/scripts/mail-service-notification.sh" ]
    vars.notification_interval = "120"
}

zones.d/master/commands.conf

object NotificationCommand "notify-command-service" {
    import "plugin-notification-command"
    import "notification-command-template-service"

    command = [ "/etc/icinga2/scripts/mail-service-notification.sh" ]
    arguments += {
        "-d" = {
            required = true
            value = "$notification_date$"
        }
        "-e" = {
            required = true
            value = "$notification_servicename$"
        }
        "-n" = {
            required = true
            value = "$notification_hostdisplayname$"
        }
        "-o" = {
            required = true
            value = "$notification_serviceoutput$"
        }
        "-r" = {
            required = true
            value = "$user.email$"
        }
        "-s" = {
            required = true
            value = "$notification_servicestate$"
        }
        "-t" = {
            required = true
            value = "$notification_type$"
        }
        "-u" = {
            required = true
            value = "$service.display_name$"
        }
    }
    vars.notification_date = "$icinga.long_date_time$"
    vars.notification_hostdisplayname = "$host.display_name$"
    vars.notification_servicedisplayname = "$service.display_name$"
    vars.notification_servicename = "$service.name$"
    vars.notification_serviceoutput = "$service.output$"
    vars.notification_servicestate = "$service.state$"
    vars.notification_type = "$notification.type$"
    vars.notification_useremail = "$user.email$"
}

zones.d/master/service_templates.conf

template Service "Jenkins_TCP" {
    check_command = "check-tcp-8080"
    max_check_attempts = "2"
    check_period = "workday1"
    check_interval = 15s
    retry_interval = 10s
    check_timeout = 1000s
    enable_notifications = true
    enable_active_checks = true
    enable_passive_checks = true
    enable_event_handler = true
    enable_flapping = true
    enable_perfdata = true
    volatile = true
    icon_image = "https://cdn2.iconfinder.com/data/icons/networking-glyph-2/128/81-512.png"
    groups = [ "application_monitoring" ]
    command_endpoint = host_name
}

zones.d/master/hosts.conf

object Host "jenkinsbuild2x.server.cz" {
    import "generic-host"

    display_name = "jenkinsbuild2x"
    address = "jenkinsbuild2x.server.cz"
    icon_image = "https://www.zdrojak.cz/wp-content/uploads/2016/11/226px-Jenkins_logo.svg_-72x100.png"
    groups = [ "DevOps", "Jenkins", "On-Premise" ] 
}

zones.d/master/agent_endpoints.conf

object Endpoint "jenkinsbuild2x.server.cz" {
    host = "jenkinsbuild2x.server.cz"
    log_duration = 0s
}

zones.d/master/agent_zones.conf

object Zone "jenkinsbuild2x.server.cz" {
    parent = "master"
    endpoints = [ "jenkinsbuild2x.server.cz" ]
}

Thank you for any advice and for you amazing work on Icinga2.

Best regards

Stepan

dnsmichi · February 14, 2019, 11:04am

Hi,

I have a few questions next to your thankfully detailed description

Can you show the Notification object from within Icinga combined from the template tree?

icinga2 object list --type Notification --name *Jenkins*notification-service*

Can you show a list/screenshot of the notification mails happening here?
Can you enable the debug log and trigger such a notification e.g. with a dummy service just notifying you … I would be interested in the interval of the notification event logging.
Is there a specific reason to use times.begin and times.end in that low window? I’m not sure if that would work here with the notification component running in a 5 second timer interval.
Last but not least, is that service flapping between OK and NOT-OK? If that is the case, you should enable flapping detection (off by default) to prevent unwanted notifications here.

Cheers,
Michael

Wolfgang · February 14, 2019, 1:46pm

This will send a notification every time the service is checked AND the state is not-ok.

FQDN · February 15, 2019, 1:20pm

Thank you very much for helping me Michael. It’s nice to see advice from Icinga Master.

Thank you @Wolfgang , I think that was the main problem.
(I found more misconfigurations in the end but I didn’t get what volatile checkbox does)

I started to write answers but I tested it for long to be correct with my answer so sorry for long responding time.

Notifications are working now so we can close this issue.

Cheers,

Stepan

Just if you are interested, I am posting some answers from my old configuration that I wrote before.

So here are some of my answers:

Can you show the Notification object from within Icinga combined from the template tree?

[root@icingaserver sdavid]# icinga2 object list --type Notification --name *Jenkins*notification-service*

[...]

Object 'jenkinsbuild2x.server.cz!Jenkins_TCP!notification-service' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 1:0-1:51
  * __name = "jenkinsbuild2x.server.cz!Jenkins_TCP!notification-service"
  * command = "notify-command-service"
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 6:5-6:38
  * command_endpoint = ""
  * host_name = "jenkinsbuild2x.server.cz"
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 1:0-1:51
  * interval = 7200
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 7:5-7:17
  * name = "notification-service"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 1:0-1:51
  * period = "workday1"
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 8:5-8:23
  * service_name = "Jenkins_TCP"
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 1:0-1:51
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 51
    * last_line = 1
    * path = "/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf"
  * states = [ "Critical", "OK", "Warning" ]
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 9:5-9:38
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 5:5-5:38
  * templates = [ "notification-service", "notification-template-service" ]
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 1:0-1:51
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 1:0-1:52
  * times
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 2:5-5:5
    * begin = 1
    * end = 3
  * type = "Notification"
  * types = [ "Custom", "DowntimeEnd", "DowntimeRemoved", "DowntimeStart", "FlappingEnd", "FlappingStart", "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 10:5-18:5
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 6:5-15:5
  * user_groups = null
  * users = [ "sdavid", "tbohunek" ]
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_templates.conf', lines 19:5-19:36
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 16:5-16:36
  * vars
    * notification_interval = "120"
      % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 17:5-17:38
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/packages/director/dbab72c4-d067-4d2b-92b2-2c373bd153ea/zones.d/director-global/notification_apply.conf', lines 1:0-1:51

	...

Can you show a list/screenshot of the notification mails happening here?

Here is screenshot from Outlook.
jenkinsbuild1x has exactly the same configuration in Icinga and it’s replication of jenkinsbuild2x.
Maybe I didn’t mentioned that I have the same issue with all my services and notifications.
If you want to see notification history from Icinga I can send it too.

.

Can you enable the debug log and trigger such a notification e.g. with a dummy service just notifying you … I would be interested in the interval of the notification event logging.

I don’t know how to make dummy service to send notifications via director because it is “OK” so it doesn’t send anything.
Here is the debug log without dummy (I hope that it helps anyway somehow)

[root@icingaserver sdavid]# tail -f /var/log/icinga2/debug.log | grep mail
[2019-02-14 13:00:41 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:41 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 4434
[2019-02-14 13:00:41 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:41 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 4442
[2019-02-14 13:00:41 +0100] notice/Process: PID 4434 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:41 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:00:41 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 4434, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:41 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:00:41 +0100] notice/Process: PID 4442 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:41 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:00:41 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 4442, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:41 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:00:56 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:56 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5036
[2019-02-14 13:00:56 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:56 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5037
[2019-02-14 13:00:56 +0100] notice/Process: PID 5037 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:56 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:00:56 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5037, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:56 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:00:56 +0100] notice/Process: PID 5036 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:56 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:00:56 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5036, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:00:56 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:01:11 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5209
[2019-02-14 13:01:11 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5210
[2019-02-14 13:01:11 +0100] notice/Process: PID 5210 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:01:11 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5210, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:01:11 +0100] notice/Process: PID 5209 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:01:11 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5209, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:01:11 +0100] notice/Process: PID 5209 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:01:11 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5209, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:01:26 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5401
[2019-02-14 13:01:26 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5402
[2019-02-14 13:01:26 +0100] notice/Process: PID 5402 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:01:26 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5402, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:01:26 +0100] notice/Process: PID 5401 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:01:26 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5401, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:01:26 +0100] notice/Process: PID 5401 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:01:26 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5401, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:01:26 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:02:11 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:02:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5739
[2019-02-14 13:02:11 +0100] notice/Process: Running command '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:02:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU': PID 5740
[2019-02-14 13:02:11 +0100] notice/Process: PID 5740 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:02:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:02:11 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5740, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:02:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'tomas.bohunek@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)
[2019-02-14 13:02:11 +0100] notice/Process: PID 5739 ('/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:02:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67
[2019-02-14 13:02:11 +0100] warning/PluginNotificationTask: Notification command for object 'myczvl1dd0scm1.ux.mbid.cz!CPU' (PID: 5739, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-d' '2019-02-14 13:02:11 +0100' '-e' 'CPU' '-n' 'scmdrcx' '-o' 'CRITICAL: CPU Idle = 1.69% ' '-r' 'stepan.david@moneta.cz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'CPU') terminated with exit code 67, output: WARNING: RunAsUser for MSP ignored, check group ids (egid=298, want=51)

Is there a specific reason to use times.begin and times.end in that low window? I’m not sure if that would work here with the notification component running in a 5 second timer interval.

No there is not any specific reason. I just thought that it would be better to use lower numbers for testing and that 5s would be enough.

…