Intermittent duplicate notifications

I have encountered a strange issue where some notifications are triggered twice. After looking for patterns in the log files it seems to me that the issue have something to do with the fact that a reminder is triggered first, then the “correct” notification is triggered.

Icinga2 environment:

master1 <–> master2

master1 and master2 are both configured to handle notifications.
version: r2.11.3-1

Master1

Disabled features: command compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql influxdb mainlog notification

Master 2
Disabled features: command compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql influxdb mainlog notification

Notification object for Service ‘DK-MA-BRO-P013!CPU’.

Object 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' of type 'Notification':
  % declared in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
  * __name = "DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api"
  * command = "api-service-zendesk"
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 19:5-19:35
  * command_endpoint = ""
  * host_name = "DK-MA-BRO-P013"
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
  * interval = 0
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 14:5-14:17
  * name = "zendesk-digital-service-alerts-api"
  * package = "_cluster"
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
  * period = ""
  * service_name = "CPU"
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 66
    * last_line = 11
    * path = "/var/lib/icinga2/api/zones/master/director/notification_apply.conf"
  * states = [ "Critical", "OK", "Warning" ]
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 16:5-16:38
  * templates = [ "zendesk-digital-service-alerts-api", "Zendesk Service Notification Template" ]
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 18:1-18:61
  * times = null
  * type = "Notification"
  * types = [ "FlappingEnd", "FlappingStart", "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 17:5-17:61
  * user_groups = null
  * users = [ "zendesk-digital-api" ]
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 18:5-18:37
  * vars
    * notification_address = "$address$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 20:5-20:43
    * notification_address6 = "$address6$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 21:5-21:45
    * notification_author = "$notification.author$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 22:5-22:54
    * notification_comment = "$notification.comment$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 23:5-23:56
    * notification_date = "$icinga.long_date_time$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 24:5-24:54
    * notification_doclink = "$host.notes_url$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 25:5-25:50
    * notification_hostdisplayname = "$host.display_name$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 26:5-26:61
    * notification_hostname = "$host.name$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 27:5-27:46
    * notification_notes = "$host.notes$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 28:5-28:44
    * notification_servicedisplayname = "$service.display_name$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 29:5-29:67
    * notification_servicename = "$service.name$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 30:5-30:52
    * notification_serviceoutput = "$service.output$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 31:5-31:56
    * notification_servicestate = "$service.state$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 32:5-32:54
    * notification_type = "$notification.type$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 33:5-33:50
    * notification_useremail = "$user.email$"
      % = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 34:5-34:48
  * zone = "nordic"
    % = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66

Log files from master servers:

Notification for ‘DK-MA-BRO-P013!CPU’ triggered twice:

Master1
[2020-05-07 22:33:49 +0200] information/Checkable: Checkable 'DK-MA-BRO-P013!CPU' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.

Master2
[2020-05-07 22:33:49 +0200] information/Notification: Sending reminder 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 22:33:49 +0200] information/Notification: Completed sending 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for checkable 'DK-MA-BRO-P013!CPU' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
[2020-05-07 22:33:49 +0200] information/Checkable: Checkable 'DK-MA-BRO-P013!CPU' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-05-07 22:33:49 +0200] information/Notification: Sending 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 22:33:49 +0200] information/Notification: Completed sending 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for checkable 'DK-MA-BRO-P013!CPU' and user 'zendesk-digital-api' using command 'api-service-zendesk'.

Notification for ‘SE-RO-LUN-P003!Memory’ triggered twice:

Master1
[2020-05-07 08:32:13 +0200] information/Checkable: Checkable 'SE-RO-LUN-P003!Memory' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.

Master2
[2020-05-07 08:32:13 +0200] information/Notification: Sending reminder 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 08:32:13 +0200] information/Notification: Completed sending 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for checkable 'SE-RO-LUN-P003!Memory' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
[2020-05-07 08:32:13 +0200] information/Checkable: Checkable 'SE-RO-LUN-P003!Memory' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-05-07 08:32:13 +0200] information/Notification: Sending 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 08:32:13 +0200] information/Notification: Completed sending 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for checkable 'SE-RO-LUN-P003!Memory' and user 'zendesk-digital-api' using command 'api-service-zendesk'.

Notification for ‘NO-MA-BLT-A001!IVY Motion Detection A’ were not triggered twice:

Master1
[2020-05-07 06:30:00 +0200] information/Checkable: Checkable 'NO-MA-BLT-A001!IVY Motion Detection A' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-05-07 06:30:00 +0200] information/Notification: Sending 'Problem' notification 'NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 06:30:00 +0200] information/Notification: Completed sending 'Problem' notification 'NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api' for checkable 'NO-MA-BLT-A001!IVY Motion Detection A' and user 'zendesk-digital-api' using command 'api-service-zendesk'.

Master2
[2020-05-07 06:30:00 +0200] information/Checkable: Checkable 'NO-MA-BLT-A001!IVY Motion Detection A' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.

Notification object for ‘NO-MA-BLT-A001!IVY Motion Detection A’:

Object 'NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
  * __name = "NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api"
  * command = "api-service-zendesk"
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 19:5-19:35
  * command_endpoint = ""
  * host_name = "NO-MA-BLT-A001"
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
  * interval = 0
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 14:5-14:17
  * name = "zendesk-digital-service-alerts-api"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
  * period = ""
  * service_name = "IVY Motion Detection A"
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 66
    * last_line = 11
    * path = "/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf"
  * states = [ "Critical", "OK", "Warning" ]
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 16:5-16:38
  * templates = [ "zendesk-digital-service-alerts-api", "Zendesk Service Notification Template" ]
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 18:1-18:61
  * times = null
  * type = "Notification"
  * types = [ "FlappingEnd", "FlappingStart", "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 17:5-17:61
  * user_groups = null
  * users = [ "zendesk-digital-api" ]
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 18:5-18:37
  * vars
    * notification_address = "$address$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 20:5-20:43
    * notification_address6 = "$address6$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 21:5-21:45
    * notification_author = "$notification.author$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 22:5-22:54
    * notification_comment = "$notification.comment$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 23:5-23:56
    * notification_date = "$icinga.long_date_time$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 24:5-24:54
    * notification_doclink = "$host.notes_url$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 25:5-25:50
    * notification_hostdisplayname = "$host.display_name$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 26:5-26:61
    * notification_hostname = "$host.name$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 27:5-27:46
    * notification_notes = "$host.notes$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 28:5-28:44
    * notification_servicedisplayname = "$service.display_name$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 29:5-29:67
    * notification_servicename = "$service.name$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 30:5-30:52
    * notification_serviceoutput = "$service.output$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 31:5-31:56
    * notification_servicestate = "$service.state$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 32:5-32:54
    * notification_type = "$notification.type$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 33:5-33:50
    * notification_useremail = "$user.email$"
      % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 34:5-34:48
  * zone = "nordic"
    % = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66

Looking for help troubleshooting the issue further so all input is appreciated.

Best regards

Hello msfdl,
I hope you are doing well. Do you have any apply notification rules set to send out notifications?

My notification get assigned by variables. When I added an apply rule for the same user that was already assigned by the variable, I would receive two notifications. I needed an ignore statement in my apply rule if the user name already match in the variable.

Also do you have ha_enable = true attribute set on the NotificationComponent?

Regards
Alex

@aclark6996: Yes, the notifications are assigned with apply rules.

apply Notification "zendesk-digital-host-alerts-api" to Host {
    command = "api-host-zendesk"
    interval = 0s
    assign where host.address
    states = [ Down, Up ]
    types = [ FlappingEnd, FlappingStart, Problem, Recovery ]
    users = [ "zendesk-digital-api" ]
    vars.notification_address = "$address$"
    vars.notification_address6 = "$address6$"
    vars.notification_author = "$notification.author$"
    vars.notification_comment = "$notification.comment$"
    vars.notification_date = "$icinga.long_date_time$"
    vars.notification_doclink = "$host.notes_url$"
    vars.notification_hostdisplayname = "$host.display_name$"
    vars.notification_hostname = "$host.name$"
    vars.notification_hostoutput = "$host.output$"
    vars.notification_hoststate = "$host.state$"
    vars.notification_notes = "$host.notes$"
    vars.notification_type = "$notification.type$"
    vars.notification_useremail = "$user.email$"
}

and

apply Notification "zendesk-digital-service-alerts-api" to Service {
    command = "api-service-zendesk"
    interval = 0s
    assign where service.host
    states = [ Critical, OK, Warning ]
    types = [ FlappingEnd, FlappingStart, Problem, Recovery ]
    users = [ "zendesk-digital-api" ]
    vars.notification_address = "$address$"
    vars.notification_address6 = "$address6$"
    vars.notification_author = "$notification.author$"
    vars.notification_comment = "$notification.comment$"
    vars.notification_date = "$icinga.long_date_time$"
    vars.notification_doclink = "$host.notes_url$"
    vars.notification_hostdisplayname = "$host.display_name$"
    vars.notification_hostname = "$host.name$"
    vars.notification_notes = "$host.notes$"
    vars.notification_servicedisplayname = "$service.display_name$"
    vars.notification_servicename = "$service.name$"
    vars.notification_serviceoutput = "$service.output$"
    vars.notification_servicestate = "$service.state$"
    vars.notification_type = "$notification.type$"
    vars.notification_useremail = "$user.email$"
}

If there were duplicate notifications applied they would surely be visible when running icinga2 object list --name DK-MA-BRO-P013* --type Notification, right?

NO-MA-BLT-A001.txt (70.3 KB) DK-MA-BRO-P013.txt (58.1 KB)

enable_ha is explicitly set to true for the NotificationComponent, even though the documentation states that the default value is true.

Best regards

Yes, that is correct you will see duplication notification apply rules when running the command. I just confirmed that on my system.

Your notification interval is set to 0. Is this a correct configuration? You only want to receive one notification per state change and no addition notification if the problem is not repaired.

Did you enable the debug logs? Is there any more details in the debug that could help troubleshooting. I see you are running the latest version of Icinga. Maybe this is a bug in this version.

As you can see from my attached files I do not have any duplicate Notification objects.

Yes, only one notification is desired. This is why I think it’s strange that a “reminder” is being sent out. I was of the belief that specifying interval = 0 would disable any reminders.

Unfortunately I have not been able to enable debug logging as the issue is intermittent the logs would fill up the disk… I will investigate the possibility to enable debug logging for a shorter period of time and hopefully be able to “record” the problem.

Best regards

Update:

I’ve managed to run some debugging but I cannot see anything apparently “wrong” in the logs other that the reminder notification being triggered.

I did however notice something that might give some clues on how the reminder notifications appear but not on why they do.

Logs from both master servers and satellite for service IT-MA-AVA-A005!IVY Motion Detection A with duplicate notifications:

20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] notice/NotificationComponent: Attempting to send reminder notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api'.
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api'.
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] debug/Notification: User 'zendesk-digital-api' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api', Type 'Problem', TypeFilter: Custom, Problem and Recovery (FType=32, TypeFilter=96)
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] debug/Notification: User 'zendesk-digital-api' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api', State 'Critical', StateFilter: Critical, Down, OK, Up and Warning (FState=4, StateFilter=55)
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] information/Notification: Sending reminder 'Problem' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] notice/Process: Running command '/etc/icinga2/scripts/ZendeskAPIService.py' '-4' 'IT-MA-AVA-A005.xxx.yyy' '-6' '' '-b' '' '-c' '' '-d' '2020-05-29 13:56:40 +0200' '-e' 'IVY Motion Detection A' '-g' '' '-l' 'IT-MA-AVA-A005' '-n' 'IT-MA-AVA-A005' '-o' 'CRITICAL - Plugin timed out while executing system call' '-r' 'xxx@yyy.zzz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'IVY Motion Detection A' '-w' '': PID 28976
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] information/Notification: Completed sending 'Problem' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api' for checkable 'IT-MA-AVA-A005!IVY Motion Detection A' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
20200529135640-44 M1.log:[2020-05-29 13:56:40 +0200] debug/HttpUtility: Request body: '{"filter": "host.name==\"IT-MA-AVA-A005\" && service.name==\"IVY Motion Detection A\""}'
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] information/Checkable: Checkable 'IT-MA-AVA-A005!IVY Motion Detection A' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api'.
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] debug/Notification: User 'zendesk-digital-api' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api', Type 'Problem', TypeFilter: Custom, Problem and Recovery (FType=32, TypeFilter=96)
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] debug/Notification: User 'zendesk-digital-api' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api', State 'Critical', StateFilter: Critical, Down, OK, Up and Warning (FState=4, StateFilter=55)
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] information/Notification: Sending 'Problem' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] notice/Process: Running command '/etc/icinga2/scripts/ZendeskAPIService.py' '-4' 'IT-MA-AVA-A005.xxx.yyy' '-6' '' '-b' '' '-c' '' '-d' '2020-05-29 13:56:41 +0200' '-e' 'IVY Motion Detection A' '-g' '' '-l' 'IT-MA-AVA-A005' '-n' 'IT-MA-AVA-A005' '-o' 'CRITICAL - Plugin timed out while executing system call' '-r' 'xxx@yyy.zzz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'IVY Motion Detection A' '-w' '': PID 28978
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] information/Notification: Completed sending 'Problem' notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api' for checkable 'IT-MA-AVA-A005!IVY Motion Detection A' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] notice/ClusterEvents: Discarding 'send custom notification' message for checkable 'IT-MA-AVA-A005!IVY Motion Detection A' from 'nxicpxpr01.ad.xxx.yyy': Unauthorized access.
20200529135640-44 M1.log:[2020-05-29 13:56:41 +0200] debug/HttpUtility: Request body: '{"filter": "host.name==\"IT-MA-AVA-A005\" && service.name==\"IVY Motion Detection A\""}'
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/9e5b421a-e348-46ac-933c-207ee8584efa/conf.d/comments/IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c.conf
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] information/ConfigObjectUtility: Created and activated object 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c' of type 'Comment'.
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] notice/Comment: Added comment 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c'.
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] notice/Process: PID 28976 ('/etc/icinga2/scripts/ZendeskAPIService.py' '-4' 'IT-MA-AVA-A005.xxx.yyy' '-6' '' '-b' '' '-c' '' '-d' '2020-05-29 13:56:40 +0200' '-e' 'IVY Motion Detection A' '-g' '' '-l' 'IT-MA-AVA-A005' '-n' 'IT-MA-AVA-A005' '-o' 'CRITICAL - Plugin timed out while executing system call' '-r' 'xxx@yyy.zzz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'IVY Motion Detection A' '-w' '') terminated with exit code 0
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/9e5b421a-e348-46ac-933c-207ee8584efa/conf.d/comments/IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525.conf
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] information/ConfigObjectUtility: Created and activated object 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525' of type 'Comment'.
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] notice/Comment: Added comment 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525'.
20200529135640-44 M1.log:[2020-05-29 13:56:42 +0200] notice/Process: PID 28978 ('/etc/icinga2/scripts/ZendeskAPIService.py' '-4' 'IT-MA-AVA-A005.xxx.yyy' '-6' '' '-b' '' '-c' '' '-d' '2020-05-29 13:56:41 +0200' '-e' 'IVY Motion Detection A' '-g' '' '-l' 'IT-MA-AVA-A005' '-n' 'IT-MA-AVA-A005' '-o' 'CRITICAL - Plugin timed out while executing system call' '-r' 'xxx@yyy.zzz' '-s' 'CRITICAL' '-t' 'PROBLEM' '-u' 'IVY Motion Detection A' '-w' '') terminated with exit code 0
20200529135640-44 M2.log:[2020-05-29 13:56:40 +0200] notice/NotificationComponent: Reminder notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api': HA cluster active, this endpoint does not have the authority (paused=true). Skipping.
20200529135640-44 M2.log:[2020-05-29 13:56:41 +0200] information/Checkable: Checkable 'IT-MA-AVA-A005!IVY Motion Detection A' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
20200529135640-44 M2.log:[2020-05-29 13:56:41 +0200] notice/Notification: Notification 'IT-MA-AVA-A005!IVY Motion Detection A!zendesk-digital-service-alerts-api': HA cluster active, this endpoint does not have the authority (paused=true). Skipping.
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Received config update for object: {"config":"object Comment \"143e89ec-07e3-4e00-b744-81fee347e16c\" ignore_on_error {\n\tauthor = \"ZendeskIntegration\"\n\tentry_time = 1590753402.215275\n\tentry_type = 1.000000\n\texpire_time = 0.000000\n\thost_name = \"IT-MA-AVA-A005\"\n\tpersistent = false\n\tservice_name = \"IVY Motion Detection A\"\n\ttext = \"Zendesk ticket: <a href=\\\"https://xxx.zendesk.com/agent/tickets/128038\\\" target=\\\"_blank\\\">128038</a>\"\n\tversion = 1590753402.215293\n\tzone = \"nordic\"\n}\n","modified_attributes":{},"name":"IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c","original_attributes":[],"type":"Comment","version":1590753402.215293}
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/651cbe04-7b28-40d5-8de2-12bf0aa23aa4/conf.d/comments/IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c.conf
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] information/ConfigObjectUtility: Created and activated object 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c' of type 'Comment'.
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Discarding config update from 'cwicappr01.ad.xxx.yyy' (endpoint: 'cwicappr01.ad.xxx.yyy', zone: 'master') for object 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c': Object version 1590753402.215293 is more recent than the received version 1590753402.215293.
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Received config update for object: {"config":"object Comment \"80caed2f-905c-4956-ab27-71a4748cb525\" ignore_on_error {\n\tauthor = \"ZendeskIntegration\"\n\tentry_time = 1590753402.537013\n\tentry_type = 1.000000\n\texpire_time = 0.000000\n\thost_name = \"IT-MA-AVA-A005\"\n\tpersistent = false\n\tservice_name = \"IVY Motion Detection A\"\n\ttext = \"Zendesk ticket: <a href=\\\"https://xxx.zendesk.com/agent/tickets/128044\\\" target=\\\"_blank\\\">128044</a>\"\n\tversion = 1590753402.537038\n\tzone = \"nordic\"\n}\n","modified_attributes":{},"name":"IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525","original_attributes":[],"type":"Comment","version":1590753402.537038}
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/651cbe04-7b28-40d5-8de2-12bf0aa23aa4/conf.d/comments/IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525.conf
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] information/ConfigObjectUtility: Created and activated object 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525' of type 'Comment'.
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Discarding config update from 'cwicappr01.ad.xxx.yyy' (endpoint: 'cwicappr01.ad.xxx.yyy', zone: 'master') for object 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525': Object version 1590753402.537038 is more recent than the received version 1590753402.537038.
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] debug/IdoMysqlConnection: Query: UPDATE icinga_comments SET author_name = 'ZendeskIntegration',  comment_data = 'Zendesk ticket: <a href=\"https://xxx.zendesk.com/agent/tickets/128038\" target=\"_blank\">128038</a>',  comment_source = '1',  comment_time = FROM_UNIXTIME(1590753402),  comment_type = '1',  endpoint_object_id = 266,  entry_time = FROM_UNIXTIME(1590753402),  entry_time_usec = '215275',  entry_type = '1',  expires = '0',  instance_id = 1,  internal_comment_id = '1974',  is_persistent = '0',  name = 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c',  object_id = 34710,  session_token = 1590732244 WHERE entry_time = FROM_UNIXTIME(1590753402) AND name = 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c' AND object_id = 34710
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] debug/IdoMysqlConnection: Query: INSERT INTO icinga_commenthistory (author_name, comment_data, comment_source, comment_time, comment_type, endpoint_object_id, entry_time, entry_time_usec, entry_type, expires, instance_id, internal_comment_id, is_persistent, name, object_id) VALUES ('ZendeskIntegration', 'Zendesk ticket: <a href=\"https://xxx.zendesk.com/agent/tickets/128038\" target=\"_blank\">128038</a>', '1', FROM_UNIXTIME(1590753402), '1', 266, FROM_UNIXTIME(1590753402), '215275', '1', '0', 1, '1974', '0', 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c', 34710)
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] debug/IdoMysqlConnection: Query: UPDATE icinga_comments SET author_name = 'ZendeskIntegration',  comment_data = 'Zendesk ticket: <a href=\"https://xxx.zendesk.com/agent/tickets/128044\" target=\"_blank\">128044</a>',  comment_source = '1',  comment_time = FROM_UNIXTIME(1590753402),  comment_type = '1',  endpoint_object_id = 266,  entry_time = FROM_UNIXTIME(1590753402),  entry_time_usec = '537013',  entry_type = '1',  expires = '0',  instance_id = 1,  internal_comment_id = '1977',  is_persistent = '0',  name = 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525',  object_id = 34710,  session_token = 1590732244 WHERE entry_time = FROM_UNIXTIME(1590753402) AND name = 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525' AND object_id = 34710
20200529135640-44 M2.log:[2020-05-29 13:56:42 +0200] debug/IdoMysqlConnection: Query: INSERT INTO icinga_commenthistory (author_name, comment_data, comment_source, comment_time, comment_type, endpoint_object_id, entry_time, entry_time_usec, entry_type, expires, instance_id, internal_comment_id, is_persistent, name, object_id) VALUES ('ZendeskIntegration', 'Zendesk ticket: <a href=\"https://xxx.zendesk.com/agent/tickets/128044\" target=\"_blank\">128044</a>', '1', FROM_UNIXTIME(1590753402), '1', 266, FROM_UNIXTIME(1590753402), '537013', '1', '0', 1, '1977', '0', 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525', 34710)
20200529135640-44 M2.log:[2020-05-29 13:56:43 +0200] debug/IdoMysqlConnection: Query: DELETE FROM icinga_comments WHERE entry_time = FROM_UNIXTIME(1590753402) AND name = 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c' AND object_id = 34710
20200529135640-44 M2.log:[2020-05-29 13:56:43 +0200] debug/IdoMysqlConnection: Query: INSERT INTO icinga_comments (author_name, comment_data, comment_source, comment_time, comment_type, endpoint_object_id, entry_time, entry_time_usec, entry_type, expires, instance_id, internal_comment_id, is_persistent, name, object_id, session_token) VALUES ('ZendeskIntegration', 'Zendesk ticket: <a href=\"https://xxx.zendesk.com/agent/tickets/128038\" target=\"_blank\">128038</a>', '1', FROM_UNIXTIME(1590753402), '1', 266, FROM_UNIXTIME(1590753402), '215275', '1', '0', 1, '1974', '0', 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c', 34710, 1590732244)
20200529135640-44 M2.log:[2020-05-29 13:56:43 +0200] debug/IdoMysqlConnection: Query: DELETE FROM icinga_comments WHERE entry_time = FROM_UNIXTIME(1590753402) AND name = 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525' AND object_id = 34710
20200529135640-44 M2.log:[2020-05-29 13:56:43 +0200] debug/IdoMysqlConnection: Query: INSERT INTO icinga_comments (author_name, comment_data, comment_source, comment_time, comment_type, endpoint_object_id, entry_time, entry_time_usec, entry_type, expires, instance_id, internal_comment_id, is_persistent, name, object_id, session_token) VALUES ('ZendeskIntegration', 'Zendesk ticket: <a href=\"https://xxx.zendesk.com/agent/tickets/128044\" target=\"_blank\">128044</a>', '1', FROM_UNIXTIME(1590753402), '1', 266, FROM_UNIXTIME(1590753402), '537013', '1', '0', 1, '1977', '0', 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525', 34710, 1590732244)
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Received config update for object: {"config":"object Comment \"143e89ec-07e3-4e00-b744-81fee347e16c\" ignore_on_error {\n\tauthor = \"ZendeskIntegration\"\n\tentry_time = 1590753402.215275\n\tentry_type = 1.000000\n\texpire_time = 0.000000\n\thost_name = \"IT-MA-AVA-A005\"\n\tpersistent = false\n\tservice_name = \"IVY Motion Detection A\"\n\ttext = \"Zendesk ticket: <a href=\\\"https://xxx.zendesk.com/agent/tickets/128038\\\" target=\\\"_blank\\\">128038</a>\"\n\tversion = 1590753402.215293\n\tzone = \"nordic\"\n}\n","modified_attributes":{},"name":"IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c","original_attributes":[],"type":"Comment","version":1590753402.215293}
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/nxicpxpr01-1572356190-1/conf.d/comments/IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c.conf
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] information/ConfigObjectUtility: Created and activated object 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c' of type 'Comment'.
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Discarding config update from 'cwicappr01.ad.xxx.yyy' (endpoint: 'cwicappr01.ad.xxx.yyy', zone: 'master') for object 'IT-MA-AVA-A005!IVY Motion Detection A!143e89ec-07e3-4e00-b744-81fee347e16c': Object version 1590753402.215293 is more recent than the received version 1590753402.215293.
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Received config update for object: {"config":"object Comment \"80caed2f-905c-4956-ab27-71a4748cb525\" ignore_on_error {\n\tauthor = \"ZendeskIntegration\"\n\tentry_time = 1590753402.537013\n\tentry_type = 1.000000\n\texpire_time = 0.000000\n\thost_name = \"IT-MA-AVA-A005\"\n\tpersistent = false\n\tservice_name = \"IVY Motion Detection A\"\n\ttext = \"Zendesk ticket: <a href=\\\"https://xxx.zendesk.com/agent/tickets/128044\\\" target=\\\"_blank\\\">128044</a>\"\n\tversion = 1590753402.537038\n\tzone = \"nordic\"\n}\n","modified_attributes":{},"name":"IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525","original_attributes":[],"type":"Comment","version":1590753402.537038}
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] notice/ConfigCompiler: Compiling config file: /var/lib/icinga2/api/packages/_api/nxicpxpr01-1572356190-1/conf.d/comments/IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525.conf
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] information/ConfigObjectUtility: Created and activated object 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525' of type 'Comment'.
20200529135640-44 SAT.log:[2020-05-29 13:56:42 +0200] notice/ApiListener: Discarding config update from 'cwicappr01.ad.xxx.yyy' (endpoint: 'cwicappr01.ad.xxx.yyy', zone: 'master') for object 'IT-MA-AVA-A005!IVY Motion Detection A!80caed2f-905c-4956-ab27-71a4748cb525': Object version 1590753402.537038 is more recent than the received version 1590753402.537038.

History from host:

History from service:

Based on this my working hypothesis is that the host becomes DOWN/CRITICAL at 14:00 April 28, then the service becomes CRITICAL at 14:16 April 28, the notification of the service is blocked by the implicit dependency to the host (https://icinga.com/docs/icinga2/latest/doc/03-monitoring-basics/#implicit-dependencies-for-services-on-host).

The host becomes UP/OK at 13:56 May 29 while the service still is CRITICAL. Both the reminder and the correct notification of the service is triggered. So it seems to me that the notification that was suppressed at 14:16 April 28 is the one causing a reminder to be sent out.

Is this the intended behaviour?

Best regards