I have encountered a strange issue where some notifications are triggered twice. After looking for patterns in the log files it seems to me that the issue have something to do with the fact that a reminder is triggered first, then the “correct” notification is triggered.
Icinga2 environment:
master1 ↔ master2
master1 and master2 are both configured to handle notifications.
version: r2.11.3-1Master1
Disabled features: command compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql influxdb mainlog notificationMaster 2
Disabled features: command compatlog debuglog elasticsearch gelf graphite livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql influxdb mainlog notification
Notification object for Service ‘DK-MA-BRO-P013!CPU’.
Object 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' of type 'Notification':
% declared in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
* __name = "DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api"
* command = "api-service-zendesk"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 19:5-19:35
* command_endpoint = ""
* host_name = "DK-MA-BRO-P013"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
* interval = 0
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 14:5-14:17
* name = "zendesk-digital-service-alerts-api"
* package = "_cluster"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
* period = ""
* service_name = "CPU"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
* source_location
* first_column = 1
* first_line = 11
* last_column = 66
* last_line = 11
* path = "/var/lib/icinga2/api/zones/master/director/notification_apply.conf"
* states = [ "Critical", "OK", "Warning" ]
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 16:5-16:38
* templates = [ "zendesk-digital-service-alerts-api", "Zendesk Service Notification Template" ]
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 18:1-18:61
* times = null
* type = "Notification"
* types = [ "FlappingEnd", "FlappingStart", "Problem", "Recovery" ]
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 17:5-17:61
* user_groups = null
* users = [ "zendesk-digital-api" ]
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 18:5-18:37
* vars
* notification_address = "$address$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 20:5-20:43
* notification_address6 = "$address6$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 21:5-21:45
* notification_author = "$notification.author$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 22:5-22:54
* notification_comment = "$notification.comment$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 23:5-23:56
* notification_date = "$icinga.long_date_time$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 24:5-24:54
* notification_doclink = "$host.notes_url$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 25:5-25:50
* notification_hostdisplayname = "$host.display_name$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 26:5-26:61
* notification_hostname = "$host.name$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 27:5-27:46
* notification_notes = "$host.notes$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 28:5-28:44
* notification_servicedisplayname = "$service.display_name$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 29:5-29:67
* notification_servicename = "$service.name$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 30:5-30:52
* notification_serviceoutput = "$service.output$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 31:5-31:56
* notification_servicestate = "$service.state$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 32:5-32:54
* notification_type = "$notification.type$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 33:5-33:50
* notification_useremail = "$user.email$"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_templates.conf', lines 34:5-34:48
* zone = "nordic"
% = modified in '/var/lib/icinga2/api/zones/master/director/notification_apply.conf', lines 11:1-11:66
Log files from master servers:
Notification for ‘DK-MA-BRO-P013!CPU’ triggered twice:
Master1
[2020-05-07 22:33:49 +0200] information/Checkable: Checkable 'DK-MA-BRO-P013!CPU' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
Master2
[2020-05-07 22:33:49 +0200] information/Notification: Sending reminder 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 22:33:49 +0200] information/Notification: Completed sending 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for checkable 'DK-MA-BRO-P013!CPU' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
[2020-05-07 22:33:49 +0200] information/Checkable: Checkable 'DK-MA-BRO-P013!CPU' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-05-07 22:33:49 +0200] information/Notification: Sending 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 22:33:49 +0200] information/Notification: Completed sending 'Problem' notification 'DK-MA-BRO-P013!CPU!zendesk-digital-service-alerts-api' for checkable 'DK-MA-BRO-P013!CPU' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
Notification for ‘SE-RO-LUN-P003!Memory’ triggered twice:
Master1
[2020-05-07 08:32:13 +0200] information/Checkable: Checkable 'SE-RO-LUN-P003!Memory' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
Master2
[2020-05-07 08:32:13 +0200] information/Notification: Sending reminder 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 08:32:13 +0200] information/Notification: Completed sending 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for checkable 'SE-RO-LUN-P003!Memory' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
[2020-05-07 08:32:13 +0200] information/Checkable: Checkable 'SE-RO-LUN-P003!Memory' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-05-07 08:32:13 +0200] information/Notification: Sending 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 08:32:13 +0200] information/Notification: Completed sending 'Problem' notification 'SE-RO-LUN-P003!Memory!zendesk-digital-service-alerts-api' for checkable 'SE-RO-LUN-P003!Memory' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
Notification for ‘NO-MA-BLT-A001!IVY Motion Detection A’ were not triggered twice:
Master1
[2020-05-07 06:30:00 +0200] information/Checkable: Checkable 'NO-MA-BLT-A001!IVY Motion Detection A' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-05-07 06:30:00 +0200] information/Notification: Sending 'Problem' notification 'NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api' for user 'zendesk-digital-api'
[2020-05-07 06:30:00 +0200] information/Notification: Completed sending 'Problem' notification 'NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api' for checkable 'NO-MA-BLT-A001!IVY Motion Detection A' and user 'zendesk-digital-api' using command 'api-service-zendesk'.
Master2
[2020-05-07 06:30:00 +0200] information/Checkable: Checkable 'NO-MA-BLT-A001!IVY Motion Detection A' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
Notification object for ‘NO-MA-BLT-A001!IVY Motion Detection A’:
Object 'NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api' of type 'Notification':
% declared in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
* __name = "NO-MA-BLT-A001!IVY Motion Detection A!zendesk-digital-service-alerts-api"
* command = "api-service-zendesk"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 19:5-19:35
* command_endpoint = ""
* host_name = "NO-MA-BLT-A001"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
* interval = 0
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 14:5-14:17
* name = "zendesk-digital-service-alerts-api"
* package = "director"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
* period = ""
* service_name = "IVY Motion Detection A"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
* source_location
* first_column = 1
* first_line = 11
* last_column = 66
* last_line = 11
* path = "/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf"
* states = [ "Critical", "OK", "Warning" ]
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 16:5-16:38
* templates = [ "zendesk-digital-service-alerts-api", "Zendesk Service Notification Template" ]
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 18:1-18:61
* times = null
* type = "Notification"
* types = [ "FlappingEnd", "FlappingStart", "Problem", "Recovery" ]
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 17:5-17:61
* user_groups = null
* users = [ "zendesk-digital-api" ]
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 18:5-18:37
* vars
* notification_address = "$address$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 20:5-20:43
* notification_address6 = "$address6$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 21:5-21:45
* notification_author = "$notification.author$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 22:5-22:54
* notification_comment = "$notification.comment$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 23:5-23:56
* notification_date = "$icinga.long_date_time$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 24:5-24:54
* notification_doclink = "$host.notes_url$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 25:5-25:50
* notification_hostdisplayname = "$host.display_name$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 26:5-26:61
* notification_hostname = "$host.name$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 27:5-27:46
* notification_notes = "$host.notes$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 28:5-28:44
* notification_servicedisplayname = "$service.display_name$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 29:5-29:67
* notification_servicename = "$service.name$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 30:5-30:52
* notification_serviceoutput = "$service.output$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 31:5-31:56
* notification_servicestate = "$service.state$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 32:5-32:54
* notification_type = "$notification.type$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 33:5-33:50
* notification_useremail = "$user.email$"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_templates.conf', lines 34:5-34:48
* zone = "nordic"
% = modified in '/var/lib/icinga2/api/packages/director/0b0705c4-455d-4346-b176-f8c3497a6b84/zones.d/master/notification_apply.conf', lines 11:1-11:66
Looking for help troubleshooting the issue further so all input is appreciated.
Best regards