I can’t publish the output of the command it contains too much sensitive information, it would take hours to obfuscate the entire output. Which specific area do you require?
This notification section does not contain sensitive information.
Is that sufficient.
Object 'debug-file' of type 'FileLogger':
% declared in '/etc/icinga2/features-enabled/debuglog.conf', lines 7:1-7:30
* __name = "debug-file"
* name = "debug-file"
* package = "_etc"
* path = "/var/log/icinga2/debug.log"
% = modified in '/etc/icinga2/features-enabled/debuglog.conf', lines 9:3-9:30
* severity = "debug"
% = modified in '/etc/icinga2/features-enabled/debuglog.conf', lines 8:3-8:20
* source_location
* first_column = 1
* first_line = 7
* last_column = 30
* last_line = 7
* path = "/etc/icinga2/features-enabled/debuglog.conf"
* templates = [ "debug-file" ]
% = modified in '/etc/icinga2/features-enabled/debuglog.conf', lines 7:1-7:30
* type = "FileLogger"
* zone = ""
Object 'mail-service-notification' of type 'NotificationCommand':
% declared in '/etc/icinga2/conf.d/commands.conf', lines 68:1-68:54
* __name = "mail-service-notification"
* arguments
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 71:3-118:3
* -4 = "$notification_address$"
* -6 = "$notification_address6$"
* -b = "$notification_author$"
* -c = "$notification_comment$"
* -d
* required = true
* value = "$notification_date$"
* -e
* required = true
* value = "$notification_servicename$"
* -f
* description = "Set from address. Requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE)"
* value = "$notification_from$"
* -i = "$notification_icingaweb2url$"
* -l
* required = true
* value = "$notification_hostname$"
* -n
* required = true
* value = "$notification_hostdisplayname$"
* -o
* required = true
* value = "$notification_serviceoutput$"
* -r
* required = true
* value = "$notification_useremail$"
* -s
* required = true
* value = "$notification_servicestate$"
* -t
* required = true
* value = "$notification_type$"
* -u
* required = true
* value = "$notification_servicedisplayname$"
* -v = "$notification_logtosyslog$"
* command = [ "/etc/icinga2/scripts/mail-service-notification.sh" ]
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 69:3-69:67
* env = null
* execute
% = modified in 'methods-itl.conf', lines 23:3-23:30
* arguments = [ "notification", "user", "cr", "nr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
* deprecated = false
* name = "Internal#PluginNotification"
* side_effect_free = false
* type = "Function"
* name = "mail-service-notification"
* package = "_etc"
* source_location
* first_column = 1
* first_line = 68
* last_column = 54
* last_line = 68
* path = "/etc/icinga2/conf.d/commands.conf"
* templates = [ "mail-service-notification", "plugin-notification-command" ]
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 68:1-68:54
% = modified in 'methods-itl.conf', lines 22:2-22:122
* timeout = 60
* type = "NotificationCommand"
* vars
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 120:3-134:3
* notification_address = "$address$"
* notification_address6 = "$address6$"
* notification_author = "$notification.author$"
* notification_comment = "$notification.comment$"
* notification_date = "$icinga.long_date_time$"
* notification_hostdisplayname = "$host.display_name$"
* notification_hostname = "$host.name$"
* notification_servicedisplayname = "$service.display_name$"
* notification_servicename = "$service.name$"
* notification_serviceoutput = "$service.output$"
* notification_servicestate = "$service.state$"
* notification_type = "$notification.type$"
* notification_useremail = "$user.email$"
* zone = ""
Object 'mail-host-notification' of type 'NotificationCommand':
% declared in '/etc/icinga2/conf.d/commands.conf', lines 9:1-9:51
* __name = "mail-host-notification"
* arguments
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 12:3-51:3
* -4 = "$notification_address$"
* -6 = "$notification_address6$"
* -b = "$notification_author$"
* -c = "$notification_comment$"
* -d
* required = true
* value = "$notification_date$"
* -f
* description = "Set from address. Requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE)"
* value = "$notification_from$"
* -i = "$notification_icingaweb2url$"
* -l
* required = true
* value = "$notification_hostname$"
* -n
* required = true
* value = "$notification_hostdisplayname$"
* -o
* required = true
* value = "$notification_hostoutput$"
* -r
* required = true
* value = "$notification_useremail$"
* -s
* required = true
* value = "$notification_hoststate$"
* -t
* required = true
* value = "$notification_type$"
* -v = "$notification_logtosyslog$"
* command = [ "/etc/icinga2/scripts/mail-host-notification.sh" ]
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 10:3-10:64
* env = null
* execute
% = modified in 'methods-itl.conf', lines 23:3-23:30
* arguments = [ "notification", "user", "cr", "nr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
* deprecated = false
* name = "Internal#PluginNotification"
* side_effect_free = false
* type = "Function"
* name = "mail-host-notification"
* package = "_etc"
* source_location
* first_column = 1
* first_line = 9
* last_column = 51
* last_line = 9
* path = "/etc/icinga2/conf.d/commands.conf"
* templates = [ "mail-host-notification", "plugin-notification-command" ]
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 9:1-9:51
% = modified in 'methods-itl.conf', lines 22:2-22:122
* timeout = 60
* type = "NotificationCommand"
* vars
% = modified in '/etc/icinga2/conf.d/commands.conf', lines 53:3-65:3
* notification_address = "$address$"
* notification_address6 = "$address6$"
* notification_author = "$notification.author$"
* notification_comment = "$notification.comment$"
* notification_date = "$icinga.long_date_time$"
* notification_hostdisplayname = "$host.display_name$"
* notification_hostname = "$host.name$"
* notification_hostoutput = "$host.output$"
* notification_hoststate = "$host.state$"
* notification_type = "$notification.type$"
* notification_useremail = "$user.email$"
* zone = ""
Object 'notification' of type 'NotificationComponent':
% declared in '/etc/icinga2/features-enabled/notification.conf', lines 5:1-5:43
* __name = "notification"
* enable_ha = true
* name = "notification"
* package = "_etc"
* source_location
* first_column = 1
* first_line = 5
* last_column = 43
* last_line = 5
* path = "/etc/icinga2/features-enabled/notification.conf"
* templates = [ "notification" ]
% = modified in '/etc/icinga2/features-enabled/notification.conf', lines 5:1-5:43
* type = "NotificationComponent"
* zone = ""
I’ve filtered the debug log for the hostname and the word notification, I’ve found the following;
2020-01-27 08:57:20 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0', active_checks_enabled = '1', check_command = 'service-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '2', current_notification_number = '1', current_state = '2', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '0.031000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115440), last_hard_state = '0', last_hard_state_change = FROM_UNIXTIME(1580115186), last_notification = FROM_UNIXTIME(1580115186), last_state_change = FROM_UNIXTIME(1580115381), last_time_critical = FROM_UNIXTIME(1580115440), last_time_ok = FROM_UNIXTIME(1580115186), last_time_unknown = FROM_UNIXTIME(1580113895), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115500), next_notification = FROM_UNIXTIME(1580116066), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'SERVICE \"TermService\" CRITICAL NOT RUNNING ', passive_checks_enabled = '1', percent_state_change = '21.600000', perfdata = 'service=1;;;1;7', problem_has_been_acknowledged = '0', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 413, should_be_scheduled = '1', state_type = '0', status_update_time = FROM_UNIXTIME(1580115440) WHERE service_object_id = 413
[2020-01-27 08:57:51 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0', active_checks_enabled = '1', check_command = 'load-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '1', current_notification_number = '0', current_state = '0', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '1.858000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115471), last_hard_state = '0', last_hard_state_change = FROM_UNIXTIME(1580113984), last_state_change = FROM_UNIXTIME(1580113984), last_time_ok = FROM_UNIXTIME(1580115471), last_time_unknown = FROM_UNIXTIME(1580113855), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115767), next_notification = FROM_UNIXTIME(1580115711), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'LOAD OK 4.29708% ', passive_checks_enabled = '1', percent_state_change = '5.400000', perfdata = 'load=4.29708%;;;0;100', problem_has_been_acknowledged = '0', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 389, should_be_scheduled = '1', state_type = '1', status_update_time = FROM_UNIXTIME(1580115471) WHERE service_object_id = 389
[2020-01-27 08:57:53 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0', active_checks_enabled = '1', check_command = 'network-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '1', current_notification_number = '0', current_state = '0', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '1.896000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115473), last_hard_state = '0', last_hard_state_change = FROM_UNIXTIME(1580113998), last_notification = FROM_UNIXTIME(1580113998), last_state_change = FROM_UNIXTIME(1580113998), last_time_ok = FROM_UNIXTIME(1580115473), last_time_unknown = FROM_UNIXTIME(1580113895), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115772), next_notification = FROM_UNIXTIME(1580115711), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'NETWORK OK 5231B/s ', passive_checks_enabled = '1', percent_state_change = '5.400000', perfdata = 'network=5231B;;;0; network_in=4582B network_out=649B \'Npcap Loopback Adapter_in\'=0B \'Npcap Loopback Adapter_out\'=0B Ethernet_in=4582B Ethernet_out=649B', problem_has_been_acknowledged = '0', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 393, should_be_scheduled = '1', state_type = '1', status_update_time = FROM_UNIXTIME(1580115473) WHERE service_object_id = 393
[2020-01-27 08:57:54 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '1', active_checks_enabled = '1', check_command = 'service-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '1', current_notification_number = '0', current_state = '3', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '0.025000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115474), last_hard_state = '3', last_hard_state_change = FROM_UNIXTIME(1580113932), last_state_change = FROM_UNIXTIME(1580112271), last_time_unknown = FROM_UNIXTIME(1580115474), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115770), next_notification = FROM_UNIXTIME(1580115711), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'The specified service does not exist as an installed service.', passive_checks_enabled = '1', percent_state_change = '0', perfdata = '', problem_has_been_acknowledged = '1', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 399, should_be_scheduled = '1', state_type = '1', status_update_time = FROM_UNIXTIME(1580115474) WHERE service_object_id = 399
[2020-01-27 08:58:20 +0000] information/Checkable: Checkable 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2020-01-27 08:58:20 +0000] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service'.
[2020-01-27 08:58:20 +0000] debug/Notification: User 'MIS' notification 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service', Type 'Problem', TypeFilter: Acknowledgement, Custom, DowntimeEnd, DowntimeRemoved, DowntimeStart, FlappingEnd, FlappingStart, Problem and Recovery (FType=32, TypeFilter=108)
[2020-01-27 08:58:20 +0000] debug/Notification: User 'MIS' notification 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service', State 'Critical', StateFilter: Down and Up (FState=4, StateFilter=48)
[2020-01-27 08:58:20 +0000] notice/Notification: Not sending notifications for notification object 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service and user 'MIS': state 'Critical' does not match state filter: Down and Up.
[2020-01-27 08:58:20 +0000] notice/Notification: Notification object 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service': Filters for user 'MIS' not matched. Not sending notification.
[2020-01-27 08:58:20 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0', active_checks_enabled = '1', check_command = 'service-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '1', current_notification_number = '1', current_state = '2', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '0.031000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115500), last_hard_state = '2', last_hard_state_change = FROM_UNIXTIME(1580115500), last_notification = FROM_UNIXTIME(1580115186), last_state_change = FROM_UNIXTIME(1580115381), last_time_critical = FROM_UNIXTIME(1580115500), last_time_ok = FROM_UNIXTIME(1580115186), last_time_unknown = FROM_UNIXTIME(1580113895), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115800), next_notification = FROM_UNIXTIME(1580116066), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'SERVICE \"TermService\" CRITICAL NOT RUNNING ', passive_checks_enabled = '1', percent_state_change = '21.200000', perfdata = 'service=1;;;1;7', problem_has_been_acknowledged = '0', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 413, should_be_scheduled = '1', state_type = '1', status_update_time = FROM_UNIXTIME(1580115500) WHERE service_object_id = 413
[2020-01-27 08:59:20 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0', active_checks_enabled = '1', check_command = 'service-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '1', current_notification_number = '1', current_state = '2', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '0.068000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115559), last_hard_state = '2', last_hard_state_change = FROM_UNIXTIME(1580114663), last_notification = FROM_UNIXTIME(1580114663), last_state_change = FROM_UNIXTIME(1580114546), last_time_critical = FROM_UNIXTIME(1580115559), last_time_ok = FROM_UNIXTIME(1580114247), last_time_unknown = FROM_UNIXTIME(1580113890), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115849), next_notification = FROM_UNIXTIME(1580115563), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'SERVICE \"gpsvc\" CRITICAL NOT RUNNING ', passive_checks_enabled = '1', percent_state_change = '10.600000', perfdata = 'service=1;;;1;7', problem_has_been_acknowledged = '0', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 404, should_be_scheduled = '1', state_type = '1', status_update_time = FROM_UNIXTIME(1580115559) WHERE service_object_id = 404
[2020-01-27 08:59:25 +0000] notice/NotificationComponent: Attempting to send reminder notification 'w10.wrk.fqdn.com!winservice Windows Group Policy Client!MIS_MainNotification_Service'.
[2020-01-27 08:59:25 +0000] notice/Notification: Attempting to send reminder notifications of type 'Problem' for notification object 'w10.wrk.fqdn.com!winservice Windows Group Policy Client!MIS_MainNotification_Service'.
[2020-01-27 08:59:25 +0000] debug/Notification: User 'MIS' notification 'w10.wrk.fqdn.com!winservice Windows Group Policy Client!MIS_MainNotification_Service', Type 'Problem', TypeFilter: Acknowledgement, Custom, DowntimeEnd, DowntimeRemoved, DowntimeStart, FlappingEnd, FlappingStart, Problem and Recovery (FType=32, TypeFilter=108)
[2020-01-27 08:59:25 +0000] debug/Notification: User 'MIS' notification 'w10.wrk.fqdn.com!winservice Windows Group Policy Client!MIS_MainNotification_Service', State 'Critical', StateFilter: Down and Up (FState=4, StateFilter=48)
[2020-01-27 08:59:25 +0000] notice/Notification: Not reminder sending notifications for notification object 'w10.wrk.fqdn.com!winservice Windows Group Policy Client!MIS_MainNotification_Service and user 'MIS': state 'Critical' does not match state filter: Down and Up.
[2020-01-27 08:59:25 +0000] notice/Notification: Notification object 'w10.wrk.fqdn.com!winservice Windows Group Policy Client!MIS_MainNotification_Service': Filters for user 'MIS' not matched. Not sending notification.
[2020-01-27 09:00:11 +0000] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0', active_checks_enabled = '1', check_command = 'service-windows', check_source = 'w10.wrk.fqdn.com', check_type = '0', current_check_attempt = '1', current_notification_number = '0', current_state = '0', endpoint_object_id = 237, event_handler_enabled = '1', execution_time = '0.047000', flap_detection_enabled = '0', has_been_checked = '1', instance_id = 1, is_flapping = '0', is_reachable = '1', last_check = FROM_UNIXTIME(1580115611), last_hard_state = '0', last_hard_state_change = FROM_UNIXTIME(1580113901), last_state_change = FROM_UNIXTIME(1580113901), last_time_ok = FROM_UNIXTIME(1580115611), latency = '0', long_output = '', max_check_attempts = '3', next_check = FROM_UNIXTIME(1580115906), next_notification = FROM_UNIXTIME(1580115711), normal_check_interval = '5', notifications_enabled = '1', original_attributes = 'null', output = 'SERVICE \"ProfSvc\" OK RUNNING ', passive_checks_enabled = '1', percent_state_change = '5.300000', perfdata = 'service=4;;;1;7', problem_has_been_acknowledged = '0', process_performance_data = '1', retry_check_interval = '1', scheduled_downtime_depth = '0', service_object_id = 414, should_be_scheduled = '1', state_type = '1', status_update_time = FROM_UNIXTIME(1580115611) WHERE service_object_id = 414
As i already wrote:
in other words from your debug log:
notice/Notification: Not sending notifications for notification object 'w10.wrk.fqdn.com!winservice Windows Remote Desktop Services!MIS_MainNotification_Service and user 'MIS': state 'Critical' does not match state filter: Down and Up.
Yes I can see that in the log.
Does that mean I’m missing Down and UP, or is the Host notification impact the logic. The workflow is clear to me.
Adding all the notification States and Types to my user seems to have correctly generated a notification for the service going down.
[2020-01-27 10:02:21 +0000] information/Notification: Completed sending 'Problem' notification 'wrkstation.fqdn!winservice Windows Group Policy Client!MIS_MainNotification_Service' for checkable 'wrkstation.fqdn!winservice Windows Group Policy Client' and user 'MIS' using command 'mail-service-notification'.
Not sure why I didn’t get a recovery message mind.
Sorted the recovery…
I think that’s it resolved.
Thanks for all your help.
1 Like
hi @ctrmint ,
may i know what solved the issue.
Hi @NithyaR! Since this post is about 2 years old, you may want to post your specific issue in a new thread.