Notifications fail occasionaly

leziri · March 23, 2026, 1:05pm

Hi!

I noticed that the last 3 days since the last patch some notifications did not work correctly and couldn’t find out the problem. May someone from the developers can explain this behaviour?

The problem is that at some notification events an OK transition is send and the previous CRITICAL notification is missing . Unfortunately I cannot reproduce the failure.

Here an example:

The critical event is triggered from the engine but the Notificaton Contact is set to None

The OK event is triggered with the correct Contact:

I am not sure if this proves that the filters a correct. However I rechecked the the filters of course but they are definitely correct. The service should trigger at both events (CRIT and OK)

I cloned the same notification rule responsible for this failure and did tests and they behaved all as expected correctly.

So can anyone direct me into the right direction or can someone explain me why the Contact at the CRITICAL event is set to None?

rpm -qa | grep icinga
icinga-php-incubator-0.23.0-1.el8.noarch
php-icinga-2.12.6-1.el8.noarch
vim-icinga2-2.15.2-1.el8.x86_64
icingaweb2-common-2.12.6-1.el8.noarch
icinga-director-daemon-1.11.6-1.el8.noarch
icinga-php-thirdparty-0.14.0-1.el8.noarch
icingadb-web-1.3.0-1.el8.noarch
icinga-selinux-common-1.0.0-1.el8.icinga.noarch
icinga2-ido-mysql-2.15.2-1.el8.x86_64
icinga-director-php-1.11.6-1.el8.noarch
icinga-php-library-0.18.1-1.el8.noarch
icingacli-2.12.6-1.el8.noarch
icingaweb2-selinux-2.12.6-1.el8.noarch
icinga2-2.15.2-1.el8.x86_64
icinga-director-1.11.6-1.el8.noarch
icingadb-redis-8.2.5-1.el8.x86_64
icinga2-bin-2.15.2-1.el8.x86_64
icinga2-common-2.15.2-1.el8.x86_64
icingadb-1.5.1-4.el8.x86_64
icingaweb2-2.12.6-1.el8.noarch
icinga-director-web-1.11.6-1.el8.noarch
icinga-l10n-1.3.0-1.el8.noarch
icinga2-selinux-2.15.2-1.el8.x86_64

apenning · March 23, 2026, 1:31pm

Thanks for posting and please excuse the inconvenience.

Just to be sure: This isn’t just an UI bug in Icinga Web 2, but there is actually no notification being sent for the critical service state, right?

Could you please share your Icinga 2 Notification object definition with us? Furthermore, can you check when the last critical service notification was sent to a notified contact?

You mentioned a patch update. What did you update?

Is there anything relevant in the Icinga 2 logs? How does the icinga2 daemon -C --dump-objects output look like?

leziri · March 23, 2026, 2:06pm

Just to be sure: This isn’t just an UI bug in Icinga Web 2, but there is actually no notification being sent for the critical service state, right?

Yes I checked the Inbox and the notifications are indeed missing.

Could you please share your Icinga 2 Notification object definition with us?

icinga2 object list --type Notification --name "*<SERVICENAME>*"
Object 'LB-JAVA FE_JAVA!HTTPS - <domain> - <SERVICENAME>!notification_MAIL-daily_24x7-<company>_group_ib' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 100:1-100:74
  * __name = "LB-JAVA FE_JAVA!HTTPS - <domain> - <SERVICENAME>!notification_MAIL-daily_24x7-<company>_group_ib"
  * command = "email_service_alarm"
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf', lines 10:5-10:35
  * command_endpoint = ""
  * host_name = "LB-JAVA FE_JAVA"
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 100:1-100:74
  * interval = 0
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf', lines 11:5-11:17
  * name = "notification_MAIL-daily_24x7-<company>_group_ib"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 100:1-100:74
  * period = "daily_24x7"
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf', lines 12:5-12:25
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 103:5-103:25
  * service_name = "HTTPS - <domain> - <SERVICENAME>"
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 100:1-100:74
  * source_location
    * first_column = 1
    * first_line = 100
    * last_column = 74
    * last_line = 100
    * path = "/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf"
  * states = [ "Critical", "OK", "Unknown", "Warning" ]
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf', lines 13:5-13:47
  * templates = [ "notification_MAIL-daily_24x7-<company>_group_ib", "MAIL Service Notification" ]
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 100:1-100:74
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf', lines 9:1-9:49
  * times = null
  * type = "Notification"
  * types = [ "Acknowledgement", "Custom", "Problem", "Recovery" ]
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf', lines 14:5-14:58
  * user_groups = [ "<company>_group_ib" ]
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 105:5-105:37
  * users = null
  * vars = null
  * zone = "<server>"
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 100:1-100:74
warning/cli: This data is 2 hours, 24 minutes and 30 seconds older than the last Icinga config (re)load. It may be outdated. Consider running 'icinga2 daemon -C --dump-objects' first.

Furthermore, can you check when the last critical service notification was sent to a notified contact?

In November 2025

You mentioned a patch update. What did you update?

dnf history info last | grep icinga
Upgrade icinga2-2.15.2-1.el8.x86_64 Upgraded icinga2-2.15.1-1.el8.x86_64 Upgrade icinga2-bin-2.15.2-1.el8.x86_64 Upgraded icinga2-bin-2.15.1-1.el8.x86_64 Upgrade icinga2-common-2.15.2-1.el8.x86_64 Upgraded icinga2-common-2.15.1-1.el8.x86_64 Upgrade icinga2-ido-mysql-2.15.2-1.el8.x86_64 Upgraded icinga2-ido-mysql-2.15.1-1.el8.x86_64 Upgrade icinga2-selinux-2.15.2-1.el8.x86_64 Upgraded icinga2-selinux-2.15.1-1.el8.x86_64 Upgrade icingadb-1.5.1-4.el8.x86_64 Upgraded icingadb-1.5.1-1.el8.x86_64 Upgrade icingadb-redis-8.2.5-1.el8.x86_64 Upgraded icingadb-redis-7.2.12-1.el8.x86_64 Upgrade icinga-director-1.11.6-1.el8.noarch Upgraded icinga-director-1.11.5-1.el8.noarch Upgrade icinga-director-daemon-1.11.6-1.el8.noarch Upgraded icinga-director-daemon-1.11.5-1.el8.noarch Upgrade icinga-director-php-1.11.6-1.el8.noarch Upgraded icinga-director-php-1.11.5-1.el8.noarch Upgrade icinga-director-web-1.11.6-1.el8.noarch Upgraded icinga-director-web-1.11.5-1.el8.noarch Upgrade vim-icinga2-2.15.2-1.el8.x86_64 Upgraded vim-icinga2-2.15.1-1.el8.x86_64 @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System @COMPANY_Icinga2_Icinga2_-RHEL8 @ @System

Is there anything relevant in the Icinga 2 logs? How does the icinga2 daemon -C --dump-objects output look like?

In the icinga.logs I observed that the “Completed Sending” msg is missing

>>>[2026-03-22 16:12:16 +0100] information/Checkable: Checkable ‘LB-JAVA FE_JAVA!HTTPS - ’ has 1 notification(s). Checking filters for type ‘Problem’, sends will be logged.
[2026-03-22 16:13:35 +0100] information/IcingaDB: Pending queries: 0 (Input: 20/s; Output: 20/s)
[2026-03-22 16:14:15 +0100] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2026-03-22 16:15:12 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 27.55/s (1653/min 8370/5min 25084/15min);
[2026-03-22 16:15:12 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2026-03-22 16:18:35 +0100] information/IcingaDB: Pending queries: 0 (Input: 20/s; Output: 20/s)
[2026-03-22 16:19:15 +0100] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’

[2026-03-22 16:19:25 +0100] information/Checkable: Checkable ‘LB-NAME FE-WEB!HTTPS -  - NAME GeoServer’ has 2 notification(s). Checking filters for type ‘Problem’, sends will be logged.
[2026-03-22 16:19:25 +0100] information/Notification: Sending ‘Problem’ notification ‘LB-NAME FE-WEB!HTTPS -  - NAME GeoServer!notification_MAIL-daily_24x7-_group_NAME’ for user ‘_user_NAME_monitoring’
[2026-03-22 16:19:25 +0100] information/Notification: Sending ‘Problem’ notification ‘LB-NAME FE-WEB!HTTPS -  - NAME GeoServer!notification_MAIL-daily_24x7-_group_ib’ for user ‘_user_ib’
[2026-03-22 16:19:25 +0100] information/Notification: Completed sending ‘Problem’ notification ‘LB-NAME FE-WEB!HTTPS -  - NAME GeoServer!notification_MAIL-daily_24x7-_group_NAME’ for checkable ‘LB-NAME FE-WEB!HTTPS -  - NAME GeoServer’ and user ‘_user_NAME_monitoring’ using command ‘email_service_alarm’.

apenning · March 23, 2026, 3:53pm

Thanks for the fast and detailed response. Could you please share the Director-generated Icinga 2 object from the following files with us:

/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf: The object around line 100 to 105.
/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_templates.conf: The object around line 9 to 14.

leziri:

  * user_groups = [ "<company>_group_ib" ]
    % = modified in '/var/lib/icinga2/api/packages/director/7900bcbb-0d00-4f88-aebb-f24491fed27d/zones.d/<server>/notification_apply.conf', lines 105:5-105:37

Is this the UserGroup receiving the messages?

Thanks for checking. I know, this is a few days back, but do you know by any chance what you did back then?

leziri · March 24, 2026, 6:52am

Thank you for your help!
Here the Objects you requested in detail and resolved:

object Service "<affectedSERVICE>" {
    host_name = "LB-JAVA FE_JAVA"
    import "Service HTTP(s)"

    groups = [
        "notification_MAIL-daily_24x7-<company>_group_ib"
    ]
    vars.http_sni = true
    vars.http_ssl = true
    vars.http_string = "Benutzer"
    vars.http_uri = "/xxxxx"
    vars.http_vhost = "xxxxx"
    vars.uuid_service = "5158afcb-124f-443e-9f3a-b4df6f2cf243"
}

apply Notification "notification_MAIL-daily_24x7-<company>_group_ib" to Service {
     import "MAIL Service Notification"
 
     period = "daily_24x7"
     assign where "notification_MAIL-daily_24x7-<company>_group_ib" in service.groups
     user_groups = [ "<company>_group_ib" ]
 }

template Notification "MAIL Service Notification" {
     command = "email_service_alarm"
     interval = 0s
     period = "daily_24x7"
     states = [ Critical, OK, Unknown, Warning ]
     types = [ Acknowledgement, Custom, Problem, Recovery ]
}

zones.d/director-global/timeperiods.conf
object TimePeriod "daily_24x7" {
    import "legacy-timeperiod"
    display_name = "daily_24x7"
    ranges = {
        "friday"	= "00:00-24:00"
        "monday"	= "00:00-24:00"
        "saturday"	= "00:00-24:00"
        "sunday"	= "00:00-24:00"
        "thursday"	= "00:00-24:00"
        "tuesday"	= "00:00-24:00"
        "wednesday"	= "00:00-24:00"
    }
}

object User "<company>_user_ib" {
    import "COMPANY User Template"

    display_name = "SECTION IB-Verteiler"
    email = "xxx@<domain>"
    groups = [ "<company>_group_ib" ]
}

Yes the members of the group recieve the message. The OK msg has been recieved by all members the CRITICAL as mentioned by no one.

Thanks for checking. I know, this is a few days back, but do you know by any chance what you did back then?

I will check the history of the single objects but I would say nothing has changed as far as i remember - I am the only person whos managing that server. Regarding updates I stick quite tight to the patch-cycles being published by netways and github.

apenning · March 24, 2026, 10:37am

At both first and second glance, your configuration seems valid.

Please enable the debug out / debuglog and capture the log messages when creating such a notification for this service. There should be some debug/Notification log entries indicating how the Notification should be routed.

The output should look like the following, which I just produced in a demo setup.

[2026-03-24 10:35:53 +0000] information/Checkable: Checkable 'docker-master' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2026-03-24 10:35:53 +0000] notice/Notification: Attempting to send notifications of type 'Problem' for notification object 'docker-master!mail-icingaadmin'.
[2026-03-24 10:35:53 +0000] debug/Notification: Type 'Problem', TypeFilter: Acknowledgement, Custom, DowntimeEnd, DowntimeRemoved, DowntimeStart, FlappingEnd, FlappingStart, Problem and Recovery (FType=32, TypeFilter=511)
[2026-03-24 10:35:53 +0000] debug/Notification: State 'Down', StateFilter: Down and Up (FState=32, StateFilter=48)
[2026-03-24 10:35:53 +0000] debug/Notification: User 'icingaadmin' notification 'docker-master!mail-icingaadmin', Type 'Problem', TypeFilter: Acknowledgement, Custom, DowntimeEnd, DowntimeRemoved, DowntimeStart, FlappingEnd, FlappingStart, Problem and Recovery (FType=32, TypeFilter=511)
[2026-03-24 10:35:53 +0000] debug/Notification: User 'icingaadmin' notification 'docker-master!mail-icingaadmin', State 'Down', StateFilter: Critical, Down, OK, Unknown, Up and Warning (FState=32, StateFilter=-1)
[2026-03-24 10:35:53 +0000] information/Notification: Sending 'Problem' notification 'docker-master!mail-icingaadmin' for user 'icingaadmin'
[2026-03-24 10:35:53 +0000] notice/Process: Running command '/data/etc/icinga2/scripts/mail-host-notification.sh' '-4' '127.0.0.1' '-6' '::1' '-X' '' '-b' '' '-c' '' '-d' '2026-03-24 10:35:53 +0000' '-l' 'docker-master' '-n' 'docker-master' '-o' 'RIP' '-r' 'icinga@localhost' '-s' 'DOWN' '-t' 'PROBLEM' '-v' 'false': PID 82074
[2026-03-24 10:35:53 +0000] information/Notification: Completed sending 'Problem' notification 'docker-master!mail-icingaadmin' for checkable 'docker-master' and user 'icingaadmin' using command 'mail-host-notification'.

leziri · March 24, 2026, 12:18pm

ok thank you very much for your support i will monitor this for a while and get back then. My colleagues will check this at the next maintenance window.