Service notifying every 30 mins although daily is configured

Dear Community,

I try it here first, maybe it turns out that this is related to director which we use as well…

Issue
There is a service “Updates - APT” checking critical updates and notifying us every 30 minutes, although configured to do it daily.
The service is applied through a host template “os_linux”.
It’s check_interval and retry_interval is set to 86400, it’s max_check_attempts is 1.

The object list of this service is somehow confirming my assumption (at least what I can see or how I understand that:

Object 'my-server!Updates - APT' of type 'Service':
  % declared in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
  * __name = "my-server!Updates - APT"
  * action_url = ""
  * check_command = "apt"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 437:5-437:25
  * check_interval = 86400
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 63:5-63:23
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 439:5-439:23
  * check_period = "24-7"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 62:5-62:25
  * check_timeout = null
  * command_endpoint = "my-server"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 441:5-441:32
  * display_name = "Updates - APT"
  * enable_active_checks = true
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 66:5-66:31
  * enable_event_handler = true
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 68:5-68:31
  * enable_flapping = false
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 69:5-69:27
  * enable_notifications = true
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 65:5-65:31
  * enable_passive_checks = true
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 67:5-67:32
  * enable_perfdata = true
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 70:5-70:26
  * event_command = ""
  * flapping_ignore_states = null
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "my-server"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 1
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 61:5-61:28
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 438:5-438:28
  * name = "Updates - APT"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
  * notes = ""
  * notes_url = ""
  * package = "_cluster"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
  * retry_interval = 86400
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 64:5-64:24
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 440:5-440:23
  * source_location
    * first_column = 1
    * first_line = 77
    * last_column = 29
    * last_line = 77
    * path = "/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"
  * templates = [ "Updates - APT", "Updates - APT", "service-base", "host var overrides (Director)" ]
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 434:1-434:32
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 60:1-60:31
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf', lines 28:3-28:43
  * type = "Service"
  * vars
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf', lines 41:7-41:51
    * apt_only_critical = true
      % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 442:5-442:33
  * volatile = false
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 71:5-71:20
  * zone = "hq"
    % = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29

Expected Behaviour
The service should be checked once each day, notifying directly on critical only (that part) works and should re-notify next day again if nothing changes.

AFAIU this is done by

  • having the check_interval set to 86400 seconds (= 24 hours), so in OK state icinga will check it daily
  • having the max_check_attempts set to 1, so icinga will notify immediately without any retry
  • having the retry_interval set to 86400 seconds (= 24 hours), so in Non-OK state icinga will check for a change after 1 day again

Environment

Master:
Linux vmmonitormaster 5.11.0-1022-azure #23~20.04.1-Ubuntu SMP Fri Nov 19 10:20:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 20.04.3 LTS (Focal Fossa)
  Kernel: Linux
  Kernel version: 5.11.0-1022-azure
  Architecture: x86_64

Build information:
  Compiler: GNU 9.3.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Disabled features: command compatlog debuglog elasticsearch gelf icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker graphite ido-pgsql mainlog notification
[2021-12-02 16:35:40 +0000] information/cli: Icinga application loader (version: r2.13.2-1)
[2021-12-02 16:35:40 +0000] information/cli: Loading configuration file(s).
[2021-12-02 16:35:40 +0000] information/ConfigItem: Committing config item(s).
[2021-12-02 16:35:40 +0000] information/ApiListener: My API identity: vmmonitormaster
[2021-12-02 16:35:40 +0000] warning/ApplyRule: Apply rule 'host-dependency' (in /var/lib/icinga2/api/packages/director/d1f9e3e4-aa30-4859-bfdc-6c351b7fdb7e/zones.d/director-global/dependency_apply.conf: 1:0-1:41) for type 'Dependency' does not match anywhere!
[2021-12-02 16:35:40 +0000] warning/ApplyRule: Apply rule 'PING' (in /var/lib/icinga2/api/packages/director/d1f9e3e4-aa30-4859-bfdc-6c351b7fdb7e/zones.d/director-global/service_apply.conf: 105:1-105:20) for type 'Service' does not match anywhere!
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 697 Notifications.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 8 Dependencies.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 5 HostGroups.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 67 Hosts.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 39 Zones.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 37 Endpoints.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 4 ApiUsers.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 249 CheckCommands.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 User.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 11 TimePeriods.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 630 Services.
[2021-12-02 16:35:40 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-12-02 16:35:40 +0000] information/cli: Finished validating the configuration file(s).

Satellite:
Linux vmmonitorhq 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.1-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 20.04.3 LTS (Focal Fossa)
  Kernel: Linux
  Kernel version: 5.4.0-58-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 9.3.0
  Build host: runner-hh8q3bz2-project-508-concurrent-0
  OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog
[2021-12-02 16:30:55 +0000] information/cli: Icinga application loader (version: r2.13.1-1)
[2021-12-02 16:30:55 +0000] information/cli: Loading configuration file(s).
[2021-12-02 16:30:55 +0000] information/ConfigItem: Committing config item(s).
[2021-12-02 16:30:55 +0000] information/ApiListener: My API identity: vmmonitorhq
[2021-12-02 16:30:55 +0000] warning/ApplyRule: Apply rule 'host-dependency' (in /var/lib/icinga2/api/zones/director-global/director/dependency_apply.conf: 1:0-1:41) for type 'Dependency' does not match anywhere!
[2021-12-02 16:30:55 +0000] warning/ApplyRule: Apply rule 'PING' (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 105:1-105:20) for type 'Service' does not match anywhere!
[2021-12-02 16:30:55 +0000] warning/ApplyRule: Apply rule 'PING' (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 117:1-117:20) for type 'Service' does not match anywhere!
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 8 Dependencies.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 5 HostGroups.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 59 Hosts.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 39 Zones.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 37 Endpoints.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 249 CheckCommands.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 11 TimePeriods.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 612 Services.
[2021-12-02 16:30:55 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-12-02 16:30:55 +0000] information/cli: Finished validating the configuration file(s).

Looking forward for your input.

Best,
Matthias

The notification interval defaults to 30min. You need to change this at your apply rule(s) to

interval = 24h

(Don’t know how to do this with the director)

1 Like

As Roland said, the notification interval is not the same as the check interval.
You need to go into your notification template (or your notification apply rule) and set interval = 0s there, so there is just a single notification sent.

This is not entirely correct. The retry interval only is used during soft states. As you have set max_check_attempts to 1 you don’t get any soft states and the check will always use the check_interval (makes no difference in this case though).
Click here for the passage in the docs

Also not completely correct. Setting this value to 1 only will make the service directly change into a HARD state. This does not necessarily mean “send notifications directly”, as notifications can also be configured to have a delay, their own interval (as you already experienced) and their own time periods.
Docs about Notifications: Monitoring Basics - Icinga 2

1 Like

Thanks @rsx and @log1c … my bad!