Dear Community,
I try it here first, maybe it turns out that this is related to director which we use as well…
Issue
There is a service “Updates - APT” checking critical updates and notifying us every 30 minutes, although configured to do it daily.
The service is applied through a host template “os_linux”.
It’s check_interval
and retry_interval
is set to 86400, it’s max_check_attempts
is 1.
The object list of this service is somehow confirming my assumption (at least what I can see or how I understand that:
Object 'my-server!Updates - APT' of type 'Service':
% declared in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
* __name = "my-server!Updates - APT"
* action_url = ""
* check_command = "apt"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 437:5-437:25
* check_interval = 86400
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 63:5-63:23
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 439:5-439:23
* check_period = "24-7"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 62:5-62:25
* check_timeout = null
* command_endpoint = "my-server"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 441:5-441:32
* display_name = "Updates - APT"
* enable_active_checks = true
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 66:5-66:31
* enable_event_handler = true
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 68:5-68:31
* enable_flapping = false
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 69:5-69:27
* enable_notifications = true
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 65:5-65:31
* enable_passive_checks = true
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 67:5-67:32
* enable_perfdata = true
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 70:5-70:26
* event_command = ""
* flapping_ignore_states = null
* flapping_threshold = 0
* flapping_threshold_high = 30
* flapping_threshold_low = 25
* groups = [ ]
* host_name = "my-server"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
* icon_image = ""
* icon_image_alt = ""
* max_check_attempts = 1
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 61:5-61:28
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 438:5-438:28
* name = "Updates - APT"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
* notes = ""
* notes_url = ""
* package = "_cluster"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
* retry_interval = 86400
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 64:5-64:24
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 440:5-440:23
* source_location
* first_column = 1
* first_line = 77
* last_column = 29
* last_line = 77
* path = "/var/lib/icinga2/api/zones/director-global/director/service_apply.conf"
* templates = [ "Updates - APT", "Updates - APT", "service-base", "host var overrides (Director)" ]
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 434:1-434:32
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 60:1-60:31
% = modified in '/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf', lines 28:3-28:43
* type = "Service"
* vars
% = modified in '/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf', lines 41:7-41:51
* apt_only_critical = true
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 442:5-442:33
* volatile = false
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf', lines 71:5-71:20
* zone = "hq"
% = modified in '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf', lines 77:1-77:29
Expected Behaviour
The service should be checked once each day, notifying directly on critical only (that part) works and should re-notify next day again if nothing changes.
AFAIU this is done by
- having the check_interval set to 86400 seconds (= 24 hours), so in OK state icinga will check it daily
- having the max_check_attempts set to 1, so icinga will notify immediately without any retry
- having the retry_interval set to 86400 seconds (= 24 hours), so in Non-OK state icinga will check for a change after 1 day again
Environment
Master:
Linux vmmonitormaster 5.11.0-1022-azure #23~20.04.1-Ubuntu SMP Fri Nov 19 10:20:52 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)
Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Ubuntu
Platform version: 20.04.3 LTS (Focal Fossa)
Kernel: Linux
Kernel version: 5.11.0-1022-azure
Architecture: x86_64
Build information:
Compiler: GNU 9.3.0
Build host: runner-hh8q3bz2-project-298-concurrent-0
OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Disabled features: command compatlog debuglog elasticsearch gelf icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker graphite ido-pgsql mainlog notification
[2021-12-02 16:35:40 +0000] information/cli: Icinga application loader (version: r2.13.2-1)
[2021-12-02 16:35:40 +0000] information/cli: Loading configuration file(s).
[2021-12-02 16:35:40 +0000] information/ConfigItem: Committing config item(s).
[2021-12-02 16:35:40 +0000] information/ApiListener: My API identity: vmmonitormaster
[2021-12-02 16:35:40 +0000] warning/ApplyRule: Apply rule 'host-dependency' (in /var/lib/icinga2/api/packages/director/d1f9e3e4-aa30-4859-bfdc-6c351b7fdb7e/zones.d/director-global/dependency_apply.conf: 1:0-1:41) for type 'Dependency' does not match anywhere!
[2021-12-02 16:35:40 +0000] warning/ApplyRule: Apply rule 'PING' (in /var/lib/icinga2/api/packages/director/d1f9e3e4-aa30-4859-bfdc-6c351b7fdb7e/zones.d/director-global/service_apply.conf: 105:1-105:20) for type 'Service' does not match anywhere!
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 697 Notifications.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 8 Dependencies.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 5 HostGroups.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 67 Hosts.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 39 Zones.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 37 Endpoints.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 4 ApiUsers.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 249 CheckCommands.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 1 User.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 11 TimePeriods.
[2021-12-02 16:35:40 +0000] information/ConfigItem: Instantiated 630 Services.
[2021-12-02 16:35:40 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-12-02 16:35:40 +0000] information/cli: Finished validating the configuration file(s).
Satellite:
Linux vmmonitorhq 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.1-1)
Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Ubuntu
Platform version: 20.04.3 LTS (Focal Fossa)
Kernel: Linux
Kernel version: 5.4.0-58-generic
Architecture: x86_64
Build information:
Compiler: GNU 9.3.0
Build host: runner-hh8q3bz2-project-508-concurrent-0
OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog
[2021-12-02 16:30:55 +0000] information/cli: Icinga application loader (version: r2.13.1-1)
[2021-12-02 16:30:55 +0000] information/cli: Loading configuration file(s).
[2021-12-02 16:30:55 +0000] information/ConfigItem: Committing config item(s).
[2021-12-02 16:30:55 +0000] information/ApiListener: My API identity: vmmonitorhq
[2021-12-02 16:30:55 +0000] warning/ApplyRule: Apply rule 'host-dependency' (in /var/lib/icinga2/api/zones/director-global/director/dependency_apply.conf: 1:0-1:41) for type 'Dependency' does not match anywhere!
[2021-12-02 16:30:55 +0000] warning/ApplyRule: Apply rule 'PING' (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 105:1-105:20) for type 'Service' does not match anywhere!
[2021-12-02 16:30:55 +0000] warning/ApplyRule: Apply rule 'PING' (in /var/lib/icinga2/api/zones/director-global/director/service_apply.conf: 117:1-117:20) for type 'Service' does not match anywhere!
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 8 Dependencies.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 5 HostGroups.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 59 Hosts.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 39 Zones.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 37 Endpoints.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 249 CheckCommands.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 11 TimePeriods.
[2021-12-02 16:30:55 +0000] information/ConfigItem: Instantiated 612 Services.
[2021-12-02 16:30:55 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-12-02 16:30:55 +0000] information/cli: Finished validating the configuration file(s).
Looking forward for your input.
Best,
Matthias