Notification mail ignore some hosts in downtimes.conf

Service Monitoring
1

Hi,
I have configured the downtimes.conf file including all the hosts that must be excluded from the control during the backup phases, and it works correctly for most of the hosts except for three of them for which I continue to receive “Host xxxx is DOWN”, for the duration of the backup.

the content of the downtimes.conf file is that:

apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-05:30”
tuesday = “02:00-05:30”
wednesday = “02:00-05:30”
thursday = “02:00-05:30”
friday = “02:00-05:30”
saturday = “02:00-05:30”
sunday = “02:00-05:30”
}
assign where host.name == “HOST1”
}
apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-02:30”
tuesday = “02:00-02:30”
wednesday = “02:00-02:30”
thursday = “02:00-02:30”
friday = “02:00-02:30”
saturday = “02:00-02:30”
sunday = “02:00-02:30”
}
assign where host.name == “HOST2”
}
apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-03:45”
tuesday = “02:00-03:45”
wednesday = “02:00-03:45”
thursday = “02:00-03:45”
friday = “02:00-03:45”
saturday = “02:00-03:45”
sunday = “02:00-03:45”
}
assign where host.name == “HOST3”
}
apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-03:30”
tuesday = “02:00-03:30”
wednesday = “02:00-03:30”
thursday = “02:00-03:30”
friday = “02:00-03:30”
saturday = “02:00-03:30”
sunday = “02:00-03:30”
}
assign where host.name == “HOST4”
}
apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-02:20”
tuesday = “02:00-02:20”
wednesday = “02:00-02:20”
thursday = “02:00-02:20”
friday = “02:00-02:20”
saturday = “02:00-02:20”
sunday = “02:00-02:20”
}
assign where host.name == “HOST5”
}
apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-02:30”
tuesday = “02:00-02:30”
wednesday = “02:00-02:30”
thursday = “02:00-02:30”
friday = “02:00-02:30”
saturday = “02:00-02:30”
sunday = “02:00-02:30”
}
assign where host.name == “HOST6”
}
apply ScheduledDowntime “backup-downtime” to Service {
author = “user”
comment = “xxxxxxx”
ranges = {
monday = “02:00-02:25”
tuesday = “02:00-02:25”
wednesday = “02:00-02:25”
thursday = “02:00-02:25”
friday = “02:00-02:25”
saturday = “02:00-02:25”
sunday = “02:00-02:25”
}
assign where host.name == “HOST7”
}

Where am I doing wrong?

2

Can you see a scheduled downtime for the three hosts in icingaweb2?

3

Yes…

2019-07-09%2013_04_09-Window
2019-07-09 13_04_09-Window.png1367×543 43 KB

4

OK and by the host itself?

like this:

tempsnip

5

In the individual hosts I don’t have “Schedule downtime”

2019-07-09%2014_16_59-Window2

6

I think thats a matching problem. Your downtime does not match this host.
Typing error; mistake in host object/ display name concerning assign rule?

7

I don’t think this is the problem, because even for all the other hosts (except three) I don’t have scheduled downtime but it works correctly in fact notifications don’t arrive during the pre-established downtime.

8

In addition…when I run icinga2 daemon -C the result is:

[2019-07-12 08:45:06 +0200] information/ConfigItem: Instantiated 74 ScheduledDowntimes.

One for each host service :thinking:

If I manually added the scheduling via icingaweb in which file would the downtime setting be saved?

9

A ScheduledDowntime config object generates Downtime objects on demand at runtime, one for the next slot and once expired, the next and so on. The downtimes you’ll see in Icinga Web 2, are runtime downtimes - not comparable to ScheduledDowntime objects.

If it is easier to divide, mark a ScheduledDowntime as “recurring downtime” in your brain, while a “downtime” is just a runtime object.

Cheers,
Michael

10

In fact inside the folder /var/lib/icinga2/api/packages/_api/04b4c295-4f89-4f17-9ee3-3b0305be65ce/conf.d/downtimes I compared the two types of downtime relating to a service, the first created with the runtime and the second via icingaweb and the result is the following.

Runtime:
object Downtime “8234928d-5700-42c4-8550-ab130abf87b2” ignore_on_error {
author = “xxxxxxxx”
authoritative_zone = “xxxx.xxxxxx.com”
comment = “Scheduled downtime for backup”
config_owner = “xxxxxx!ping4!backup-downtime”
duration = 0.000000
end_time = 1562983200.000000
entry_time = 1562916768.215225
fixed = true
host_name = “xxxxxx”
scheduled_by = “xxxxxx!ping4!backup-downtime”
service_name = “ping4”
start_time = 1562976000.000000
triggered_by = “”
version = 1562916768.215256
}

Icingaweb:
object Downtime “65240cce-c974-4cc8-94be-6869d62f79d2” ignore_on_error {
author = “xxxxxxxx”
comment = “test”
config_owner = “”
duration = 0.000000
end_time = 1562983200.000000
entry_time = 1562917462.094900
fixed = true
host_name = “xxxxxx”
scheduled_by = “”
service_name = “ping4”
start_time = 1562976000.000000
triggered_by = “”
version = 1562917462.094921
}

Can it mean something that I have no authoritative_zone in the second object?

11

authoritative_zone is a special attribute for ScheduledDowntime generated objects, there’s no specific difference here.

In order to troubleshoot further, I’d suggest modifying the notification apply rule, command and script and add a parameter for passing the value of $host.downtime_depth$. If this is 0, you’ll know that the downtime wasn’t triggered (or none existed).

To dig deeper, I’d modify the notification script and query the REST API with some curl commands, e.g. the runtime state of /v1/objects/downtimes filtered by the host.name. In the event of the notification, this adds more details - do that for a host where only you get notified from though.

Cheers,
Michael

1 Like