Scheduled downtime not visible in Icinga Web 2

Icinga Web
1

Hi,

I’m running into an issue with scheduled downtimes atm.
After configuring a scheduled downtime in /etc/icinga2/zones.d/master/downtimes.conf on the config master and reloading the icinga2 daemon I can see the downtime in the object list on both master nodes (icinga2 object list --type ScheduledDowntime --name mydowntime).

In Icinga Web 2 though I only see the scheduled downtime on objects where the check source is the config master. For all other objects the downtime does not show up even under Overview -> Downtimes.

I’m running two master nodes with separate IDO DBs locally on each node. enable_ha is set to false on both nodes in the ido-mysql.conf.
The scheduled downtime object looks like this:

apply ScheduledDowntime "mydowntime" to Service {
  author = "Schedule"
  comment = "Some comment"

  fixed = true

  ranges = {
    monday    = "01:50-03:10"
    tuesday   = "01:50-03:10"
    wednesday = "01:50-03:10"
    thursday  = "01:50-03:10"
    friday    = "01:50-03:10"
    saturday  = "01:50-03:10"
    sunday    = "01:50-03:10"
  }

  assign where service.name == "Redacted Service"
}

Icinga 2 Version: 2.10.5 (was also present in 2.10.4, earlier version unknown)
Icinga Web 2 Version: 2.6.3

Update: Just upgraded to Icinga Web 2 Version 2.6.3, problem persists.

2

I don’t quite understand this explanation. Do you have additional screenshots on the specific runtime Downtime objects being generated from the SD rules?

Cheers,
Michael

3

Sure, I’ve replaced some values with dummies.
host01 and host02 are two hosts which each have the service service01 and this service has a ScheduledDowntime object mydowntime.
The resulting service host01!service01 has master01 (the config master) as the check source and host02!service01 has master02 as the check source.

In Icinga Web 2 I only see first type (in the Downtimes view and on the service objects), meaning a downtime on a service which has the config master node as the check source (host01!service01!mydowntime).
If the other master node is the check source, the downtime is not visible in Icinga Web 2 (host02!service01!mydowntime).
Last night the downtimes should have triggered for the first time but in the history I only see that they were triggered on the first type (where the config master is the check source), which could either mean I cannot see them because Icinga Web doesn’t know about their existence or that the Downtimes were not triggered at all. I’m still looking into that.

Object from the secondary master:

Object 'host01!service01!mydowntime' of type 'ScheduledDowntime':
  % declared in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * __name = "host01!service01!mydowntime"
  * author = "Schedule"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 2:3-2:21
  * child_options = "DowntimeNoChildren"
  * comment = "Scheduled downtime xyz"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 3:3-3:88
  * duration = 0
  * fixed = true
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 5:3-5:14
  * host_name = "host01"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * name = "mydowntime"
  * package = "_cluster"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * ranges
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 7:3-15:3
    * friday = "01:50-03:10"
    * monday = "01:50-03:10"
    * saturday = "01:50-03:10"
    * sunday = "01:50-03:10"
    * thursday = "01:50-03:10"
    * tuesday = "01:50-03:10"
    * wednesday = "01:50-03:10"
  * service_name = "service01"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 62
    * last_line = 1
    * path = "/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf"
  * templates = [ "mydowntime" ]
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * type = "ScheduledDowntime"
  * vars = null
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62

Object 'host02!service01!mydowntime' of type 'ScheduledDowntime':
  % declared in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * __name = "host02!service01!mydowntime"
  * author = "Schedule"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 2:3-2:21
  * child_options = "DowntimeNoChildren"
  * comment = "Scheduled downtime xyz)"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 3:3-3:88
  * duration = 0
  * fixed = true
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 5:3-5:14
  * host_name = "host02"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * name = "mydowntime"
  * package = "_cluster"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * ranges
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 7:3-15:3
    * friday = "01:50-03:10"
    * monday = "01:50-03:10"
    * saturday = "01:50-03:10"
    * sunday = "01:50-03:10"
    * thursday = "01:50-03:10"
    * tuesday = "01:50-03:10"
    * wednesday = "01:50-03:10"
  * service_name = "service01"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 62
    * last_line = 1
    * path = "/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf"
  * templates = [ "mydowntime" ]
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62
  * type = "ScheduledDowntime"
  * vars = null
  * zone = "master"
    % = modified in '/var/lib/icinga2/api/zones/master/_etc/scheduled_downtimes.conf', lines 1:0-1:62

host01:

grafik
grafik.png732×828 23.7 KB

host02 (note the missing downtime):

grafik
grafik.png775×790 21 KB

4

Does the config master accept config from the secondary master?

5

It shouldn’t, no. See the api.conf files:

config master:

object ApiListener "api" {
  accept_commands = true
  accept_config = false
  bind_host = "master1"
  bind_port = 5665

  ticket_salt = TicketSalt
}

secondary master:

object ApiListener "api" {
  accept_config = true
  accept_commands = true
  bind_host = "master2"
  bind_port = 5665
}
6

It won’t accept downtime/comment objects then, leaving them only active and visible on the secondary master. Try enabling this and re-checking the observed behaviour.

Cheers,
Michael

7

Oooh of course. I never made that connection. It’s working now.

So basically the ScheduledDowntime object definitions are synced to the secondary master on config reload but the secondary master enables them and needs to sync that back to the config master, right?

8

Close, but not exactly. The missing bit: A ScheduledDowntime object (static or via apply rule) creates a runtime Downtime object, the first from the matching ranges. This Downtime object is e.g. created by a SD on the secondary master being authoritative. The creation event triggers a runtime cluster sync, reaching the first master - this one says “no”, because of the global accept_config setting.

That being said, the configuration authority for ScheduledDowntimes is your config master. For runtime created (config) objects, each endpoint needs to accept configuration. Internally, Icinga treats Comments and Downtimes as config objects, for easier syncing and API access/filters.

Cheers,
Michael

1 Like
9

Good to know, thank you for the clarification.