Multi Master apply config after sync issue

Hi,

i have installed a multi master setup. Most of the defined services are running fine. But some of the Services are remaining at “pending” state. They never get executed. All Templates and definitions are in global-zone and the master zone. All objects are synced to /var/lib/icinga2/api but the second master does not load some of them. If master1 is the only active, all services get executed, if master2 is the only active, the services are disappearing from icingaweb2.

The services in question cannot be queried on the second host:

  • master1:

icinga2 object list --type service -c

Found 236 Service objects

  • master2:

icinga2 object list --type service -c

Found 218 Service objects

The service objects missing are the ones, which are not checked.

For example, the built in cluster check:

master1:

Object ‘master2.localdomain.local!cluster’ of type ‘Service’:
% declared in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23

  • __name = “master2.localdomain.local!cluster”
  • action_url = “”
  • check_command = “cluster”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 3:3-3:27
  • check_interval = 5
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 4:3-4:21
  • check_period = “”
  • check_timeout = null
  • command_endpoint = “”
  • display_name = “cluster”
  • enable_active_checks = true
  • enable_event_handler = true
  • enable_flapping = false
  • enable_notifications = true
  • enable_passive_checks = true
  • enable_perfdata = true
  • event_command = “”
  • flapping_threshold = 0
  • flapping_threshold_high = 30
  • flapping_threshold_low = 25
  • groups =
  • host_name = “master2.localdomain.local”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23
  • icon_image = “”
  • icon_image_alt = “”
  • max_check_attempts = 3
  • name = “cluster”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23
  • notes = “”
  • notes_url = “”
  • package = “_etc”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23
  • retry_interval = 1
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 5:3-5:21
  • source_location
    • first_column = 1
    • first_line = 2
    • last_column = 23
    • last_line = 2
    • path = “/etc/icinga2/zones.d/global-templates/icinga/cluster.conf”
  • templates = [ “cluster” ]
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23
  • type = “Service”
  • vars = null
  • volatile = false
  • zone = “master”
    % = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23

master2:

icinga2 object list --name “cluster” --type service

cat /var/lib/icinga2/api/zones/global-templates/_etc/icinga/cluster.conf

apply Service “cluster” {
check_command = “cluster”
check_interval = 5s
retry_interval = 1s

assign where match(“icinga2-master”, host.vars.roles, MatchAny)
}

cat /var/log/icinga2/icinga2.log | grep -E “(cluster.conf|Apply)”
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//.timestamp
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/ansible/groups.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/default.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/groups/standard.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/icinga/cluster.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/linux/certificates.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/linux/default.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/network/ios.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/app.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/ffc.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/generic.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/notifications.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/timeperiods/sample.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones/global-templates’ (14343 Bytes). Received timestamp ‘2019-08-06 11:13:16 +0200’ (1565082796.126668), Current timestamp ‘1970-01-01 01:00:00 +0100’ (0.000000).
[2019-08-06 11:13:17 +0200] information/ApiListener: Syncing configuration files for global zone ‘global-templates’ to endpoint ‘master01.localdomain.local’.
[2019-08-06 11:14:01 +0200] information/ApiListener: Syncing configuration files for global zone ‘global-templates’ to endpoint ‘master01.localdomain.local’.
[2019-08-06 11:14:01 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//.timestamp
[2019-08-06 11:14:01 +0200] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones/global-templates’ (17 Bytes). Received timestamp ‘2019-08-06 11:14:01 +0200’ (1565082841.380516), Current timestamp ‘2019-08-06 11:13:16 +0200’ (1565082796.126668).

master1 zones.conf:

object Endpoint “master1.localdomain.local” {
}
object Endpoint “master2.localdomain.local” {
host = “xx.xx.xx.xx”
}
object Zone “master” {
endpoints = [ “master1.localdomain.local”, “master2.localdomain.local” ]
}
object Zone “global-templates” {
global = true
}

master2 zones.conf:

object Endpoint “master1.localdomain.local” {
host = “xx.xx.xx.xx”
}
object Endpoint “master2.localdomain.local” {
}
object Zone “master” {
endpoints = [ “master1.localdomain.local”, “master2.localdomain.local” ]
}
object Zone “global-templates” {
global = true
}

Distribution: CentOS7
Icinga2 Version: 2.10.5

I don’t know why there are single objects, that are not synced. Hosts can communicate and both are executing checks and syncing services. Any idea would be appreciated.

Regards

Problem could be solved. I had used the path_exists() Function in my configuration. That paths are not present on the second master, as they are referring to local config files. So the returnvalue was different on masters hosts.

Just curious - why would you use path_exists() in your config?

to import a template, if it exists.

had some sort of following snippet

if ( path_exists(/etc/…/templates/templatename.conf) ) {
import “templatename”
}

Interesting approach. That implies that you generate your configuration somehow, can you provide a little more details on your use case? Maybe that can be turned into a general feature.

Host object and template are supposed to “merge” vars from 2 different sources. That way passwords does not have to be stored together in a widely accessible repository. Second reason is some Host vars doesn’t change that often, while others do. Files are both generated.

But i am able to do it at time of generation. Additional time is needed for config deployment, as Icinga2 DSL is a lot faster than ansible scripts. As number of hosts grow, this becomes noticeable.

Tried to leverage get_object(), but that seems to be a dead end, too.

1 Like

Interesting, thanks for sharing. A similar question was asked this week already, so I was wondering which intentions were behind this.

You would only want to check for the existence of a given template, but none of its attributes, right? Then this would likely be possible as a DSL feature.

Yes, thats exactly what i want to achieve. Thanks for your interest!