Hi,
i have installed a multi master setup. Most of the defined services are running fine. But some of the Services are remaining at “pending” state. They never get executed. All Templates and definitions are in global-zone and the master zone. All objects are synced to /var/lib/icinga2/api but the second master does not load some of them. If master1 is the only active, all services get executed, if master2 is the only active, the services are disappearing from icingaweb2.
The services in question cannot be queried on the second host:
- master1:
icinga2 object list --type service -c
…
Found 236 Service objects
- master2:
icinga2 object list --type service -c
…
Found 218 Service objects
The service objects missing are the ones, which are not checked.
For example, the built in cluster check:
master1:
Object ‘master2.localdomain.local!cluster’ of type ‘Service’:
% declared in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23
- __name = “master2.localdomain.local!cluster”
- action_url = “”
- check_command = “cluster”
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 3:3-3:27- check_interval = 5
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 4:3-4:21- check_period = “”
- check_timeout = null
- command_endpoint = “”
- display_name = “cluster”
- enable_active_checks = true
- enable_event_handler = true
- enable_flapping = false
- enable_notifications = true
- enable_passive_checks = true
- enable_perfdata = true
- event_command = “”
- flapping_threshold = 0
- flapping_threshold_high = 30
- flapping_threshold_low = 25
- groups =
- host_name = “master2.localdomain.local”
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23- icon_image = “”
- icon_image_alt = “”
- max_check_attempts = 3
- name = “cluster”
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23- notes = “”
- notes_url = “”
- package = “_etc”
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23- retry_interval = 1
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 5:3-5:21- source_location
- first_column = 1
- first_line = 2
- last_column = 23
- last_line = 2
- path = “/etc/icinga2/zones.d/global-templates/icinga/cluster.conf”
- templates = [ “cluster” ]
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23- type = “Service”
- vars = null
- volatile = false
- zone = “master”
% = modified in ‘/etc/icinga2/zones.d/global-templates/icinga/cluster.conf’, lines 2:1-2:23
master2:
icinga2 object list --name “cluster” --type service
cat /var/lib/icinga2/api/zones/global-templates/_etc/icinga/cluster.conf
apply Service “cluster” {
check_command = “cluster”
check_interval = 5s
retry_interval = 1sassign where match(“icinga2-master”, host.vars.roles, MatchAny)
}
…cat /var/log/icinga2/icinga2.log | grep -E “(cluster.conf|Apply)”
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//.timestamp
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/ansible/groups.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/default.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/groups/standard.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/icinga/cluster.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/linux/certificates.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/linux/default.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/network/ios.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/app.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/ffc.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/generic.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates/notifications.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/timeperiods/sample.conf
[2019-08-06 11:13:16 +0200] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones/global-templates’ (14343 Bytes). Received timestamp ‘2019-08-06 11:13:16 +0200’ (1565082796.126668), Current timestamp ‘1970-01-01 01:00:00 +0100’ (0.000000).
[2019-08-06 11:13:17 +0200] information/ApiListener: Syncing configuration files for global zone ‘global-templates’ to endpoint ‘master01.localdomain.local’.
[2019-08-06 11:14:01 +0200] information/ApiListener: Syncing configuration files for global zone ‘global-templates’ to endpoint ‘master01.localdomain.local’.
[2019-08-06 11:14:01 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//.timestamp
[2019-08-06 11:14:01 +0200] information/ApiListener: Applying configuration file update for path ‘/var/lib/icinga2/api/zones/global-templates’ (17 Bytes). Received timestamp ‘2019-08-06 11:14:01 +0200’ (1565082841.380516), Current timestamp ‘2019-08-06 11:13:16 +0200’ (1565082796.126668).
…
master1 zones.conf:
object Endpoint “master1.localdomain.local” {
}
object Endpoint “master2.localdomain.local” {
host = “xx.xx.xx.xx”
}
object Zone “master” {
endpoints = [ “master1.localdomain.local”, “master2.localdomain.local” ]
}
object Zone “global-templates” {
global = true
}
master2 zones.conf:
object Endpoint “master1.localdomain.local” {
host = “xx.xx.xx.xx”
}
object Endpoint “master2.localdomain.local” {
}
object Zone “master” {
endpoints = [ “master1.localdomain.local”, “master2.localdomain.local” ]
}
object Zone “global-templates” {
global = true
}
Distribution: CentOS7
Icinga2 Version: 2.10.5
I don’t know why there are single objects, that are not synced. Hosts can communicate and both are executing checks and syncing services. Any idea would be appreciated.
Regards