Hi there, I still seem to have a problem with a proper HA setup for Icinga2. I noticed that while deploying Notifications and Downtimes via the director (Setup is director managed, no local files used). It seems both masters seem to ignore each for zones as they think to have the autoritative version. Though only master01 should have autoritative version, right?
The setup is deployed via the official ansible-collection-icinga in combination with Linuxfabrik-Monitoring-Plugins.
For the monitoring-plugins part I am using their initial basket and noticed that assign where match("A*", host.vars.criticality) does not seem to be working correctly because the Infra-Team testuser is not applied to the hosts accordingly for Notifications on Host/Services. Also no notifications (also Custom / + forced is not working). As soon as I switch back to C/B/A only and change from a wildcard to hardcoded criticalities it starts working (though, still no visible user for hosts/services). But I have a feeling this is relied to the underlying sync problem.
I now noticed that a scheduled downtime is not applied to a host and only to 12 of 24 services of a host. Or another scenario where I have i.e. 10 hosts, the downtime is applied only to 3/10 hosts.
As soon as I stop the second master, everything seems to be read / applied currently a few minutes later. Full host and services coverage, the Notification user visible even with the wildcard applied.
Please let me know if I forgot a config needed here which is needed to drill down.
Thanks in advance for your help and time!
Passwords etc. are randomized or if I forgot, it does not matter since its a test cluster.
Setup Details per opening
- Version used (
icinga2 --version)- r2.14.5-1
- Operating System and version
- Ubuntu 22.04.5 LTS
- Enabled features (
icinga2 feature list)- BOTH MASTERS
Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker icingadb mainlog notification
- Icinga Web 2 version and modules (System - About)
- 2.12.4
- Config validation (
icinga2 daemon -C
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 7 NotificationCommands.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 Notification.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 Host.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 12 Downtimes.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 5 Zones.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 4 Endpoints.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 ApiUser.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 2 Users.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 799 CheckCommands.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 5 TimePeriods.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 25 ScheduledDowntimes.
[2025-05-25 12:34:40 +0200] information/ConfigItem: Instantiated 24 Services.
[2025-05-25 12:34:40 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-05-25 12:34:40 +0200] information/cli: Finished validating the configuration file(s).
- If you run multiple Icinga 2 instances, the
zones.conffile (oricinga2 object list --type Endpointandicinga2 object list --type Zone) from all affected nodes- see below
Ansible Playbook
Play
- hosts: icinga_masters
become: yes
tasks:
- ansible.builtin.include_role:
name: mrlesmithjr.mariadb-galera-cluster
apply:
tags: mariadb
tags: always
- ansible.builtin.include_role:
name: icinga.icinga.repos
- ansible.builtin.include_role:
name: icinga.icinga.icinga2
- ansible.builtin.include_role:
name: icinga.icinga.icingadb
- ansible.builtin.include_role:
name: icinga.icinga.icingadb_redis
- ansible.builtin.include_role:
name: icinga.icinga.icingaweb2
Group Vars
# Variables for mrlesmithjr.mariadb-galera-cluster
mariadb_version: "*"
mariadb_upgrade: true
galera_enable_mariadb_repo: false
mariadb_bind_address: 0.0.0.0
galera_cluster_bind_interface: "{{ ansible_default_ipv4['interface'] }}"
galera_cluster_name: icinga-cluster
galera_cluster_nodes_group: icinga_masters
mariadb_databases:
- name: icingadb
- name: icingaweb
- name: vspheredb
- name: "{{ icingaweb2_resources['director_db']['dbname'] }}"
mariadb_mysql_users:
- name: icingaweb
hosts:
- "%"
- "127.0.0.1"
- "::1"
- "localhost"
password: "{{ icingaweb2_db['password'] }}"
priv: "icingaweb.*:ALL"
- name: icingadb
hosts:
- "%"
- "127.0.0.1"
- "::1"
- "localhost"
password: "{{ icingadb_database_password }}"
priv: "icingadb.*:ALL"
- name: vspheredb
hosts:
- "%"
- "127.0.0.1"
- "::1"
- "localhost"
password: 12379KJDAHueu231
priv: "vspheredb.*:ALL"
- name: "{{ icingaweb2_resources['director_db']['username'] }}"
hosts:
- "%"
- "127.0.0.1"
- "::1"
- "localhost"
password: "{{ icingaweb2_resources['director_db']['password'] }}"
priv: "{{ icingaweb2_resources['director_db']['dbname'] }}.*:ALL"
mariadb_mysql_root_password: 12731UWA71233z7hd
# Variables for collection icinga role icinga2
icinga2_constants:
TicketSalt: "{{ icinga2_global_ticketsalt }}"
NodeName: "{{ inventory_hostname }}"
ZoneName: "main"
icinga2__api_root_password: 12837HJJAWDhd
icinga2__api_root_user: ApiRootUser
icinga2_confd: local.d
icinga2_purge_features: yes
icinga2_config_directories:
- zones.d/main/commands
- zones.d/main/hosts
- zones.d/main/services
- "{{ icinga2_confd }}"
icinga2_objects:
- name: "{{ icinga2__api_root_user }}"
type: ApiUser
file: "{{ icinga2_confd }}/api-users.conf"
password: "{{ icinga2__api_root_password }}"
permissions:
- "*"
# Variables for collection icinga role icingadb
icingadb_database_type: mysql
icingadb_database_host: localhost
icingadb_database_user: icingadb
icingadb_database_password: 981273HdkawjuZ
icingadb_database_import_schema: true
# Variables for collection icinga role icingaweb2
icingaweb2_db:
type: mysql
name: icingaweb
host: localhost
user: icingaweb
password: 71623Hawdaw
icingaweb2_db_import_schema: true
icingaweb2_admin_username: devadmin
icingaweb2_admin_password: 986354hdzWe
icingaweb2_authentication:
icingaweb2:
backend: db
resource: icingaweb2_db
icingaweb2_groups:
icingaweb2:
backend: db
resource: icingaweb2_db
icingaweb2_modules:
director:
enabled: true
source: package
import_schema: true
run_kickstart: true
kickstart:
config:
endpoint: "{{ inventory_hostname }}"
host: 127.0.0.1
username: "{{ icinga2__api_root_user }}"
password: "{{ icinga2__api_root_password }}"
config:
db:
resource: director_db
icingadb:
enabled: true
source: package
commandtransports:
instance01:
transport: api
host: 127.0.0.1
username: "{{ icinga2__api_root_user }}"
password: "{{ icinga2__api_root_password }}"
config:
icingadb:
resource: icingadb
redis:
tls: '0'
redis:
redis1:
host: 127.0.0.1
businessprocess:
enabled: true
source: package
icingaweb2_resources:
director_db:
type: db
db: mysql
host: localhost
dbname: directordb
username: director
password: 123456abcd
charset: utf8
icingadb:
type: db
db: mysql
host: localhost
dbname: icingadb
username: icingadb
password: "{{ icingadb_database_password }}"
charset: utf8mb4
Host vars icingamaster01
# Variables for collection icinga role icinga2
icinga2_features:
- name: icingadb
host: 127.0.0.1
- name: notification
- name: checker
- name: mainlog
- name: api
ca_host: none
cert_name: "{{ inventory_hostname }}"
force_newcert: no
accept_config: true
accept_commands: true
endpoints:
- name: NodeName
- name: "{{ groups['icinga_masters'][1] }}"
host: "{{ hostvars[groups['icinga_masters'][1]]['ansible_host'] }}"
- name: "{{ groups['icinga_satellites'][0] }}"
host: "{{ hostvars[groups['icinga_satellites'][0]]['ansible_host'] }}"
zones:
- name: ZoneName
endpoints:
- NodeName
- "{{ groups['icinga_masters'][1] }}"
- name: satellite
parent: ZoneName
endpoints:
- "{{ groups['icinga_satellites'][0] }}"
- name: global-templates
global: true
- name: director-global
global: true
Host vars icingamaster01
# Variables for collection icinga role icinga2
icinga2_features:
- name: icingadb
host: 127.0.0.1
- name: notification
- name: checker
- name: mainlog
- name: api
ca_host: "{{ groups['icinga_masters'][0] }}"
cert_name: "{{ inventory_hostname }}"
force_newcert: no
accept_config: true
accept_commands: true
endpoints:
- name: NodeName
- name: "{{ groups['icinga_masters'][0] }}"
- name: "{{ groups['icinga_satellites'][0] }}"
host: "{{ hostvars[groups['icinga_satellites'][0]]['ansible_host'] }}"
zones:
- name: ZoneName
endpoints:
- "{{ groups['icinga_masters'][0] }}"
- NodeName
- name: satellite
parent: ZoneName
endpoints:
- "{{ groups['icinga_satellites'][0] }}"
- name: global-templates
global: true
- name: director-global
global: true
Master01
/etc/icinga2/constants.conf
const PluginDir = "/usr/lib/nagios/plugins"
const ManubulonPluginDir = "/usr/lib/nagios/plugins"
const PluginContribDir = "/usr/lib/nagios/plugins"
const NodeName = "icingamaster01"
const ZoneName = "main"
const TicketSalt = "12345"
/etc/icinga2/zones.conf
object Endpoint NodeName {
}
object Endpoint "icingamaster02" {
host = "192.168.2.81"
}
object Endpoint "icingasatellite" {
host = "192.168.2.82"
}
object Zone ZoneName {
endpoints = [ NodeName, "icingamaster02", ]
}
object Zone "director-global" {
global = true
}
object Zone "global-templates" {
global = true
}
object Zone "satellite" {
parent = ZoneName
endpoints = [ "icingasatellite", ]
}
/etc/icinga2/features-enabled/api.conf
object ApiListener "api" {
accept_config = true
accept_commands = true
ticket_salt = TicketSalt
}
/var/lib/icinga2/icingadb.env
"db1afb3c781b8ab36b02a04d03f15d9636a9c6fc"
/etc/icinga2/local.d/api-users.conf
object ApiUser "ApiRootUser" {
password = "987654"
permissions = [ "*", ]
}
Master02
/etc/icinga2/constants.conf
const PluginDir = "/usr/lib/nagios/plugins"
const ManubulonPluginDir = "/usr/lib/nagios/plugins"
const PluginContribDir = "/usr/lib/nagios/plugins"
const NodeName = "icingamaster02"
const ZoneName = "main"
const TicketSalt = "12345"
/etc/icinga2/zones.conf
object Endpoint NodeName {
}
object Endpoint "icingamaster01" {
}
object Endpoint "icingasatellite" {
host = "192.168.2.82"
}
object Zone ZoneName {
endpoints = [ "icingamaster01", NodeName, ]
}
object Zone "director-global" {
global = true
}
object Zone "global-templates" {
global = true
}
object Zone "satellite" {
parent = ZoneName
endpoints = [ "icingasatellite", ]
}
/etc/icinga2/features-enabled/api.conf
object ApiListener "api" {
accept_config = true
accept_commands = true
ticket_salt = TicketSalt
}
/var/lib/icinga2/icingadb.env
"db1afb3c781b8ab36b02a04d03f15d9636a9c6fc"
/etc/icinga2/local.d/api-users.conf
object ApiUser "ApiRootUser" {
password = "987654"
permissions = [ "*", ]
}