Both masters are connected to the same database via a Galera cluster.
What exactly do you mean by enabling HA? As far as I know, there’s nothing specific to “start” HA in Icinga.Right ? or im wrong ?
Could you clarify if there’s something particular you’re referring to?
Some features require enable_ha set to true or they run in split brain.
$ grep 'enable_ha' /etc/icinga2/features-enabled/*
/etc/icinga2/features-enabled/gelf.conf: enable_ha = true
/etc/icinga2/features-enabled/ido-mysql.conf: enable_ha = true
/etc/icinga2/features-enabled/influxdb.conf: enable_ha = true
master1
# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch graphite influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker gelf icingadb ido-mysql influxdb mainlog notification
master2
# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch graphite influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker gelf icingadb ido-mysql influxdb mainlog notification
Does your second master accept config and commands?
Both of my /etc/icinga2/features-enabled/api.conf look like this:
/**
* The API listener is used for distributed monitoring setups.
*/
object ApiListener "api" {
accept_config = true
accept_commands = true
ticket_salt = TicketSalt
}
Maybe you want to investigate setting this up via Ansible?
Mine are setup with this role:
Its my config :
Master 1 :
API :
/**
* The API listener is used for distributed monitoring setups.
*/
object ApiListener "api" {
accept_config = true
accept_commands = true
ticket_salt = TicketSalt
}
influxdb :
/**
* The InfluxdbWriter type writes check result metrics and
* performance data to an InfluxDB HTTP API
*/
object InfluxdbWriter "influxdb" {
host = "127.0.0.1"
port = 8086
database = "icinga"
flush_threshold = 1024
flush_interval = 10s
host_template = {
measurement = "$host.check_command$"
tags = {
hostname = "$host.name$"
}
}
service_template = {
measurement = "$service.check_command$"
tags = {
hostname = "$host.name$"
service = "$service.name$"
}
}
enable_ha = true
}
Ido-mysql :
/**
* The IdoMysqlConnection type implements MySQL support
* for DB IDO.
*/
object IdoMysqlConnection "ido-mysql" {
user = "icinga"
password = "adb7f1558c63c890fcef"
host = "localhost"
database = "icinga"
enable_ha = true
}
Feature list :
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker command ido-mysql influxdb mainlog notification
Master2 :
api.conf:
/**
* The API listener is used for distributed monitoring setups.
*/
object ApiListener "api" {
accept_config = true
accept_commands = true
ticket_salt = TicketSalt
}
influxdb.conf:
/**
* The InfluxdbWriter type writes check result metrics and
* performance data to an InfluxDB v1 HTTP API
*/
object InfluxdbWriter "influxdb" {
host = "127.0.0.1"
port = 8086
database = "icinga"
flush_threshold = 1024
flush_interval = 10s
host_template = {
measurement = "$host.check_command$"
tags = {
hostname = "$host.name$"
}
}
service_template = {
measurement = "$service.check_command$"
tags = {
hostname = "$host.name$"
service = "$service.name$"
}
}
enable_ha = true
}
ido-mysql.conf:
/**
* The IdoMysqlConnection type implements MySQL support
* for DB IDO.
*/
object IdoMysqlConnection "ido-mysql" {
user = "icinga"
password = "adb7f1558c63c890fcef"
host = "localhost"
database = "icinga"
enable_ha = true
}
feature list :
Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 journald livestatus notification opentsdb perfdata syslog
Enabled features: api checker command ido-mysql influxdb mainlog
I just need to mention something: the first master didn’t have a database named icingaweb2, but the second one did. I added this database to the first master as well. I’ll also share the screenshots of the resources from both Master 1 and Master 2.
Master1 :
Master2 :
This is ido log :
[2024-11-22 11:21:41 +0100] information/DbConnection: ‘ido-mysql’ started.
[2024-11-22 11:26:11 +0100] information/IdoMysqlConnection: ‘ido-mysql’ resumed.
[2024-11-22 11:26:11 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2024-11-22 11:26:31 +0100] information/IdoMysqlConnection: Last update by endpoint ‘smon02’ was 36.7798s ago. Taking over ‘ido-mysql’ in HA zone ‘smon’.
[2024-11-22 11:26:32 +0100] information/IdoMysqlConnection: Finished reconnecting to ‘ido-mysql’ database ‘icinga’ in 0.819029 second(s).
But didint work again !
At least the IdoMysqlConnection isn’t looking like both nodes icinga and/or Galera are communicating properly. Can you try with a single MySQL DB on a external Host that both icinga masters connect to?
The error related to ido occurred when I disabled the service on the second master.
When it is enabled, I don’t encounter any errors regarding ido.
I think I found the problem.
Currently, I have Icinga Director enabled on one of my master servers.
The second master is also included in the master zone configuration.
The database is synchronized using Galera.
However, when I start the icinga2 service on the second master server, the dashboard becomes empty.
In the logs( /var/lib/icinga2/api/zones-stage-startup-last-failed.log), I see many errors indicating that hosts or host templates are not defined or do not exist.
I’ll share a few examples here.
Should the Director only be enabled on the first master? Will these errors be resolved by enabling the Director?
[2024-12-10 16:26:59 +0100] information/cli: Icinga application loader (version: r2.14.0-1)
[2024-12-10 16:26:59 +0100] information/cli: Loading configuration file(s).
[2024-12-10 16:26:59 +0100] information/ConfigItem: Committing config item(s).
[2024-12-10 16:26:59 +0100] information/ApiListener: My API identity: smon02
[2024-12-10 16:26:59 +0100] warning/Zone: The Zone object 'satellite' has more than two endpoints. Due to a known issue this type of configuration is strongly discouraged and may cause Icinga to use excessive amounts of CPU time.
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'AW - Aula' of type 'Host'; Attribute 'groups': Object 'Service-Zentrum' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf: 8016:5-8016:34
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(8014): display_name = "AW - Aula"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(8015): address = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(8016): groups = [ "Service-Zentrum" ]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(8017): }
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(8018):
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'aw-mul1-le4' of type 'Host'; Attribute 'groups': Object 'ACI' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf: 7689:5-7689:22
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7687): display_name = "aw-mul1-le4"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7688): address = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7689): groups = [ "ACI" ]
^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7690): }
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7691):
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'aw-c7005-admin2' of type 'Host'; Attribute 'groups': Object 'c7000_synergy' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf: 90:5-90:32
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(88): check_command = "hostalive"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(89): command_endpoint = "satellite01"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(90): groups = [ "c7000_synergy" ]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(91): vars.snmp_community = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(92): }
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'aw-bul1-uz2 at' of type 'Host'; Attribute 'groups': Object 'usv_zentral' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf: 177:5-177:30
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(175): check_command = "hostalive"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(176): command_endpoint = "satellite02"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(177): groups = [ "usv_zentral" ]
^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(178): vars.snmp_community = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(179): }
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'aw-v1l1-ats3' of type 'Host'; Attribute 'groups': Object 'ats' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf: 7944:5-7944:22
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7942): display_name = "aw-v1l1-ats3"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7943): address = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7944): groups = [ "ats" ]
^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7945): vars["_override_servicevars"] += {
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7946): Strom = {
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'aw-del1-li1' of type 'Host'; Attribute 'groups': Object 'ACI' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf: 7626:5-7626:22
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7624): display_name = "aw-del1-li1"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7625): address = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7626): groups = [ "ACI" ]
^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7627): }
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(7628):
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'aw-nora04' of type 'Host'; Attribute 'groups': Object 'oracle_servers' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf: 1264:5-1264:33
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(1262): display_name = "aw-nora04"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(1263): address = "x"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(1264): groups = [ "oracle_servers" ]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(1265): vars.hpilocommunity = "schubert"
/var/lib/icinga2/api/zones-stage//satellite/director/hosts.conf(1266): vars.hpiloservertype = "blade"
[2024-12-10 16:26:59 +0100] critical/config: Error: Validation failed for object 'fs-fa-transfer.mdw.ac.at' of type 'Host'; Attribute 'groups': Object 'netapp' of type 'HostGroup' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf: 260:5-260:25
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(258): check_command = "hostalive"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(259): command_endpoint = "satellite02"
/var/lib/icinga2/api/zones-stage//satellite/director/host_templates.conf(260): groups = [ "netapp" ]
^^^^^^^^^^^^^^^^^^^^^
Hi Dominik,
Don’t I need to have the keys for both masters on each master?
root@smon01:/var/lib/icinga2/certs # ll
-rw-r--r-- 1 icinga icinga 1720 Nov 29 14:32 ca.crt
-rw-r--r-- 1 icinga icinga 1720 Oct 8 10:00 ca.crt.orig
-rw-r--r-- 1 icinga icinga 1765 Nov 29 14:32 smon01.x.crt
-rw-r--r-- 1 icinga icinga 1655 Nov 29 14:32 smon01.x.csr
-rw------- 1 icinga icinga 3243 Nov 29 14:32 smon01.x.key
smon02:/var/lib/icinga2/certs # ll
total 32
-rw-r--r-- 1 icinga icinga 1720 Dec 9 10:47 ca.crt
-rw-r--r-- 1 icinga icinga 1720 Dec 9 10:47 ca.crt.orig
-rw-r--r-- 1 icinga icinga 1765 Dec 9 10:47 smon02.x.crt
-rw-r--r-- 1 icinga icinga 1765 Dec 9 10:47 smon02.x.orig
-rw-r--r-- 1 icinga icinga 1655 Nov 29 17:03 smon02.x.csr
-rw------- 1 icinga icinga 3247 Dec 9 10:47 smon02.x.key
-rw------- 1 icinga icinga 3243 Dec 9 10:47 smon02.x.key.orig
-rw------- 1 icinga icinga 40 Dec 9 10:48 ticket
Thanks
No, you don’t and I also haven’t.
Thanks a lot for your help.
I found something new.
Here are the logs I have:
icinga-test:~ # tail -f /var/log/icingaweb2/icingaweb2.log
2024-12-17T16:02:45+01:00 - ERROR - Failed to load pending migrations: PDOException in /usr/share/icinga-php/ipl/vendor/ipl/sql/src/Connection.php:401 with message: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'icinga.icingaweb_schema' doesn't exist
I added the schema, but it seems to exist:
icinga-test:~ # mysql -u root -p icingaweb2 < /usr/share/icingaweb2/schema/mysql.schema.sql
Enter password:
--------------
CREATE TABLE `icingaweb_group`(
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`name` varchar(64) COLLATE utf8mb4_unicode_ci NOT NULL,
`parent` int(10) unsigned NULL DEFAULT NULL,
`ctime` timestamp NULL DEFAULT NULL,
`mtime` timestamp NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (`id`),
UNIQUE KEY `idx_name` (`name`),
CONSTRAINT `fk_icingaweb_group_parent_id` FOREIGN KEY (`parent`)
REFERENCES `icingaweb_group` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=DYNAMIC
--------------
ERROR 1050 (42S01) at line 3: Table 'icingaweb_group' already exists
I deleted the tables once and added the schema again, but the issue wasn’t resolved.
have you any idea ?