Hi,
I’m trying to run a HA Icinga Setup with two masters. I used the official Icinga installation guide for this. But as soon as I configured and started my second master, the backend in Icingaweb2 stops running. I can’t see any issues or error messages which would help me to find the problem.
The icinga2 daemon -C shows no errors.
Could you pls give me an advise how to solve this problem?
Thank you in advance!
regards,
Aaron
Hi @aaron,
do both masters have the IDO feature enabled and write to the same database? If not, have you disabled HA mode in the IDO features config?
Greetings
Noah
hi @nhilverling
thank you for your reply! Both masters have the IDO feature enabled and are supposed to write to the same database. In the ido-mysql.conf file is HA mode set = true.
What else can it be? The message that the Icinga backend isn’t running anymore appears after 1minute of starting the second master. If I stop the second master instance, the backend in Icingaweb turns back to running!
Greetings
What does grep -Fie ido < /var/log/icinga2/icinga2.log say on the 2nd master?
Thx for your reply!
The output from the 2nd master is:
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:33:40 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:33:42 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 2, rate: 2.63333/s (158/min 690/5min 690/15min);
[2020-08-12 17:33:48 +0200] information/DbConnection: Pausing IDO connection: ido-mysql
[2020-08-12 17:33:48 +0200] information/IdoMysqlConnection: 'ido-mysql' paused.
[2020-08-12 17:33:48 +0200] information/DbConnection: 'ido-mysql' stopped.
[2020-08-12 17:33:48 +0200] information/DbConnection: 'ido-mysql' started.
[2020-08-12 17:33:52 +0200] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2020-08-12 17:33:52 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2020-08-12 17:33:52 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:33:52 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:33:58 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 0, rate: 0.183333/s (11/min 11/5min 11/15min);
[2020-08-12 17:34:02 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:02 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:12 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:12 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:22 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:22 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:32 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:32 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:42 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:42 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[root@vmicingamaster2 mssadmin]#
Just another thought, or does it has something to do with the director which I’m running on the 1st master? Is there a special config for running a HA Setup with a working director?
Thx in advance!
Are you sure about that?
thats what I thought? The ido_feature is enabled also the ha mode set to true. How and where can I have to fix this?
I’m sorry for this stupid question. I’m still new in the Icinga World.
greetings
In /etc/icinga2/features-available/ido-mysql.conf by specifying the correct host.
hi @Al2Klimov, I’ve specified the 1st master with it’s IP Adress as host. I also looked up for the Grants in the mysql database on the 1st master, to make sure I’ve set the right privileges and password. But I get this error-message
[2020-08-13 16:39:13 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-13 16:39:17 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 6, rate: 1.46667/s (88/min 88/5min 88/15min);
[2020-08-13 16:39:23 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'root' on '172.16.23.62:3306' failed: "Lost connection to MySQL server at 'reading initial communication packet', system error: 0"
(0) Reconnecting to MySQL IDO database 'ido-mysql'
Do I miss out something else important? Thank you for your help, I really appreciate it!
greetings,
aaron
Please share the host setting from both masters.
Master2:
object IdoMysqlConnection "ido-mysql" {
user = "root"
password = "*6691484EA6B50DDDE1926A220DA01FA9E575C18A"
host = "172.16.23.62"
database = "icinga"
enable_ha = true
}
Master1:
object IdoMysqlConnection "ido-mysql" {
//user = "root"
//password = "*6acfed4de24ae18b7a824573464bce8b438bb51c"
//host = "localhost"
//database = "icinga"
enable_ha = true
}Are you sure that’s the right address?
hmmm… I’ve changed the Host settings on the 1st master to the following:
object IdoMysqlConnection "ido-mysql" {
//user = "root"
//password = "*6acfed4de24ae18b7a824573464bce8b438bb51c"
host = "172.16.23.62"
//database = "icinga"
enable_ha = true
}
But now I get the same error message out of the log as for the second master and the backend stops running in Icingaweb. Even if I uncommented all entries in the file.
I also tried to stop and start the 2nd master, but it doesn’t change the result.
Error log form 1st master now:
(0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-13 18:10:39 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-13 18:10:49 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on '172.16.23.62:3306' failed: "Lost connection to MySQL server at 'reading initial communication packet', system error: 0"
I’m sorry 
Can you login with the user icinga and password icinga to the mysql database on host 172.16.23.62?
What happens if you do mysql -u icinga -p on the host 172.16.23.62?
What happens if you do mysql -h 172.16.23.62 -u icinga -p on the master server?
Is the root login to the database allowed from external sources, meaning not the db server itself?
Hi @log1c,
On the 1st master (172.16.23.62) I can enter the database with this user: icinga and pw: icinga
On the 2nd master (172.16.23.63) I can’t enter the database with this credentials. Which should be ok?
This on the other hand this doesn’t work on both masters. I cant login from the 2nd master with the credentials user:icinga; pw:icinga, neither from the 1st master.
I get the following error message:
[root@vmicingamaster2 ms]# mysql -h 172.16.23.62 -u icinga -p
Enter password:
ERROR 1130 (HY000): Host 'Icingamaster2' is not allowed to connect to this MariaDB serve
What do you mean by the root login into the database from external source? What did I wrong or how could I fix it?
Do I also have to uncomment the login data on the 1st master in the ido-mysql.conf file, or just on the 2nd master?
Thank you for your help!! 
You have this user in your ido config on master2.
Normally (I think) root can only login from the DB server itself, not from other remote servers.
Also I wouldn’t use the root user for the ido connection.
A login should be working from the second master as well.
So check the user login permissions on the database.
Basic tutorial: https://linuxize.com/post/how-to-create-mysql-user-accounts-and-grant-privileges/
You will need to allow the icinga user to login from the master2 ip and edit your ido config on master2 accordingly.
Hi.
Additionally, please make sure, that a remote login to MySQL (or MariaDB) is allowed.
Caution:
If you change the following, especially the IP, please - previously - make sure, that this has no side effects to anything else using the SQL-Server.
The address has to be changed there, accordingly.
The related files are:
MySQL : /etc/mysql/mysql.conf.d/50-server.cnf
MariaDB : /etc/mysql/mariadb.conf.d/50-server.cnf
After changes, the SQL-Server has to be restarted.
Another note:
The connecting user has to be allowed to connect remotely.
This can be done by adding a suffix other than localhost (or similar) at creation time.
The suffix can be an IP/Subnet or wildcard, e.g.:
GRANT ALL ON <database>.* TO '<the_user>'@'<remote_ip' IDENTIFIED BY '<the_user_password>';
You can list the users and IPs by:
select user, host from mysql.user;
Greetings
It works!! Thank you all for your help. 
At the end was it the Remote Login from the correct user. Thank you!