# Backend is not running with 2nd master

**URL:** <https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234>\
**Category:** Icinga 2\
**Tags:** ha, master, icingaweb2\
**Created:** [August 10, 2020, 3:57pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234 "2020-08-10T15:57:11Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 10, 2020, 3:57pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/1 "2020-08-10T15:57:11Z")

</div>

Hi,

I’m trying to run a HA Icinga Setup with two masters. I used the official Icinga installation guide for this. But as soon as I configured and started my second master, the backend in Icingaweb2 stops running. I can’t see any issues or error messages which would help me to find the problem.  
The icinga2 daemon -C shows no errors.

Could you pls give me an advise how to solve this problem?

Thank you in advance!

regards,  
Aaron

---

<div class="post-metadata">

**Author:** ![nhilverling](https://community.icinga.com/user_avatar/community.icinga.com/nhilverling/32/4524_2.png) [@nhilverling](https://community.icinga.com/u/nhilverling)\
**Post date:** [August 11, 2020, 11:01am UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/2 "2020-08-11T11:01:16Z")

</div>

Hi @aaron,

do both masters have the IDO feature enabled and write to the same database? If not, have you disabled HA mode in the IDO features config?

Greetings  
Noah

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 11, 2020, 2:13pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/3 "2020-08-11T14:13:34Z")

</div>

hi @nhilverling

thank you for your reply! Both masters have the IDO feature enabled and are supposed to write to the same database. In the ido-mysql.conf file is HA mode set = true.

What else can it be? The message that the Icinga backend isn’t running anymore appears after 1minute of starting the second master. If I stop the second master instance, the backend in Icingaweb turns back to running!

Greetings

---

<div class="post-metadata">

**Author:** ![Al2Klimov](https://community.icinga.com/user_avatar/community.icinga.com/al2klimov/32/959_2.png) [@Al2Klimov](https://community.icinga.com/u/Al2Klimov)\
**Post date:** [August 12, 2020, 2:46pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/4 "2020-08-12T14:46:13Z")

</div>

What does `grep -Fie ido < /var/log/icinga2/icinga2.log` say on the 2nd master?

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 12, 2020, 3:41pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/5 "2020-08-12T15:41:34Z")

</div>

Thx for your reply!  
The output from the 2nd master is:

```
 (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:33:40 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:33:42 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 2, rate: 2.63333/s (158/min 690/5min 690/15min);
[2020-08-12 17:33:48 +0200] information/DbConnection: Pausing IDO connection: ido-mysql
[2020-08-12 17:33:48 +0200] information/IdoMysqlConnection: 'ido-mysql' paused.
[2020-08-12 17:33:48 +0200] information/DbConnection: 'ido-mysql' stopped.
[2020-08-12 17:33:48 +0200] information/DbConnection: 'ido-mysql' started.
[2020-08-12 17:33:52 +0200] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2020-08-12 17:33:52 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2020-08-12 17:33:52 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
    (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:33:52 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:33:58 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 0, rate: 0.183333/s (11/min 11/5min 11/15min);
[2020-08-12 17:34:02 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
    (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:02 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:12 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
    (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:12 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:22 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
    (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:22 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:32 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
    (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:32 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-12 17:34:42 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
    (0) Reconnecting to MySQL IDO database 'ido-mysql'
[2020-08-12 17:34:42 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[root@vmicingamaster2 mssadmin]#

```

Just another thought, or does it has something to do with the director which I’m running on the 1st master? Is there a special config for running a HA Setup with a working director?

Thx in advance!

---

<div class="post-metadata">

**Author:** ![Al2Klimov](https://community.icinga.com/user_avatar/community.icinga.com/al2klimov/32/959_2.png) [@Al2Klimov](https://community.icinga.com/u/Al2Klimov)\
**Post date:** [August 12, 2020, 3:54pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/6 "2020-08-12T15:54:04Z")

</div>

> [@aaron](#):
>
> are supposed to write to the same database.

Are you sure about that?

> [@aaron](#):
>
> ```auto
> [2020-08-12 17:33:52 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on 'localhost:3306' failed: "Access denied for user 'icinga'@'localhost' (using password: YES)"
> 
> ```

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 12, 2020, 3:59pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/7 "2020-08-12T15:59:51Z")

</div>

😅 thats what I thought? The ido\_feature is enabled also the ha mode set to true. How and where can I have to fix this?  
I’m sorry for this stupid question. I’m still new in the Icinga World.

greetings

---

<div class="post-metadata">

**Author:** ![Al2Klimov](https://community.icinga.com/user_avatar/community.icinga.com/al2klimov/32/959_2.png) [@Al2Klimov](https://community.icinga.com/u/Al2Klimov)\
**Post date:** [August 12, 2020, 4:11pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/8 "2020-08-12T16:11:56Z")

</div>

> [@aaron](#):
>
> How and where can I have to fix this?

In `/etc/icinga2/features-available/ido-mysql.conf` by specifying the correct host.

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 13, 2020, 2:45pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/9 "2020-08-13T14:45:21Z")

</div>

hi @Al2Klimov, I’ve specified the 1st master with it’s IP Adress as host. I also looked up for the Grants in the mysql database on the 1st master, to make sure I’ve set the right privileges and password. But I get this error-message

```
[2020-08-13 16:39:13 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
[2020-08-13 16:39:17 +0200] information/WorkQueue: #5 (IdoMysqlConnection, ido-mysql) items: 6, rate: 1.46667/s (88/min 88/5min 88/15min);
[2020-08-13 16:39:23 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'root' on '172.16.23.62:3306' failed: "Lost connection to MySQL server at 'reading initial communication packet', system error: 0"
        (0) Reconnecting to MySQL IDO database 'ido-mysql'

```

Do I miss out something else important? Thank you for your help, I really appreciate it!

greetings,  
aaron

---

<div class="post-metadata">

**Author:** ![Al2Klimov](https://community.icinga.com/user_avatar/community.icinga.com/al2klimov/32/959_2.png) [@Al2Klimov](https://community.icinga.com/u/Al2Klimov)\
**Post date:** [August 13, 2020, 2:50pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/10 "2020-08-13T14:50:24Z")

</div>

Please share the host setting from both masters.

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 13, 2020, 3:09pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/11 "2020-08-13T15:09:07Z")

</div>

Master2:

```
object IdoMysqlConnection "ido-mysql" {
 user = "root"
 password = "*6691484EA6B50DDDE1926A220DA01FA9E575C18A"
 host = "172.16.23.62"
 database = "icinga"
 enable_ha = true
}

```

Master1:

```
object IdoMysqlConnection "ido-mysql" {
  //user = "root"
  //password = "*6acfed4de24ae18b7a824573464bce8b438bb51c"
  //host = "localhost"
  //database = "icinga"
  enable_ha = true
}
```

---

<div class="post-metadata">

**Author:** ![Al2Klimov](https://community.icinga.com/user_avatar/community.icinga.com/al2klimov/32/959_2.png) [@Al2Klimov](https://community.icinga.com/u/Al2Klimov)\
**Post date:** [August 13, 2020, 3:20pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/12 "2020-08-13T15:20:25Z")

</div>

> [@aaron](#):
>
> ```auto
> //host = "localhost"
> 
> ```

Are you sure that’s the right address?

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 13, 2020, 4:14pm UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/13 "2020-08-13T16:14:04Z")

</div>

hmmm… I’ve changed the Host settings on the 1st master to the following:

```
 object IdoMysqlConnection "ido-mysql" {
  //user = "root"
  //password = "*6acfed4de24ae18b7a824573464bce8b438bb51c"
  host = "172.16.23.62"
  //database = "icinga"
  enable_ha = true
}

```

But now I get the same error message out of the log as for the second master and the backend stops running in Icingaweb. Even if I uncommented all entries in the file.  
I also tried to stop and start the 2nd master, but it doesn’t change the result.

Error log form 1st master now:

```
 (0) Reconnecting to MySQL IDO database 'ido-mysql'
    [2020-08-13 18:10:39 +0200] critical/IdoMysqlConnection: Exception during database operation: Verify that your database is operational!
    [2020-08-13 18:10:49 +0200] critical/IdoMysqlConnection: Connection to database 'icinga' with user 'icinga' on '172.16.23.62:3306' failed: "Lost connection to MySQL server at 'reading initial communication packet', system error: 0"

```

I’m sorry 🙄

---

<div class="post-metadata">

**Author:** ![log1c](https://community.icinga.com/user_avatar/community.icinga.com/log1c/32/2071_2.png) [@log1c](https://community.icinga.com/u/log1c)\
**Post date:** [August 14, 2020, 6:08am UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/14 "2020-08-14T06:08:46Z")

</div>

Can you login with the user `icinga` and password `icinga` to the mysql database on host 172.16.23.62?

What happens if you do `mysql -u icinga -p` on the host 172.16.23.62?  
What happens if you do `mysql -h 172.16.23.62 -u icinga -p` on the master server?  
Is the root login to the database allowed from external sources, meaning not the db server itself?

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 14, 2020, 8:20am UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/15 "2020-08-14T08:20:18Z")

</div>

Hi @log1c,

> [@log1c](#):
>
> What happens if you do `mysql -u icinga -p` on the host 172.16.23.62?

On the 1st master (172.16.23.62) I can enter the database with this user: icinga and pw: icinga  
On the 2nd master (172.16.23.63) I can’t enter the database with this credentials. Which should be ok?

> [@log1c](#):
>
> What happens if you do `mysql -h 172.16.23.62 -u icinga -p` on the master server?

This on the other hand this doesn’t work on both masters. I cant login from the 2nd master with the credentials user:icinga; pw:icinga, neither from the 1st master.  
I get the following error message:

```
[root@vmicingamaster2 ms]# mysql -h 172.16.23.62 -u icinga -p
Enter password:
ERROR 1130 (HY000): Host 'Icingamaster2' is not allowed to connect to this MariaDB serve 

```

What do you mean by the root login into the database from external source? What did I wrong or how could I fix it?  
Do I also have to uncomment the login data on the 1st master in the ido-mysql.conf file, or just on the 2nd master?

Thank you for your help!! 🙂

---

<div class="post-metadata">

**Author:** ![log1c](https://community.icinga.com/user_avatar/community.icinga.com/log1c/32/2071_2.png) [@log1c](https://community.icinga.com/u/log1c)\
**Post date:** [August 14, 2020, 8:40am UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/16 "2020-08-14T08:40:39Z")

</div>

> [@aaron](#):
>
> ```auto
> user = "root"
> password = "*6691484EA6B50DDDE1926A220DA01FA9E575C18A"
> 
> ```

You have this user in your ido config on master2.  
Normally (I think) root can only login from the DB server itself, not from other remote servers.  
Also I wouldn’t use the root user for the ido connection.

> [@aaron](#):
>
> On the 2nd master (172.16.23.63) I can’t enter the database with this credentials. Which should be ok?

A login should be working from the second master as well.  
So check the user login permissions on the database.  
Basic tutorial: [https://linuxize.com/post/how-to-create-mysql-user-accounts-and-grant-privileges/](https://linuxize.com/post/how-to-create-mysql-user-accounts-and-grant-privileges/)  
You will need to allow the icinga user to login from the master2 ip and edit your ido config on master2 accordingly.

---

<div class="post-metadata">

**Author:** ![homerjay](https://community.icinga.com/user_avatar/community.icinga.com/homerjay/32/377_2.png) [@homerjay](https://community.icinga.com/u/homerjay)\
**Post date:** [August 14, 2020, 9:03am UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/17 "2020-08-14T09:03:40Z")

</div>

Hi.

Additionally, please make sure, that a remote login to MySQL (or MariaDB) is allowed.

**Caution:  
If you change the following, especially the IP, please - previously - make sure, that this has no side effects to anything else using the SQL-Server.  
The address has to be changed there, accordingly.**

The related files are:

```auto
MySQL	: /etc/mysql/mysql.conf.d/50-server.cnf
MariaDB	: /etc/mysql/mariadb.conf.d/50-server.cnf

```

- Look for the line starting with “ **bind-address** ”. It has to contain the IP which the SQL-Server is reachable at via network.
- also, please make sure that the line containing “ **skip-networking** ” is **commented out**.

After changes, the SQL-Server has to be restarted.

**Another note:**  
The connecting user has to be allowed to connect remotely.  
This can be done by adding a suffix other than localhost (or similar) at creation time.  
The suffix can be an IP/Subnet or wildcard, e.g.:

```auto
GRANT ALL ON <database>.* TO '<the_user>'@'<remote_ip' IDENTIFIED BY '<the_user_password>';

```

You can list the users and IPs by:

```auto
select user, host from mysql.user;

```

* * *

Greetings

---

<div class="post-metadata">

**Author:** ![aaron](https://community.icinga.com/letter_avatar_proxy/v4/letter/a/e19adc/32.png) [@aaron](https://community.icinga.com/u/aaron)\
**Post date:** [August 14, 2020, 9:37am UTC](https://community.icinga.com/t/backend-is-not-running-with-2nd-master/5234/18 "2020-08-14T09:37:22Z")

</div>

It works!! Thank you all for your help. 🙂

At the end was it the Remote Login from the correct user. Thank you!