Backend icinga not running

Hi Team,

Its noted that i am getting a new error all of a sudden in the Web UI “Backend icinga not running”.
I have done configuration check using icinga2 daemon -C and could not find any relevant issue related to this.

Can someone help me with this.

Along with this, i am attaching the icinga2 and message logs for references.

i have ran “select status_update_time from icinga_programstatus;” to check if DB is getting updated. Its also showing the latest time. Attaching the snap for your references.

logs.zip (79.1 KB) mysql zones.conf (1.1 KB) zones.conf (1.1 KB)

Hi,

what outputs the following command:

systemctl status icinga2.service

From the zones.conf I can see that this is a multi-master setup, right? Do you have two IDO databases or a single one?

Best regards
Michael

Attaching the service status.

yes its multi master setup. I have only one IDO DB. Both masters connects to same DB.

Regards
Rajiv

Hi,

in the log files you provided I found the following entry:

SELinux is preventing /opt/rh/rh-php71/root/usr/sbin/php-fpm from name_connect access on the tcp_socket port 3306.

3306 is the default MySQL/MariaDB port. I think SELinux is blocking the php-fpm daemon to access the IDO database. I am not familiar with SELinux, but you can try to disable it for testing purposes and see if the error message “backend is not running” will disappear. If SELinux is the problem you can investigate how to create a exception for this.

Best regards
Michael

Thanks @mcktr for checking this.

i have checked the selinux access part and executed the following command :
setsebool -P httpd_can_network_connect_db 1

But still the issue persist.

Regards
Rajiv

Hi,

for checking the logs about SELinux issues, you better should look into /var/log/audit/audit.log. Check here if you find some entries with “denied”.
Sometimes you could find also entries in the journal. Here you could grep for “setrouble”.
Our experience is very often that if you fix one issue, the next one is comming. So we had to fix one permission issue at a time until it worked. We we knew all needed permissions, we create or own SELinux rule set if there is nothing delivered with the installation routine.

Since I have a very limited knowledge about SELinux I searched Google for how to allow Apache or PHP-FPM to connect to MySQL/MariaDB:

setsebool -P httpd_can_network_connect 1
setsebool -P httpd_can_network_connect_db 1

The article states that both commands must be run, afterwards you have to restart Apache and/or PHP-FPM.

Best regards
Michael

Hi @mcktr
I executed the both the steps,
setsebool -P httpd_can_network_connect 1
setsebool -P httpd_can_network_connect_db 1

and restarted httpd and icinga2 services.
But still the same.

Hi @stevie-sy,

We checked the audit logs, but couldnot find any denied or errors in there.

Can you please test if disabling SELinux will solve the issue. Run setenforce 0 to disable the enforcing mode, now restart Apache and/or PHP-FPM and look if the issue is resolved. If so we know that this is caused by SELinux and can look deeper into it how to resolve this with enabled SELinux. If the issue is not resoled we know there must be another issue somewhere.

Best regards
Michael

Hi @mcktr,

Already Selinux was set to permissive in the server. Already setenforce was set to 0.’’

[root@TRVUSTMMVLMON01 master]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

Regards
Rajiv

This would indicate an issue that is not related to SELinux.

You have two masters writing to one MySQL/MariaDB server? Can you please show us the ido-mysql.conf from both servers.

Best regards
Michael

Hi @mcktr,

This environment has been up for almost 2 months. The issue has started all of a sudden today.

I am attaching the ido conf from both masters.

Regards
Rajiv

ido-mysql.conf (214 Bytes) ido-mysql.conf (214 Bytes)

Do you see status changes in Icinga Web? Could you also provide the log file from the second master?

Best regards
Michael

Hi @mcktr,

Icinga web is right now installed only in primary server.

Attaching logs for your references.

logs-dr.zip (138.3 KB)

Please check the time/NTP settings on all hosts. Icinga Web is checking timestamps and if you hosts (including your client) think they are in different realities with different times, it can sum up to Icinga Web 2 thinking it doesn’t updated.

Also, check if you actually get new results. So is it about Icinga Web really not being updated or is it just this warning?

Hi Thomas,

As as i know, all the hosts are in same time zone and also they are provisioned from the same network.

I am concerned about this warning - Backend icinga not running which keeps notifying in the web.

Regards
Rajiv

Team,

Any help would be highly appreciable.

regards
rajiv

Please don’t just check the timezone but also if all systems show the same time. If e.g. ntp or chrony are not running, the could have different time settings. Please log into all systems and use date and ntpq -p to show if they show the same time and if NTP is synchronized.

1 Like