/icingaweb2/setup returns 500 error after migrating to new VM

Icinga Web
1

I had icinga 2.9.1/icingaweb2 v2.6.1/mysql 5.7 running on a CentOS 7.5.1804 VM. I had to migrate to a new CentOS 7.9.2009 VM and the default install is now icinga 2.14.1/icingaweb2 2.12.1/mariadb 10.3.14.

I copied my configuration files and my database from the old VM to the new VM.
I have icinga running and it is correctly monitoring hosts and sending out alerts
I have run the mysql_upgrade for the database and the scripts to upgrade the schema for icinga.

I installed

icingaweb2
icingaweb2-common
icingadb
icingadb-web
icingacli

and ran

icingacli setup token create

icinga version

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.1-1)

Copyright (c) 2012-2024 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.42.2.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 11.2.1
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Web Browser:

Chrome Version 120.0.6099.217 (Official Build) (64-bit)

PHP Version:

php --version
PHP 5.4.16 (cli) (built: Apr  1 2020 04:07:17) 
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2013 Zend Technologies

The rh-php73-php-* packages are installed

/opt/rh/rh-php73/root/bin/php --version
PHP 7.3.33 (cli) (built: Jul 25 2022 11:32:44) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.33, Copyright (c) 1998-2018 Zend Technologies

I have the httpd LogLevel set to debug but I am not seeing anything helpful in the logs except for messages like these

[12/Jan/2024:11:23:04 -0800] "GET /icingaweb2/application-state HTTP/1.1" 500 -
[12/Jan/2024:11:25:04 -0800] "GET /icingaweb2/application-state HTTP/1.1" 500 -
[12/Jan/2024:11:27:04 -0800] "GET /icingaweb2/application-state HTTP/1.1" 500 -

I have gone to the trouble of removing the icingaweb2 packages and re-installing them in case I made some type of configuration error.

The icingaweb2 database exists from the other VM. I am not sure if I should remove that database or not.

icinga2 has been running for a few days. I could wipe everything and start over. The loss of a few days of data won’t be a tragedy but I am not sure that that will help.

Any pointers or suggestions would be appreciated.

2

I missed 2 configuration steps. I forgot to update

/etc/httpd/conf.d/icingaweb2.conf
/etc/php-fpm.d/www.conf

Both files have 127.0.01:9000 for the address to bind to but I am using a specific IP address instead of 127.0.0.1. I updated these files and restarted httpd and php-fpm.

I now get the error

Backend unavailable
It seems that the PHP FPM service is not running. Make sure to start PHP FPM service in order to access Icinga Web 2. If you upgraded Icinga Web 2 recently, make sure to read the docs regarding PHP FPM, also locally available under /usr/share/icingaweb2/doc/02-Installation.md.

I see that php-fpm is running.

netstat -anlp | grep 9000
tcp        0      0 10.XX.XXX.XXX:9000      0.0.0.0:*               LISTEN      12562/php-fpm: mast 
systemctl status -l php-fpm
● php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2024-01-12 13:08:12 PST; 2min 6s ago
 Main PID: 12562 (php-fpm)
   Status: "Processes active: 0, idle: 5, Requests: 0, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/php-fpm.service
           ├─12562 php-fpm: master process (/etc/php-fpm.conf
           ├─12565 php-fpm: pool www              
           ├─12566 php-fpm: pool www              
           ├─12567 php-fpm: pool www              
           ├─12568 php-fpm: pool www              
           └─12569 php-fpm: pool www              

Jan 12 13:08:12 app-lvn-bvg-02 systemd[1]: Starting The PHP FastCGI Process Manager...
Jan 12 13:08:12 app-lvn-bvg-02 systemd[1]: Started The PHP FastCGI Process Manager.

Still investigating why it thinks fpm isn’t running

3

I found this in /var/log/php-fpm/www-error.log. I am not sure why there is an error because this is the latest version of the php-icinga package

[12-Jan-2024 21:21:11 UTC] PHP Parse error:  syntax error, unexpected ':', expecting ';' or '{' in /usr/share/php/Icinga/Application/ApplicationBootstrap.php on line 741
[12-Jan-2024 21:22:04 UTC] PHP Parse error:  syntax error, unexpected ':', expecting ';' or '{' in /usr/share/php/Icinga/Application/ApplicationBootstrap.php on line 741
[12-Jan-2024 21:24:04 UTC] PHP Parse error:  syntax error, unexpected ':', expecting ';' or '{' in /usr/share/php/Icinga/Application/ApplicationBootstrap.php on line 741

rpm -q --whatprovides /usr/share/php/Icinga/Application/ApplicationBootstrap.php 
php-icinga-2.12.1-1.el7.noarch

yum list installed |grep php-icinga
php-icinga.noarch                   2.12.1-1.el7             @icinga-stable-release

 /usr/share/php/Icinga/Application/ApplicationBootstrap.php:
730     public function hasLocales()
731     {
732         $localedir = $this->getLocaleDir();
733         return $localedir !== false && file_exists($localedir) && is_dir($localedir);
734     }
735 
736     /**
737      * Register all hooks provided by the main application
738      *
739      * @return $this
740      */
741     protected function registerApplicationHooks(): self
742     {
743         Hook::register('DbMigration', DbMigration::class, DbMigration::class);
744 
745         return $this;
746     }
747 }
4

I saw that php-fpm was from

php-fpm.x86_64                      5.4.16-48.el7            @base

I installed

rh-php73-php-fpm.x86_64             7.3.33-1.el7             @centos-sclo-rh

and disabled php-fpm and enabled rh-php73-php-fpm service but I still get that Backend unavailable.

There are no errors in

/var/opt/rh/rh-php73/log/php-fpm/error.log

[12-Jan-2024 13:59:46] NOTICE: fpm is running, pid 22681
[12-Jan-2024 13:59:46] NOTICE: ready to handle connections
[12-Jan-2024 13:59:46] NOTICE: systemd monitor interval set to 10000ms

systemctl status -l rh-php73-php-fpm
● rh-php73-php-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/usr/lib/systemd/system/rh-php73-php-fpm.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2024-01-12 13:59:46 PST; 12min ago
 Main PID: 22681 (php-fpm)
   Status: "Processes active: 0, idle: 5, Requests: 0, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/rh-php73-php-fpm.service
           ├─22681 php-fpm: master process (/etc/opt/rh/rh-php73/php-fpm.conf
           ├─22700 php-fpm: pool www                                   
           ├─22702 php-fpm: pool www                                   
           ├─22703 php-fpm: pool www                                   
           ├─22704 php-fpm: pool www                                   
           └─22705 php-fpm: pool www
5

I finally got it working after I changed the following files

/etc/opt/rh/rh-php73/php-fpm.d/www.conf
/etc/httpd/conf.d/icingaweb2.conf

back to using 127.0.0.1:9000 after tracking down that

/usr/share/icingaweb2/modules/setup/library/Setup/Webserver/Apache.php

wanted to use 127.0.0.1:9000.

My old VM also used 127.0.0.1:9000. I never had to change it to <10.XX.XXX.XXX>:9000