Semi-broken icingaweb2 with incubator and about page migration errors

Icinga Web
1

I am on Arch Linux with the following packages:

  • icinga-php-library 0.13.1-1
  • icinga-php-thirdparty 0.12.0-1 # Updated to 0.12.1 since, did not change anything
  • icinga2 2.14.0-1
  • icingaweb2 2.12.1-1
  • icingaweb2-module-director 1.11.0-2
  • icingaweb2-module-incubator 0.20.0-2
  • PHP 8.1.26 for web and 8.2.13 for CLI
  • postgresql 16.1-3

Currently, if I load my main icingaweb page, even the login page complains about incubator unable to connect to icinga.

“Incubator module is still waiting for icinga”

Second giveaway something is wrong is About page being broken completely:

Please check if a db instance exists at all

#0 /usr/share/webapps/icingaweb2/library/Icinga/Application/ProvidedHook/DbMigration.php(21): Icinga\Application\ProvidedHook\DbMigration->getWebDb()
#1 /usr/share/webapps/icingaweb2/library/Icinga/Application/Hook/DbMigrationHook.php(293): Icinga\Application\ProvidedHook\DbMigration->getDb()
#2 /usr/share/webapps/icingaweb2/library/Icinga/Application/Hook/DbMigrationHook.php(172): Icinga\Application\Hook\DbMigrationHook->load()
#3 /usr/share/webapps/icingaweb2/library/Icinga/Application/MigrationManager.php(306): Icinga\Application\Hook\DbMigrationHook->getMigrations()
#4 /usr/share/webapps/icingaweb2/library/Icinga/Application/MigrationManager.php(56): Icinga\Application\MigrationManager->load()
#5 /usr/share/webapps/icingaweb2/library/Icinga/Application/MigrationManager.php(106): Icinga\Application\MigrationManager->getPendingMigrations()
#6 /usr/share/webapps/icingaweb2/library/Icinga/Application/MigrationManager.php(69): Icinga\Application\MigrationManager->count()
#7 /usr/share/webapps/icingaweb2/application/views/scripts/about/index.phtml(99): Icinga\Application\MigrationManager->hasPendingMigrations()
#8 /usr/share/webapps/icingaweb2/library/Icinga/Web/View.php(235): include(String)
#9 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/View/Abstract.php(886): Icinga\Web\View->_run()
#10 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Action/Helper/ViewRenderer.php(910): Zend_View_Abstract->render()
#11 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Action/Helper/ViewRenderer.php(931): Zend_Controller_Action_Helper_ViewRenderer->renderScript()
#12 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Action/Helper/ViewRenderer.php(970): Zend_Controller_Action_Helper_ViewRenderer->render()
#13 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Action/HelperBroker.php(277): Zend_Controller_Action_Helper_ViewRenderer->postDispatch()
#14 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Action.php(527): Zend_Controller_Action_HelperBroker->notifyPostDispatch()
#15 /usr/share/webapps/icingaweb2/library/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch()
#16 /usr/share/icinga-php/vendor/vendor/shardj/zf1-future/library/Zend/Controller/Front.php(954): Icinga\Web\Controller\Dispatcher->dispatch()
#17 /usr/share/webapps/icingaweb2/library/Icinga/Application/Web.php(294): Zend_Controller_Front->dispatch()
#18 /usr/share/webapps/icingaweb2/library/Icinga/Application/webrouter.php(105): Icinga\Application\Web->dispatch()
#19 /usr/share/webapps/icingaweb2/public/index.php(4): require_once(String)
#20 {main}

Dashboard also does not load correctly, dashlet titles load but their contents are empty, clicking on the titles shows the services correctly however.

Director shows no pending migrations:

[0] # icingacli director migration run --verbose
[0] #

Clicking through director/icinga2/icingaweb2(+icingaweb2 user DB backend) resources I have configured all Validates.

Validation Log

Connection to icingaweb2 as icingaweb2 on localhost:5432 successful
PostgreSQL 16.1 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 13.2.1 20230801, 64-bit
1 active users

Databases are all there on PSQL side:

postgres=# \l
                                                         List of databases
    Name    |   Owner    | Encoding | Locale Provider |   Collate   |    Ctype    | ICU Locale | ICU Rules |   Access privileges   
------------+------------+----------+-----------------+-------------+-------------+------------+-----------+-----------------------
 director   | postgres   | UTF8     | libc            | en_GB.UTF-8 | en_GB.UTF-8 |            |           | =Tc/postgres         +
            |            |          |                 |             |             |            |           | postgres=CTc/postgres+
            |            |          |                 |             |             |            |           | director=CTc/postgres
 icinga     | icinga     | UTF8     | libc            | en_GB.UTF-8 | en_GB.UTF-8 |            |           | 
 icingaweb2 | icingaweb2 | UTF8     | libc            | en_GB.UTF-8 | en_GB.UTF-8 |            |           | 
 postgres   | postgres   | UTF8     | libc            | en_GB.UTF-8 | en_GB.UTF-8 |            |           | 
 template0  | postgres   | UTF8     | libc            | en_GB.UTF-8 | en_GB.UTF-8 |            |           | =c/postgres          +
            |            |          |                 |             |             |            |           | postgres=CTc/postgres
 template1  | postgres   | UTF8     | libc            | en_GB.UTF-8 | en_GB.UTF-8 |            |           | postgres=CTc/postgres+
            |            |          |                 |             |             |            |           | =c/postgres
(6 rows)

resources.ini

[icingaweb2]
type                = "db"
db                  = "pgsql"
host                = "localhost"
port                = "5432"
dbname              = "icingaweb2"
username            = "icingaweb2"
password            = "<password>"

[icinga2]
type                = "db"
db                  = "pgsql"
host                = "localhost"
port                = "5432"
dbname              = "icinga"
username            = "icinga"
password            = "<password>"

[director]
type                = "db"
db                  = "pgsql"
host                = "localhost"
port                = "5432"
dbname              = "director"
username            = "director"
password            = "<password>"
charset             = "utf8"

This setup has worked previously, but at some indeterminate point broke(I do not open the page often and the Notifications worked fine, so it went unnoticed).

2

Hello @C0rn3j,

I don’t have a solution, but this looks like a PHP related issue.

Has something changed with the PHP version on this server?
Are there any other applications running on this server that might have triggered a PHP upgrade?

In the past, when trying to upgrade PHP version on a test Icinga-server, a very similar problem has occurred to me.

Best regards,
IT guy

3

PHP version on the server might’ve jumped but enabled extensions should have stayed the same as Ansible takes care of that, and 8.2 seems supported by all Icinga parts, so hopefully not the problem there.

curl, gettext, pdo_pgsql, pgsql, intl and sockets are all enabled. Tried gd too just in case as I had that in my notes and that wasn’t it either.

EDIT: Now that I recall what I did few days ago, I might’ve broken it further by using director migration in the web and it half-failed due to gettext missing.

4

Hii, for this part of your issue see Icingadbweb2 v1.1 + Icinga2 r2.14.0-1. : Error in "About" Section.

I’m not sure about this though. Incubator is just a collection of Icinga Director dependencies and does not need any database connections.

1 Like
5

Thanks! Setting the configuration database fixed two of my issues, the one with the About page, and I can now actually properly login permanently (which I forgot to even complain about).

/etc/icingaweb2/config.ini was not present on my system at all, now it has

[global]
show_stacktraces = "1"
show_application_state_messages = "1"
module_path = "/usr/share/webapps/icingaweb2/modules"
config_resource = "icingaweb2"

[security]
use_strict_csp = "0"

[logging]
log = "syslog"
level = "ERROR"
application = "icingaweb2"
facility = "user"

Dashboard is still semi-broken, here’s a screenshot for reference

image
image912×332 18.7 KB

And here’s the (should be fully default) dashboard settings

image
image1408×783 75.6 KB

About incubator, it’s in the JS console, you can see it (at the moment of posting) here https://monitoring.rys.pw/

Another non-Arch Icinga I have loads correctly like so

…
Incubator module is still waiting for icinga
18:23:51.372 Icinga is ready, running on jQuery  3.6.0
…
6

These contents are loaded with JS and your Icinga Web doesn’t seem to load some of the Icinga Web 2 JS files, so the incubator waits for Icinga to become ready and complains constantly.

Bildschirmfoto 2023-12-22 um 10.22.46
Bildschirmfoto 2023-12-22 um 10.22.462844×554 46.3 KB

As you can see here, the bootstrap.js file is empty. You may have emptied it by accident. Please check it on your server.

7

Great catch!

The remaining issues were caused by wrong nginx config.

https://gitlab.com/C0rn3j/configs/-/commit/7c78d93180a103e37c37cb77682bef8677ff6820

Thanks a lot for your help!