Director Background keep-alive is outdated

gfragi · June 25, 2020, 9:02pm

Hi all,

I am using icinga2 last 6 months. Today I saw the following error on web Daemon keep-alive is outdated

Searching on the server I see the following:

Jun 25 23:45:18 XXXXX icingadirector[9279]: Started by systemd, notifying watchdog every 5s via /run/systemd/notify
Jun 25 23:45:18 XXXXX icingadirector[9279]: PHP Fatal error:  Uncaught Error: Call to undefined function posix_getpid() in /usr/share/icingaweb2/modules/incubator/vendor/gipfl
Jun 25 23:45:18 XXXXX icingadirector[9279]: Stack trace:
Jun 25 23:45:18 XXXXX icingadirector[9279]: #0 /usr/share/icingaweb2/modules/director/library/Director/Daemon/BackgroundDaemon.php(74): gipfl\SystemD\NotifySystemD->setReady()
Jun 25 23:45:18 XXXXX icingadirector[9279]: #1 /usr/share/icingaweb2/modules/director/library/Director/Daemon/BackgroundDaemon.php(52): Icinga\Module\Director\Daemon\Backgroun
Jun 25 23:45:18 XXXXX icingadirector[9279]: #2 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/Tick/FutureTickQueue.php(46): Icinga\Module\Director\Daemo
Jun 25 23:45:18 XXXXX icingadirector[9279]: #3 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(181): React\EventLoop\Tick\FutureTick
Jun 25 23:45:18 XXXXX icingadirector[9279]: #4 /usr/share/icingaweb2/modules/director/library/Director/Daemon/BackgroundDaemon.php(55): React\EventLoop\StreamSelectLoop->run()
Jun 25 23:45:18 XXXXX icingadirector[9279]: #5 /usr/share/icingaweb2/modules/director/application/clicommands/DaemonCommand.php(24): Icinga\Module\Direc in /usr/share/icingawe
Jun 25 23:45:18 XXXXX icingadirector[9279]: Fatal error: Uncaught Error: Call to undefined function posix_getpid() in /usr/share/icingaweb2/modules/incubator/vendor/gipfl/syst
Jun 25 23:45:18 XXXXX icingadirector[9279]: Stack trace:
Jun 25 23:45:18 XXXXX icingadirector[9279]: #0 /usr/share/icingaweb2/modules/director/library/Director/Daemon/BackgroundDaemon.php(74): gipfl\SystemD\NotifySystemD->setReady()
Jun 25 23:45:18 XXXXX icingadirector[9279]: #1 /usr/share/icingaweb2/modules/director/library/Director/Daemon/BackgroundDaemon.php(52): Icinga\Module\Director\Daemon\Backgroun
Jun 25 23:45:18 XXXXX icingadirector[9279]: #2 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/Tick/FutureTickQueue.php(46): Icinga\Module\Director\Daemo
Jun 25 23:45:18 XXXXX icingadirector[9279]: #3 /usr/share/icingaweb2/modules/reactbundle/vendor/react/event-loop/src/StreamSelectLoop.php(181): React\EventLoop\Tick\FutureTick
Jun 25 23:45:18 XXXXX icingadirector[9279]: #4 /usr/share/icingaweb2/modules/director/library/Director/Daemon/BackgroundDaemon.php(55): React\EventLoop\StreamSelectLoop->run()
Jun 25 23:45:18 XXXXX icingadirector[9279]: #5 /usr/share/icingaweb2/modules/director/application/clicommands/DaemonCommand.php(24): Icinga\Module\Direc in /usr/share/icingawe
Jun 25 23:45:18 XXXXX systemd[1]: icinga-director.service: main process exited, code=exited, status=255/n/a
Jun 25 23:45:18 XXXXX systemd[1]: Failed to start Icinga Director - Monitoring Configuration.

My version is the following
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.4-1)
Icinga Web 2 Version
2.8.0
Git commit
642ec11228c3be8d2abbdff6ef31da77e34f6c70
PHP Version
7.1.30
Git commit date
2020-06-08

I have tried to restart the service but nothing changed. I didn’t make any changes to the system itself since 2 days ago, as far as i understand that may affect this service.

Has anyone any idea?

log1c · June 26, 2020, 6:09am

https://github.com/Icinga/icingaweb2-module-vspheredb/issues/54
Looks like this machtes your problem (even though this is from the vSphereDB module)

Check if the solution posted there works for you.

gfragi · June 29, 2020, 11:21am

Thank you. I will test it and I will come back with the results.

theFeu · July 6, 2020, 7:07am

Hello @gfragi

Did you get any further with your issue?
We would love to hear back from you

Have a nice day,
Feu

gfragi · July 6, 2020, 11:23pm

Hi again,

Many tasks lately… So I have followed the solution described on the link above but nothing changed in my environment i get the very same error.

What I was missing was the “rh-php71-php-soap” package which I have installed.

gfragi · August 4, 2020, 1:06pm

I still cannot make it to work even if I have upgraded to php7.3

Even it seems that everything is working as expected I have also the following error which is may be related to the icinga director error:

Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/doc" is not a symlinAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/monitoring" is not aAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/setup" is not a symlAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/audit" is not a symlAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/ipl" is not a symlinAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/incubator" is not a 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reactbundle" is not 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/pdfexport" is not a 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/idoreports" is not aAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reporting" is not a 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/graphite" is not a sAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/director" is not a sAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/trapdirector" is notAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/x509" is not a symliAug 04 16:02:28 frmon01 icingadirector[31383]: ERROR: There is no such module or command: 'director'
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/doc" is not a symlinAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/monitoring" is not aAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/setup" is not a symlAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/audit" is not a symlAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/ipl" is not a symlinAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/incubator" is not a 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reactbundle" is not 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/pdfexport" is not a 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/idoreports" is not aAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reporting" is not a 
Aug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/graphite" is not a sAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/director" is not a sAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/trapdirector" is notAug 04 16:02:28 frmon01 icingadirector[31383]: Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/x509" is not a symli

Any idea please I am getting frustrated with this and I dont have clear mind to fix it.

theFeu · August 10, 2020, 12:24pm

Generally it’s important to note that Icinga Web 2.8 with a RHEL/CentOS has dependencies on rh-php73-* and not rh-php71-*
This means that the addon modules that have additional dependencies (like php-posix) might not work anymore.
You should find this in the release notes of Icinga 2.8 as well.

rh-php71-php-soap would probably not help much, since SOAP is required for vSphereDB, but not for the director. And it would have to be PHP 7.3 and not 7.1

I think for your last issue here, it’s not a director problem but something with you Web module configuration.
Could you run a ls -l /etc/icingaweb2/enabledModules and a ls -ld /etc/icingaweb2/enabledModules, please?

gfragi · August 11, 2020, 1:42pm

About director from your answer and from what I have (not) done I conclude that I’ve missed that part

php.ini or php-fpm settings you have tuned in the past need to be copied over to the new path: From /etc/opt/rh/rh-php71/ to /etc/opt/rh/rh-php73/ .

I will try to figure out which php.ini I need to copy to the new path

About second issue here are the outputs.

ls -l /etc/icingaweb2/enabledModules - output

lrwxrwxrwx. 1 apache icingaweb2 35 Jan 10  2020 audit -> /usr/share/icingaweb2/modules/audit
lrwxrwxrwx. 1 apache icingaweb2 38 May 13 16:15 director -> /usr/share/icingaweb2/modules/director
lrwxrwxrwx. 1 apache icingaweb2 33 Jan 10  2020 doc -> /usr/share/icingaweb2/modules/doc
lrwxrwxrwx. 1 apache icingaweb2 38 May 13 12:27 graphite -> /usr/share/icingaweb2/modules/graphite
lrwxrwxrwx. 1 root   icingaweb2 40 Jan 15  2020 idoreports -> /usr/share/icingaweb2/modules/idoreports
lrwxrwxrwx. 1 root   icingaweb2 39 Jan 13  2020 incubator -> /usr/share/icingaweb2/modules/incubator
lrwxrwxrwx. 1 root   icingaweb2 33 Jan 13  2020 ipl -> /usr/share/icingaweb2/modules/ipl
lrwxrwxrwx. 1 apache icingaweb2 40 Jan 10  2020 monitoring -> /usr/share/icingaweb2/modules/monitoring
lrwxrwxrwx. 1 root   icingaweb2 39 Jan 14  2020 pdfexport -> /usr/share/icingaweb2/modules/pdfexport
lrwxrwxrwx. 1 root   icingaweb2 41 Jan 13  2020 reactbundle -> /usr/share/icingaweb2/modules/reactbundle
lrwxrwxrwx. 1 apache icingaweb2 39 Jan 15  2020 reporting -> /usr/share/icingaweb2/modules/reporting
lrwxrwxrwx. 1 apache icingaweb2 35 Jan 10  2020 setup -> /usr/share/icingaweb2/modules/setup
lrwxrwxrwx. 1 apache icingaweb2 42 Jun 23 17:19 trapdirector -> /usr/share/icingaweb2/modules/trapdirector
lrwxrwxrwx. 1 apache icingaweb2 34 Jul 29 09:24 x509 -> /usr/share/icingaweb2/modules/x509

ls -ld /etc/icingaweb2/enabledModules - output

drwxrwS---. 2 apache icingaweb2 224 Jul 29 09:24 /etc/icingaweb2/enabledModules

theFeu · August 13, 2020, 9:42am

Hm, let’s focus first issue for now:
With rpm -qa | grep php you should get the info which package has been installed in which version.
And then you can start sorting that out from there

gfragi · August 13, 2020, 1:35pm

Here is my output I dont know from where to start

php-json-7.3.20-1.el7.remi.x86_64
php-mbstring-7.3.20-1.el7.remi.x86_64
rh-php73-php-soap-7.3.11-1.el7.x86_64
rh-php73-php-pgsql-7.3.11-1.el7.x86_64
rh-php73-php-ldap-7.3.11-1.el7.x86_64
rh-php73-php-fpm-7.3.11-1.el7.x86_64
sclo-php73-php-pecl-imagick-3.4.4-3.el7.x86_64
php-devel-7.3.20-1.el7.remi.x86_64
php-process-7.3.20-1.el7.remi.x86_64
php71-php-json-7.1.33-8.el7.remi.x86_64
rh-php73-runtime-1-1.el7.x86_64
php-fedora-autoloader-1.0.1-2.el7.noarch
php-pear-1.10.12-1.el7.remi.noarch
php-cli-7.3.20-1.el7.remi.x86_64
php-gd-7.3.20-1.el7.remi.x86_64
php-ldap-7.3.20-1.el7.remi.x86_64
php71-php-common-7.1.33-8.el7.remi.x86_64
php-Icinga-2.8.1-1.el7.icinga.noarch
rh-php73-php-json-7.3.11-1.el7.x86_64
rh-php73-php-cli-7.3.11-1.el7.x86_64
php73-php-json-7.3.20-1.el7.remi.x86_64
php-pdo-7.3.20-1.el7.remi.x86_64
rh-php73-php-gmp-7.3.11-1.el7.x86_64
php71-php-process-7.1.33-8.el7.remi.x86_64
rh-php73-php-common-7.3.11-1.el7.x86_64
rh-php73-php-mysqlnd-7.3.11-1.el7.x86_64
rh-php73-php-mbstring-7.3.11-1.el7.x86_64
rh-php73-php-xml-7.3.11-1.el7.x86_64
rh-php73-php-intl-7.3.11-1.el7.x86_64
php73-php-common-7.3.20-1.el7.remi.x86_64
php-gmp-7.3.20-1.el7.remi.x86_64
php-xml-7.3.20-1.el7.remi.x86_64
icingaweb2-vendor-lessphp-2.8.1-1.el7.icinga.noarch
oniguruma5php-6.9.5+rev1-2.el7.remi.x86_64
php-mysqlnd-7.3.20-1.el7.remi.x86_64
php-intl-7.3.20-1.el7.remi.x86_64
php71-runtime-2.0-1.el7.remi.x86_64
rh-php73-php-zip-7.3.11-1.el7.x86_64
rh-php73-php-gd-7.3.11-1.el7.x86_64
php73-php-process-7.3.20-1.el7.remi.x86_64
php-common-7.3.20-1.el7.remi.x86_64
php-fpm-7.3.20-1.el7.remi.x86_64
rh-php73-php-process-7.3.11-1.el7.x86_64
rh-php73-php-pdo-7.3.11-1.el7.x86_64
php73-runtime-2.0-1.el7.remi.x86_64
rh-php73-php-xmlrpc-7.3.11-1.el7.x86_64

theFeu · August 14, 2020, 6:33am

Okay, those packages shouldn’t interfere with each other, from the looks of em.

The -process suffix was your original issue, if I recall correctly.

Okay, let’s find out under which user the services are running with:
grep icinga /etc/passwd; grep icinga /etc/group
They need to be a member of the icingaweb2 group, else it won’t work.

If you could also post the unit file you used for the director that would be extra helpful

gfragi · August 19, 2020, 10:08pm

This is strange all users are under icingaweb2 group

icinga:x:991:987:icinga:/var/spool/icinga2:/bin/bash
icingadirector:x:989:984::/var/lib/icingadirector:/bin/false

# grep icinga /etc/group
icinga:x:987:nagios
icingacmd:x:986:icinga,apache,nagios
icingaweb2:x:984:apache,nagios,icinga,icingadirector

[Unit]
Description=Icinga Director - Monitoring Configuration
Documentation=https://icinga.com/docs/director/latest/
Wants=network.target
[Service]
EnvironmentFile=-/etc/default/icinga-director
EnvironmentFile=-/etc/sysconfig/icinga-director
ExecStart=/usr/bin/icingacli director daemon run
ExecReload=/bin/kill -HUP ${MAINPID}
User=icingadirector
SyslogIdentifier=icingadirector
Type=notify

NotifyAccess=main
WatchdogSec=10
RestartSec=30
Restart=always

[Install]
WantedBy=multi-user.target

theFeu · August 21, 2020, 6:27am

Hm, alright, then let’s try to run this manually as the icingadirector user:

su - icingadirector -s /bin/bash
icingacli director daemon run --debug --trace

And while you’re at it you could check if the icingadirector user can:

read the config at /etc/icingaweb2/modules/director/config.ini
see the symlinks at /etc/icingaweb2/enabledModules
and change into the directories like so: cd /etc/icingaweb2/enabledModules/director

If all that works out, then the issue might have something to do with SElinux… If that is the case, I’d need to go ask a colleague who knows more about that topic…

gfragi · September 3, 2020, 8:09am

The following is more than weird…

-bash-4.2$ icingacli director daemon run --debug --trace
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/doc" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/monitoring" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/setup" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/audit" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/ipl" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/incubator" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reactbundle" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/pdfexport" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/idoreports" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reporting" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/graphite" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/director" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/trapdirector" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/x509" is not a symlink
ERROR: There is no such module or command: 'director'

Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/doc" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/monitoring" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/setup" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/audit" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/ipl" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/incubator" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reactbundle" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/pdfexport" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/idoreports" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reporting" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/graphite" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/director" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/trapdirector" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/x509" is not a symlink
USAGE: icingacli [module] <command> [action] [options]

AND…

-bash-4.2$ cat /etc/icingaweb2/modules/director/config.ini
[db]
resource = "director_db"
-bash-4.2$ cd /etc/icingaweb2/enabledModules
-bash: cd: /etc/icingaweb2/enabledModules: Permission denied
-bash-4.2$ cd /etc/icingaweb2/enabledModules/director
-bash: cd: /etc/icingaweb2/enabledModules/director: Permission denied

dgoetz · September 4, 2020, 9:30am

I am just here for the SELinux part.

So this should not be part of the problem as users are always unconfined if not explicitly some confinement is configured, so it would only deny very very rare cases which are highly insecure. But to remove them from the list to check please do two things.

First verify that the problem still exists even if SELinux is in permissive mode (can be changed simply by running setenforce 0 and verified by sestatus). If problem still exists it is not caused by SELinux, if not it is SELinux.

Second I want to have some SELinux specific information.

Output of sestatus
Output of semodule -l | grep -e icinga2 -e icingaweb2 -e nagios -e apache
Output of semanage boolean -l | grep icinga
Output of ps -eZ | grep httpd
Output of audit2allow -li /var/log/audit/audit.log
Output of ls -lZ /etc/icingaweb2/enabledModules

gfragi · September 22, 2020, 7:46am

Hello sorry for the late answer but I was OOF for a couple of weeks. Thank you very much on trying to help me. Below you may find the output of the commands you asked for.

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          error (Success)
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

apache  2.7.2
icinga2 0.2.0
icingaweb2      0.0.1
nagios  1.13.0

httpd_can_connect_icinga2_api  (on   ,   on)  Allow httpd to can connect icinga2 api
icinga2_can_connect_all        (off  ,  off)  Allow icinga2 to can connect all
icinga2_run_sudo               (off  ,  off)  Allow icinga2 to run sudo
httpd_can_manage_icingaweb2_config (on   ,   on)  Allow httpd to can manage icingaweb2 config
httpd_can_write_icinga2_command (on   ,   on)  Allow httpd to can write icinga2 command
icinga2adm_exec_content        (on   ,   on)  Allow icinga2adm to exec content



system_u:system_r:httpd_t:s0     3859 ?        06:52:13 php-fpm
system_u:system_r:httpd_t:s0     4659 ?        07:34:35 php-fpm
system_u:system_r:httpd_t:s0     5297 ?        00:00:06 httpd
system_u:system_r:httpd_t:s0     5382 ?        07:34:18 php-fpm
system_u:system_r:httpd_t:s0     7364 ?        06:53:12 php-fpm
system_u:system_r:httpd_t:s0     9826 ?        07:34:23 php-fpm
system_u:system_r:httpd_t:s0    12490 ?        07:32:50 php-fpm
system_u:system_r:httpd_t:s0    13302 ?        00:00:05 httpd
system_u:system_r:httpd_t:s0    13775 ?        06:55:28 php-fpm
system_u:system_r:httpd_t:s0    15055 ?        04:15:18 php-fpm
system_u:system_r:httpd_t:s0    15075 ?        04:15:11 php-fpm
system_u:system_r:httpd_t:s0    15076 ?        04:15:12 php-fpm
system_u:system_r:httpd_t:s0    15807 ?        00:03:53 php-fpm
system_u:system_r:httpd_t:s0    15812 ?        07:36:13 php-fpm
system_u:system_r:httpd_t:s0    16235 ?        07:36:46 php-fpm
system_u:system_r:httpd_t:s0    17133 ?        07:29:54 php-fpm
system_u:system_r:httpd_t:s0    17914 ?        00:00:03 httpd
system_u:system_r:httpd_t:s0    18584 ?        07:38:28 php-fpm
system_u:system_r:httpd_t:s0    20315 ?        00:08:51 httpd
system_u:system_r:httpd_t:s0    20684 ?        07:38:42 php-fpm
system_u:system_r:httpd_t:s0    20735 ?        00:00:00 httpd
system_u:system_r:httpd_t:s0    22485 ?        00:00:00 httpd
system_u:system_r:httpd_t:s0    24471 ?        00:00:00 httpd
system_u:system_r:httpd_t:s0    24472 ?        00:00:00 httpd
system_u:system_r:httpd_t:s0    25297 ?        03:47:24 php-fpm
system_u:system_r:httpd_t:s0    29241 ?        00:00:04 httpd
system_u:system_r:httpd_t:s0    29522 ?        00:00:04 httpd
system_u:system_r:httpd_t:s0    31393 ?        00:00:12 httpd
system_u:system_r:httpd_t:s0    31504 ?        02:25:46 php-fpm
system_u:system_r:httpd_t:s0    31555 ?        06:56:24 php-fpm
system_u:system_r:httpd_t:s0    31565 ?        02:25:36 php-fpm
system_u:system_r:httpd_t:s0    31566 ?        02:25:19 php-fpm
system_u:system_r:httpd_t:s0    31575 ?        02:25:39 php-fpm
system_u:system_r:httpd_t:s0    31576 ?        02:25:19 php-fpm
system_u:system_r:httpd_t:s0    31577 ?        02:25:27 php-fpm
system_u:system_r:httpd_t:s0    31578 ?        02:25:27 php-fpm
system_u:system_r:httpd_t:s0    31587 ?        02:25:29 php-fpm
system_u:system_r:httpd_t:s0    31588 ?        02:25:29 php-fpm
system_u:system_r:httpd_t:s0    31589 ?        02:25:15 php-fpm
system_u:system_r:httpd_t:s0    31590 ?        02:25:21 php-fpm
system_u:system_r:httpd_t:s0    31591 ?        02:25:26 php-fpm
system_u:system_r:httpd_t:s0    31608 ?        02:25:23 php-fpm
system_u:system_r:httpd_t:s0    31609 ?        02:25:26 php-fpm
system_u:system_r:httpd_t:s0    31610 ?        02:25:24 php-fpm
system_u:system_r:httpd_t:s0    31611 ?        02:25:31 php-fpm
system_u:system_r:httpd_t:s0    31612 ?        02:25:29 php-fpm
system_u:system_r:httpd_t:s0    31708 ?        02:25:32 php-fpm




#============= postfix_postdrop_t ==============
allow postfix_postdrop_t icinga2_t:fifo_file { getattr write };

#============= system_mail_t ==============
allow system_mail_t icinga2_t:fifo_file getattr;



lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 audit -> /usr/share/icingaweb2/modules/audit
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 director -> /usr/share/icingaweb2/modules/director
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 doc -> /usr/share/icingaweb2/modules/doc
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 graphite -> /usr/share/icingaweb2/modules/graphite
lrwxrwxrwx. root   icingaweb2 unconfined_u:object_r:icingaweb2_config_t:s0 idoreports -> /usr/share/icingaweb2/modules/idoreports
lrwxrwxrwx. root   icingaweb2 unconfined_u:object_r:icingaweb2_config_t:s0 incubator -> /usr/share/icingaweb2/modules/incubator
lrwxrwxrwx. root   icingaweb2 unconfined_u:object_r:icingaweb2_config_t:s0 ipl -> /usr/share/icingaweb2/modules/ipl
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 monitoring -> /usr/share/icingaweb2/modules/monitoring
lrwxrwxrwx. root   icingaweb2 unconfined_u:object_r:icingaweb2_config_t:s0 pdfexport -> /usr/share/icingaweb2/modules/pdfexport
lrwxrwxrwx. root   icingaweb2 unconfined_u:object_r:icingaweb2_config_t:s0 reactbundle -> /usr/share/icingaweb2/modules/reactbundle
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 reporting -> /usr/share/icingaweb2/modules/reporting
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 setup -> /usr/share/icingaweb2/modules/setup
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 trapdirector -> /usr/share/icingaweb2/modules/trapdirector
lrwxrwxrwx. apache icingaweb2 system_u:object_r:icingaweb2_config_t:s0 x509 -> /usr/share/icingaweb2/modules/x509

gfragi · September 22, 2020, 7:50am

Hello I think you find it. Let me know how to change that specific permissions without braking something.

-bash-4.2$ icingacli director daemon run --debug --trace
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/doc" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/monitoring" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/setup" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/audit" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/ipl" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/incubator" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reactbundle" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/pdfexport" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/idoreports" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reporting" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/graphite" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/director" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/trapdirector" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/x509" is not a symlink
ERROR: There is no such module or command: 'director'

Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/doc" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/monitoring" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/setup" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/audit" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/ipl" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/incubator" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reactbundle" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/pdfexport" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/idoreports" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/reporting" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/graphite" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/director" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/trapdirector" is not a symlink
Found invalid module in enabledModule directory "/etc/icingaweb2/enabledModules": "/etc/icingaweb2/enabledModules/x509" is not a symlink



-bash-4.2$ ls /etc/icingaweb2/enabledModules
ls: cannot access /etc/icingaweb2/enabledModules/doc: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/monitoring: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/setup: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/audit: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/ipl: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/incubator: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/reactbundle: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/pdfexport: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/idoreports: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/reporting: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/graphite: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/director: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/trapdirector: Permission denied
ls: cannot access /etc/icingaweb2/enabledModules/x509: Permission denied
audit  director  doc  graphite  idoreports  incubator  ipl  monitoring  pdfexport  reactbundle  reporting  setup  trapdirector  x509


-bash-4.2$ cd /etc/icingaweb2/enabledModules/director
-bash: cd: /etc/icingaweb2/enabledModules/director: Permission denied

dgoetz · September 22, 2020, 8:20am

This makes me wonder because it seams like something is broken on the system, which could explain any strange behaviour, but if SELinux works at least somehow it should have caused some more lines in the audit2allow output if it was blocking the file access.

But I thing I found it from the earlier output.

The capital S means there is no x behind it, can you please run chmod g+x /etc/icingaweb2/enabledModules to allow the group browsing the directory.

gfragi · September 22, 2020, 9:47am

YES you found it!!!

Thank you both (Feu & Dirk) for your time!!!