DISK CRITICAL - /run/user/0/doc is not accessible: Permission denied, Fedora 34

RobbieTheK · October 8, 2021, 2:42pm

new installation of icingaweb2-2.9.3-1/icinga2-bin-2.13.1-1, PHP 7.4.24 on Fedora 34

Loaded Libraries
icinga/icinga-php-library 0.6.1
icinga/icinga-php-thirdparty 0.10.0

Loaded Modules
doc 2.9.3
monitoring 2.9.3

I don’t see in the logs the error Icinga2 is finding specifically: DISK CRITICAL - /run/user/0/doc is not accessible: Permission denied

How do you dismiss/suppress this message?

Edit: all new Fedora servers I add to the hosts file show this error. This file does not exist so perhaps there’s an exclude/ignore option that I’m not finding?

leeclemens · October 9, 2021, 2:23am

The plugin is probably running as the icinga user, so it makes sense it does not have permission to /run/user/0/. I believe the default nagios plugin excludes /run. Can you provide the command from Inspect if you’re using Director? What Command are you running? Output of df -T?

RobbieTheK · October 9, 2021, 3:40am

Yes I see it is from ps. But note that there is no /run/user/0/doc directory but there are other directories there so it’s as if something is looking for this doc directory/file.

 ls -l /run/user/0/
total 0
srw-rw-rw- 1 root root   0 Sep 24 20:42 bus
drwx------ 2 root root  60 Oct  6 14:56 dconf
dr-x------ 2 root root   0 Dec 31  1969 doc
drwx------ 3 root root  60 Oct  6 14:39 gnupg
drwx------ 2 root root  40 Oct  6 14:56 gvfs
drwx------ 2 root root  40 Oct  6 14:56 gvfsd
drwx------ 2 root root  60 Oct  7 09:12 keyring
srw-rw-rw- 1 root root   0 Sep 24 20:42 pipewire-0
-rw-r----- 1 root root   0 Oct  7 09:12 pipewire-0.lock
drwx------ 2 root root  40 Oct  6 14:57 pulse
drwxr-xr-x 6 root root 160 Oct  7 09:12 systemd

Not using Director but looking into installing it.

I’m not running any command this is from the GUI.

 df -T
Filesystem                   Type      1K-blocks       Used Available Use% Mounted on
devtmpfs                     devtmpfs   32564940          0  32564940   0% /dev
tmpfs                        tmpfs      32585572          4  32585568   1% /dev/shm
tmpfs                        tmpfs      13034232       2200  13032032   1% /run
/dev/mapper/fedora-root      xfs       629170016   16520288 612649728   3% /
tmpfs                        tmpfs      32585576       1428  32584148   1% /tmp
/dev/mapper/fedora-home      xfs        78604800     812816  77791984   2% /home
/dev/mapper/fedora-scratch   xfs      5728252316 5341539972 386712344  94% /scratch
/dev/sda2                    ext4        1889292     277940   1497332  16% /boot
/dev/mapper/fedora-var       xfs       488045740    7488700 480557040   2% /var
tmpfs                        tmpfs       6517112         92   6517020   1% /run/user/0
/dev/mapper/fedora-usr_local xfs        78604800     706524  77898276   1% /usr/local

leeclemens · October 9, 2021, 4:43pm

I do see /run/user/0/doc listed in your output, so I’m not sure if you missed it or typo’ed in your response. It’s under the mount point, so I’m not sure why it’s throwing that error exactly.

What command are you running for this check?

The CheckCommand configuration included with Icinga2 excludes tmpfs by default.

You can test the options passed to check_disk by running the command yourself (as icinga).

RobbieTheK · October 9, 2021, 6:22pm

Oops you are right it is there as a 0kb file, my bad.

When you ask me what command am I running, are you wanting to know what is in the hosts.conf file? I just have this and every Fedora host brings back this error in the GUI:

vars.disks["disk"] = { } vars.disks["disk /"] = { disk_partitions = "/"

With su icinga -c check_disk I get a This account is currently not available message. When I run it as a non root user I get the same message DISK CRITICAL - /run/user/0/doc is not accessible: Permission denied

Perhaps that’s a clue? User icinga has a nologin option set.

leeclemens · October 9, 2021, 6:47pm

It is sort of a 0kb file, but more importantly it is a directory.

When I ask about the command I am munging you directly and Icinga2 as you have it configured…I’m more interested in what specific command is being executed (regardless of who or what).

You can specify the shell by passing -s /bin/'bash to the su command.

Can you post the complete CheckCommand you are using? It looks like some may have been cutoff. I’m guessing you need to simply add the excludes that are included in Icinga’s default config to your own.

RobbieTheK · October 9, 2021, 7:11pm

Woops, directory, yep.

su - icinga -s /bin/bash -c "/usr/lib64/nagios/plugins/check_disk -m" DISK CRITICAL - /run/user/0/doc is not accessible: Permission denied

I only see comments:
grep CheckCommand /etc/icinga2/*/* /etc/icinga2/conf.d/commands.conf:#object CheckCommand "check_docker_by_ssh" { /etc/icinga2/conf.d/services.conf: * The CheckCommand objects ping4, ping6, etc /etc/icinga2/conf.d/services.conf.orig: * The CheckCommand objects ping4, ping6, etc /etc/icinga2/conf.d/templates.conf: * The CheckCommand object hostalive is provided by

Ok where is this done and what do I exclude?

leeclemens · October 9, 2021, 7:28pm

You can verify the arguments by running the command directly. Use --help for more details.

 -X, --exclude-type=TYPE
    Ignore all filesystems of indicated type (may be repeated)

From Icinga2’s included config:

         "-X" = {
             value = "$disk_exclude_type$"
             description = "Ignore all filesystems of indicated type (may be repeated)"
             repeat_key = true
         }

     vars.disk_exclude_type = [
         "none",
         "tmpfs",   <===
...

Hopefully that helps you in the right direction. Otherwise sharing more configs you are using would be useful.

RobbieTheK · October 9, 2021, 9:38pm

Yes I see that in /usr/share/icinga2/include/command-plugins.conf Does the none mean it’s disabled? It seems that /run is being included even though df -T showed it’s a tmpfs file system.

Here is icinga2 object list --type Service from one of the servers:

Object 'ourserver.edu!disk' of type 'Service':
  % declared in '/etc/icinga2/conf.d/services.conf', lines 65:1-65:53
  * __name = "ourserver.edu!disk"
  * action_url = ""
  * check_command = "disk"
    % = modified in '/etc/icinga2/conf.d/services.conf', lines 68:3-68:24
  * check_interval = 60
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 28:3-28:21
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "disk"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_ignore_states = null
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "ourserver.edu"
    % = modified in '/etc/icinga2/conf.d/services.conf', lines 65:1-65:53
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 5
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 27:3-27:24
  * name = "disk"
    % = modified in '/etc/icinga2/conf.d/services.conf', lines 65:1-65:53
  * notes = ""
  * notes_url = ""
  * package = "_etc"
    % = modified in '/etc/icinga2/conf.d/services.conf', lines 65:1-65:53
  * retry_interval = 30
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 29:3-29:22
  * source_location
    * first_column = 1
    * first_line = 65
    * last_column = 53
    * last_line = 65
    * path = "/etc/icinga2/conf.d/services.conf"
  * templates = [ "disk", "generic-service" ]
    % = modified in '/etc/icinga2/conf.d/services.conf', lines 65:1-65:53
    % = modified in '/etc/icinga2/conf.d/templates.conf', lines 26:1-26:34
  * type = "Service"
  * vars
    % = modified in '/etc/icinga2/conf.d/services.conf', lines 70:3-70:16
  * volatile = false
  * zone = ""

Edit: how can I use vars.disk_ignore_eregi_path? This way doesn’t work:

 vars.disks["disk"] = {
    }
  vars.disks["disk /"] = {
    disk_partitions = "/"
    vars.disk_ignore_eregi_path = [ "/run" ]
  }

leeclemens · October 10, 2021, 1:40am

/run is not a mount point, so I’m not sure ignoring it will help.

Can you run check_disk -X tmpfs as icinga? You can run these commands from the command line to verify what Icinga is running.

RobbieTheK · October 10, 2021, 2:30am

su - icinga -s /bin/bash -c "/usr/lib64/nagios/plugins/check_disk -X tmpfs"

DISK CRITICAL - /run/user/0/doc is not accessible: Permission denied

leeclemens · October 10, 2021, 3:50am

That is odd. /run/user/0 should be root:root 700. Can you confirm that? If that is the case, the icinga user shouldn’t even be able to see doc - so something else is going on here.

RobbieTheK · October 10, 2021, 3:53am

ls -l /run/user/0
total 0
srw-rw-rw- 1 root root 0 Sep 24 20:42 **bus**
drwx------ 2 root root 60 Oct 6 14:56 **dconf**
dr-x------ 2 root root 0 Dec 31 1969 **doc**
drwx------ 3 root root 60 Oct 6 14:39 **gnupg**
drwx------ 2 root root 40 Oct 6 14:56 **gvfs**
drwx------ 2 root root 40 Oct 6 14:56 **gvfsd**
drwx------ 2 root root 60 Oct 7 09:12 **keyring**
srw-rw-rw- 1 root root 0 Sep 24 20:42 **pipewire-0**
-rw-r----- 1 root root 0 Oct 7 09:12 pipewire-0.lock

drwx------ 2 root root 40 Oct 6 14:57 **pulse**
drwxr-xr-x 6 root root 160 Oct 7 09:12 **systemd**

ls -ld /run/user/0
drwx------ 10 root root 260 Oct 7 09:12 **/run/user/0**

Does doc having only 500 permissions an issue? The timestamp is also 1969.

Note I get this same error on 5 different Fedora hosts. How do debug this? More verbose logging?

leeclemens · October 10, 2021, 1:55pm

If you exclude it explicitly it should work. Still doesn’t explain what the issue is though:

check_disk -x /run/user/0/doc

You’ll probably want to use the CheckCommand that comes with some reasonable defaults (e.g. excluding tmpfs) as well.

RobbieTheK · October 10, 2021, 2:37pm

Yes that works:

 su - icinga -s /bin/bash -c "/usr/lib64/nagios/plugins/check_disk -x /run/user/0/doc"
DISK OK - free space: /dev 31801 MiB (100.00% inode=100%); /dev/shm 31821 MiB (99.99% inode=100%); /run 12726 MiB (99.98% inode=100%); / 598293 MiB (97.37% inode=100%); /tmp 31820 MiB (99.99% inode=100%); /home 75968 MiB (98.96% inode=100%); /scratch 431803 MiB (7.71% inode=84%); /boot 1462 MiB (84.34% inode=100%); /var 469037 MiB (98.41% inode=100%); /run/user/0 6364 MiB (99.99% inode=100%); /usr/local 76072 MiB (99.10% inode=100%);| /dev=0MiB;;;0;31801 /dev/shm=0MiB;;;0;31821 /run=2MiB;;;0;12728 /=16130MiB;;;0;614423 /tmp=1MiB;;;0;31821 /home=793MiB;;;0;76762 /scratch=5162192MiB;;;0;5593996 /boot=271MiB;;;0;1845 /var=7569MiB;;;0;476607 /run/user/0=0MiB;;;0;6364 /usr/local=689MiB;;;0;76762

What do I put in which config file then?

The default is set in /etc/icinga2/conf.d/hosts.conf
check_command = "hostalive"

GUI shows it should be excluded:

leeclemens · October 10, 2021, 6:16pm

Excluding it by type does not work (you tried that with -X earlier). The one that works (-x) excludes it by path (note the case difference).

RobbieTheK · October 10, 2021, 9:58pm

Ok so how is this done via a config file? Is this a feature request then?

leeclemens · October 11, 2021, 12:08am

The CheckCommand configuration that is packaged with Icinga2 has:

                "-x" = {
                        value = "$disk_partitions_excluded$"
                        description = "Ignore device (only works if -p unspecified)"
                }

No need for a feature request, you just need to configure your own CheckCommand (or use Director and import the packaged one) to accept that variable and configure your service accordingly.

RobbieTheK · October 11, 2021, 2:06am

Right so where do I drop in the value for what’s in /usr/share/icinga2/include/command-plugins.conf?

But is it a bug that -X doesn’t detect a tmpfs file system? I can’t be the 1st to report this.

leeclemens · October 11, 2021, 2:59am

I’d suggest using Director and importing it. If you want to configure it, you can do so anywhere that is imported (e.g. conf.d). I’d have to see your configs to help more in that regard.

If it is a bug, it is with the Nagios plugin or Fedora.