Discrepancy on the check disk command in icingaweb2

Hello everybody,
i do have an issue with one of my icinga2 checks.

I am checking a hardware machine (Proxmox VE 7.4) on its disks.

The problem i have, is it shows “disk critical” even though it is fine.

As visible on the added picture, at the header “Plugin Output” i do get a “disk critical” for 0 MB free space on “/”.
And that even while a slight look down into “Performance data” there is the “/” with only 288 GiB out of 464 GiB full.

here the configs:

service check:

apply Service "disk" {
  check_command = "disk"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.linuxbasics
  assign where host.vars.windows
}

servers conf

object Host "srv-loc-pve2" {
  check_command = "hostalive"
  /* Import the default host template defined in `templates.conf`. */
  import "generic-host"

  vars.client_endpoint = name

  vars.proxmox = true
  vars.servers = true

  vars.all = true

  vars.linuxbasics = true
  vars.loc.w = true

  /* Specify the address attributes for checks e.g. `ssh` or `http`. */
  address = "192.168.40.101"
  address6 = "::1"

  /* Set custom attribute `os` for hostgroup assignment in `groups.conf`. */
  vars.os = "Linux"

  vars.disks["disk /"] = {
    disk_partitions = "/"
  }

  vars.procs_warning = "500"
  vars.procs_critical = "600"

  /* Define notification mail attributes for notification apply rules in `notifications.conf`. */
  vars.notification["mail"] = {
    /* The UserGroup `icingaadmins` is defined in `users.conf`. */
    groups = [ "icingaadmins" ]
  }
}

object Endpoint "srv-loc-pve2" {
    host = "srv-loc-pve2"
}

object Zone "srv-wis-pve2" {
    endpoints = [ "srv-wis-pve2" ]
    parent = "master"
}

Your zone definition for that host is obviously wrong. What does Check Source in icingaweb2 report?

My apologies for my late reply.
the zone definition in the post definitely looks wrong, but that was only in the post here and not in my .conf file. sorry for the confusion

Check source is pointing to the correct hostname and IP address.

could you please tell me, is there anything else quite obviously wrong?

object Host "srv-loc-pve2" {
  check_command = "hostalive"
  /* Import the default host template defined in `templates.conf`. */
  import "generic-host"

  vars.client_endpoint = name

  vars.proxmox = true
  vars.servers = true

  vars.all = true

  vars.linuxbasics = true
  vars.loc.w = true

  /* Specify the address attributes for checks e.g. `ssh` or `http`. */
  address = "192.168.40.101"
  address6 = "::1"

  /* Set custom attribute `os` for hostgroup assignment in `groups.conf`. */
  vars.os = "Linux"

  vars.disks["disk /"] = {
    disk_partitions = "/"
  }

  vars.procs_warning = "500"
  vars.procs_critical = "600"

  /* Define notification mail attributes for notification apply rules in `notifications.conf`. */
  vars.notification["mail"] = {
    /* The UserGroup `icingaadmins` is defined in `users.conf`. */
    groups = [ "icingaadmins" ]
  }
}

object Endpoint "srv-loc-pve2" {
    host = "srv-loc-pve2"
}

object Zone "srv-loc-pve2" {
    endpoints = [ "srv-loc-pve2" ]
    parent = "master"
}

You can remove that when you use command_endpoint = host.name in your services definition instead.

This is defined but not used.

However, both mistakes has nothing to do with your original issue. But, I have no idea what could be wrong. Maybe you try that check plugin manually, first as root second as user icinga or nagios (depending on your distribution).

thanks you for pointing out those two things!

regarding the topic and trying the check plugin manually.

did it as root on the host itself:

/usr/lib/nagios/plugins# ./check_disk -w 10% -c 5%
DISK CRITICAL - free space: /dev 64402 MB (100% inode=99%); /run 12885 MB (99% inode=99%); / 0 MB (0% inode=-)]; /dev/shm 64414 MB (99% inode=99%); /run/lock 5 MB (100% inode=99%); /boot/efi 1021 MB (99% inode=-); /file2 3237796 MB (42% inode=-); /etc/pve 127 MB (99% inode=99%); /run/user/0 12887 MB (100% inode=99%);| /dev=0MB;57961;61181;0;64402 /run=1MB;11598;12242;0;12887 /=298489MB;428323;452119;0;475915 /dev/shm=21MB;57991;61213;0;64435 /run/lock=0MB;4;4;0;5 /boot/efi=0MB;918;969;0;1021 /file2=4392440MB;6867796;7249340;0;7630885 /etc/pve=0MB;115;121;0;128 /run/user/0=0MB;11598;12242;0;12887

root@srv-loc-pve2/usr/lib/nagios/plugins# ./check_disk -f
DISK OK - free space: /dev 64402 MB (100% inode=99%); /run 12885 MB (99% inode=99%); / 0 MB (0% inode=-); /dev/shm 64414 MB (99% inode=99%); /run/lock 5 MB (100% inode=99%); /boot/efi 1021 MB (99% inode=-); /file2 3237796 MB (42% inode=-); /etc/pve 127 MB (99% inode=99%); /run/user/0 12887 MB (100% inode=99%);| /dev=0MB;;;0;64402 /run=1MB;;;0;12887 /=298489MB;;;0;298489 /dev/shm=21MB;;;0;64435 /run/lock=0MB;;;0;5 /boot/efi=0MB;;;0;1021 /file2=4392440MB;;;0;7630236 /etc/pve=0MB;;;0;128 /run/user/0=0MB;;;0;12887

root@srv-loc-pve2:/usr/lib/nagios/plugins# ./check_disk -f -w 10% -c 5%
DISK CRITICAL - free space: /dev 64402 MB (100% inode=99%); /run 12885 MB (99% inode=99%); / 0 MB (0% inode=-)]; /dev/shm 64414 MB (99% inode=99%); /run/lock 5 MB (100% inode=99%); /boot/efi 1021 MB (99% inode=-); /file2 3237794 MB (42% inode=-); /etc/pve 127 MB (99% inode=99%); /run/user/0 12887 MB (100% inode=99%);| /dev=0MB;57961;61181;0;64402 /run=1MB;11598;12242;0;12887 /=298490MB;268641;283565;0;298490 /dev/shm=21MB;57991;61213;0;64435 /run/lock=0MB;4;4;0;5 /boot/efi=0MB;918;969;0;1021 /file2=4392441MB;6867212;7248724;0;7630236 /etc/pve=0MB;115;121;0;128 /run/user/0=0MB;11598;12242;0;12887

does not correlate with the linux command df -h

root@srv-loc-pve2:/usr/lib/nagios/plugins# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             63G     0   63G   0% /dev
tmpfs            13G  1.6M   13G   1% /run
/dev/sda3       465G  292G  174G  63% /
tmpfs            63G   22M   63G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sda2      1022M  592K 1022M   1% /boot/efi
/dev/sdc        7.3T  4.2T  3.1T  58% /file2
/dev/fuse       128M   20K  128M   1% /etc/pve
tmpfs            13G     0   13G   0% /run/user/0

as df -h shows “/” has 37% free space and that is the same as the picture provided above with “Performance data” showing the right result. but the check itself telling 0% free space. :upside_down_face:

i also tried to reinstall icinga host did not change my situation.

The problem will not be icinga, but the check plugin.
Is your / partition a BTRFS? Then this issue might be interesting: check_disk: BTRFS show wrong free space (0% inode) · Issue #1717 · monitoring-plugins/monitoring-plugins · GitHub
Or this: check_disk: Fixing the stuff that is broken on btrfs by waja · Pull Request #1388 · monitoring-plugins/monitoring-plugins · GitHub

you could try a different plugin, one of the LinuxFabrik checks for example, and see if that works as expected: monitoring-plugins/check-plugins/disk-usage at main · Linuxfabrik/monitoring-plugins · GitHub

thank you, yes my / partition is btrfs. gonna have a look into those topics.