Hi,
I have encountered an issue when using docker with the default check_ping.
This causes a lot of defunct zombie processes on the underlying operating system. Eventually it runs out of PIDs and then ‘resource unavailable’ errors start popping up, with the eventual death of the server.
This takes time to get to this point, depending on how many PIDs you allow on the host server:
nagios 1205 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1207 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1211 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1213 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1214 0.0 0.0 0 0 ? ZN 16:32 0:00 [ping]
nagios 1215 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1217 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1219 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1220 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1222 0.0 0.0 0 0 ? ZN 16:32 0:00 [ping]
nagios 1223 0.0 0.0 0 0 ? ZNs 16:32 0:00 [check_ping]
nagios 1224 0.0 0.0 0 0 ? ZN 16:32 0:00 [ping]
The important things to note is, this appears to only happen to this check_ping command. I read the manuals and understand this parses the output of the underlying /bin/ping.
However when I switch this to check_icmp which has ICMP built within the binary then this problem disappears.
I witnessed this in Centos8 and Debian10.
I assume this is a bug, but not sure if it’s icinga, check_ping or just the setup within docker?
I notice there may be an equivalent issue here, but unsure if it’s the same:
root@icinga2:/tmp/health_monitoring_plugins-0.2.1# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)
Copyright © 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later http://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Debian GNU/Linux
Platform version: 10 (buster)
Kernel: Linux
Kernel version: 4.19.0-14-amd64
Architecture: x86_64
Build information:
Compiler: GNU 8.3.0
Build host: runner-hh8q3bz2-project-298-concurrent-0
OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
Best Regards,