Icinga2 too many open files

Good day all

I am hoping this post finds everyone well.

I am hoping that someone can assist us with an issue that we are having with icinga2. 

We are currently running icinga2 on ubuntu 14.04 and we are monitoring a LOT of services with this. 

 Recently we have been getting cases where icinga acts up with errors stating "unable to open pipe" or "too many open files" 

After investigating we saw that it is opening way above the default configured amount of 16384 files. 

We attempted to set this via OS paramaters which had no effect and later found that icinga has its own limits. 

  We then tried to change these based on a lot of posts that we found, however it still does not recognize the changes even after server reboots. 

Can someone perhaps assist us with info on how we can change or fix this ? 

icinga version is : r2.4.1-1

Regards

I assume v2.4.1-1 is the Icinga Web 2 version, and not the Icinga 2 core version. If it is the Icinga2 core:
Update!

From this gthub issue: https://github.com/Icinga/icinga2/issues/4547

Either increase the file descriptor ulimit or modify concurrent_checks in your CheckerComponent configuration object to run fewer checks in parallel.

Are you using the command pipe as command transport?
You could also try switching to the API for command transport instead of command pipes.

I’d say that 2.4.1 is the core version, that was shipped with Ubuntu 14. So he’s not using our official repository on packages.icinga.com.

Either way, before upgrading your Icinga2 package, also ensure to upgrade Ubuntu. 14.04 LTS is EOL in 21 days.

Cheers,
Michael

We have now upgraded to icinga-core 2.10 as well as icingaweb upgraded and director installed.

concurrent checks have been modified as well, however we would still like to increase the file descriptor limit.

Anybody know how or where we can do this as the settings we have added for ICINGA2_RLIMIT_FILES is still not being honoured.

Where did you apply this setting, in the sysconfig file?

Cheers,
Michael

We have added the settings to /etc/sysconfig/icinga2 which is a symlink to /usr/lib/icinga2/icinga2 as per a previous post we found.

We also added this to the /etc/icinga2.conf as well as /etc/default/icinga2 as per other posts, however none of these are taking any effect as icinga limits still remains on the default limits.

Max open files 16384 16384 files

If you’re on Debian/Ubuntu, the sysconfig file for applications is located in /etc/default/, so you’ll need to edit /etc/default/icinga2 for Icinga 2. /etc/sysconfig is available on RHEL/SUSE based distributions.

Cheers,
Michael

We have added the configs to the file already :
root@icinga:/usr/share/icinga2/include# cat /etc/default/icinga2

default settings for icinga2’s initscript

#DAEMON_ARGS="-e /var/log/icinga2/icinga2.err"
ICINGA2_RLIMIT_FILES=65535
root@icinga:/usr/share/icinga2/include#

However, even after a server reboot, the limits are still not set correctly.

Max open files 16384 16384 files

Please share how you evaluate the limits (cli call)?

cat /proc/pid/limits

Ok, and are you using Systemd or SysVInit? Maybe you’re using an old version of those (changed during upgrade) and the included overrides are not intact anymore.

we are using systemd

Have you tried creating a file:
/etc/security/limits.d/10-icinga.conf

with content:
Icinga soft nofile 65535
icinga hard nofile 65535

And restart icinga.

1 Like

let me try that and will revert back soon.

still no change in the limits :

Limit Soft Limit Hard Limit Units
Max open files 16384 16384 files

If you su to icinga user and execute a:
ulimit -a
what is the result?

1 Like

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 64000
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 64000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

I have also added the limits for the user nagios which is the user we are using for icinga, however that also does not work.

Maybe there’s a global limit in Systemd which enforces this low limit.

Going to try the following now :

  1. create a new file /etc/systemd/system.conf.d/limits.conf with these contents:
[Manager]
DefaultLimitNOFILE=65535
  1. run systemctl daemon-reexec as root
  2. logout and login again
  3. check your new limit with ulimit -n .