Icinga in LXC. 100 % CPU when commands run over SSH

Icinga 2
1

Icinga2 main command is slow when executed over ssh connection in LXC container. Can I help with debugging this issue?

Other commands are unaffected by this issue, affected even when running the command in existing ssh session.

Normal run in our deployments:

time ssh <redacted_user>@icinga "sudo icinga2 daemon -C" 1>/dev/null 

real	0m1.613s
user	0m0.015s
sys	0m0.019s

# SSH Command to LXC container
real	7m38,839s
user	0m0,012s
sys	0m0,005s

Icinga informations:

System information:
  Platform: Ubuntu
  Platform version: 18.04.3 LTS (Bionic Beaver)
  Kernel: Linux
  Kernel version: 5.3.8-300.fc31.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 8.3.0
  Build host: runner-LTrJQZ9N-project-298-concurrent-0

Container host:

Fedora 31
Linux ondrejkolin 5.3.8-300.fc31.x86_64 #1 SMP Tue Oct 29 14:28:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Hope I can help debugging with what is going on there. :slight_smile:

2

Icinga takes 100% CPU there:

Screenshot%20from%202019-11-13%2015-11-19
Screenshot%20from%202019-11-13%2015-11-191195×78 17.4 KB

3

It looks like it hangs in libthread_db.so.1

root@icinga_server:/var/local/ondrejkolin# gdb /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Complete: gdb output
https://paste.gnome.org/pp9lke3oj

4

With using containers, shouldn’t one just exec commands in there instead of going the SSH route? Does it make a difference?

5

Of course, it does, but I wanted to test my ansible setup. Executing it directly is ok, it gets its job done. I think the enviroment gets somehow messed up. I’ll will give it a bit more of testing and I’ll try report this back.

6

Got strace ouptut: strace2.txt (33.1 KB)