Check execution time higher than manual execution

steffeneichler · January 19, 2022, 4:04pm

Hi,

if I execute a check plugin via commandline, the execution time is below 1 second.
If icinga executes the same command it runs constantly 10 seconds.

The check is executed local on the icinga server and used to measure how fast a connection to a kubernetes cluster. It executes an kubectl describe namespace.

When I first established the service this problem didn’t occurred. Over the time I added some other services which use the same plugin but to another targets. Then something happened and on all these services have the same problem but with slightly different but higher execution times as execution via commandline.
It looks like the execution via icinga stuck somewhere, but I’m not able to find out where.

My initial check plugin was written in bash. After the problem occurred, I wrote a new one which does the same in python. But the problem is still the same.

Version used (icinga2 --version)
r2.13.2-1
Operating System and version
Debian 11
Enabled features (icinga2 feature list)
Disabled features: command compatlog debuglog gelf graphite influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-mysql influxdb mainlog notification

I hope you can help me to find new ideas what could cause the problem.

Kind regards
Steffen

steffeneichler · January 19, 2022, 4:07pm

Al2Klimov · January 25, 2022, 11:53am

Hi @steffeneichler!

Icinga 2 runs checks with low priority:

github.com

Icinga/icinga2/blob/185fab3761dc4283844bab85e689ed543ee43160/lib/base/process.cpp#L168

    
      
          			_exit(128);
          		}
          
          
		(void)close(fds[0]);
          		(void)close(fds[1]);
          		(void)close(fds[2]);
          
          
#ifdef HAVE_NICE
          		if (adjustPriority) {
          			// Cheating the compiler on "warning: ignoring return value of 'int nice(int)', declared with attribute warn_unused_result [-Wunused-result]".
          			auto x (nice(5));
          			(void)x;
          		}
          #endif /* HAVE_NICE */
          
          
		sigset_t mask;
          		sigemptyset(&mask);
          		sigprocmask(SIG_SETMASK, &mask, nullptr);
          
          
		if (icinga2_execvpe(argv[0], argv, envp) < 0) {
          			char errmsg[512];

in contrast to you on the command line. I guess that slows them down.

Best,
A/K

steffeneichler · January 25, 2022, 3:40pm

hi @Al2Klimov,

thanks for the hint. It sounds like an priority problem. But how can I change that behavior?

Al2Klimov · January 25, 2022, 5:19pm

You can’t. And you shouldn’t. Checks shall not slow down the rest of the system on low resources. The only alternative: the rest of the system slows down the checks.

steffeneichler · January 27, 2022, 4:01pm

hi,

thanks for your answers @Al2Klimov . I tried to renice the processes started by the check. Unfortunately without success. The execution time didn’t change.