Perfdata not processing when exit critical but processing when check is ok or warning

Icinga 2
1

Hi I noticed I was not receiving critical emails for service checks but recovery emails were ok

.
So I cheched my mail script and logs where I found the “$service.output$” was empty
I spent time checking the my python check script to find a discrepency in the output when critical but everything looked ok.
using
OK = 0
WARNING = 1
CRITICAL = 2
UNKNOWN = 3
I tested with a failing condition (same code I just change the return value)

If I change the exit value to OK perfdata is processed and “$service.output$” is populated.
If I change the exit value to WARNING perfdata is processed and “$service.output$” is populated.
but
If I change the exit value to CRITICAL perfdata is not processed and “$service.output$” is not populated.

I have checked the logs and tried debug logging but cant find any errors
How is the perdata treated differently when CRITICAL exit code ??
How can I find out what is going wrong? where should I look next?

Python return block

perfdata = " | 'query_results_total'=" + str(result_count)+" 'time'="+ str(result_time)
msg = 'CRITICAL Query returned 0 results  ' + perfdata
print(msg)
sys.exit(CRITICAL)

Critical output from script would be:
CRITICAL Query returned 0 results | 'query_results_total'=0 'time'=38

System info

using ubuntu 20.04
|cinga Web 2 Version|2.11.3
Git commit f917436a894c1f4148a9c3d6a27f9c20f204e44f
PHP Version 7.4.3-4ubuntu2.17

Loaded Libraries

icinga/icinga-php-thirdparty 0.11.0
icinga/icinga-php-library 0.10.1

Loaded Modules

director 1.9.1
doc 2.11.3
grafana 1.4.2
incubator 0.18.0
ipl v0.5.0
monitoring 2.11.3
reactbundle

2

Hi
I continued to test and even with no perfdata its not working so perfdata is not the issue.
I noticed that on the same setup nrpe critical checks are treated correctly, so its something else going on with these critical errors that is stopping the plugin output being processed.
I’m not sure what else I can try!
Peter

3

Python return block

perfdata = " | ‘query_results_total’=" + str(result_count)+" ‘time’="+
str(result_time) msg = 'CRITICAL Query returned 0 results ’ + perfdata
print(msg)
sys.exit(WARNING)

Try changing that last line to “sys.exit(CRITICAL)” :slight_smile:

Antony.

4

LOL …that was just a copy paste error from a test. It is definatly failing with the
sys.exit(CRITICAL) and exit(CRITICAL)
Hey but thanks for taking the time to read. :slight_smile:

5

Bumping this since its quite a problem
when exit code from python script is 2 the service output data is not populated.
If i change the exit code to 1 the data is processed.
This is impacting my critical alerts.
Any ideas ?

6

The full script might help :thinking:

7

Using this mockup of your script, it works fine for me for all states

#!/usr/bin/python3
import sys
OK = 0
WARNING = 1
CRITICAL = 2
UNKNOWN = 3

result_time = 110
result_count = 99990
perfdata = " | 'query_results_total'=" + str(result_count)+" 'time'="+ str(result_time)
msg = 'CRITICAL Query returned 0 results  ' + perfdata
print(msg)
sys.exit(WARNING)

image
image
image
image

As you can see all show the perfdata values. I didn’t change the output text.

I would also advise to use an f-string when creating your perfdata output, it’s easier to read.

perfdata = f"| 'query_results_total'={result_count} 'time'={result_time}"
msg = f"CRITICAL Query returned {result_count} results {perfdata}"

I would guess there is a problem somewhere else in the script.

8

Hi I took the scrip apart and rebuilt it bit at a time. I found the error which was an exception that was catching the critical exits !
It was a basic part of the script that I had used for all my scripts and had not caused a problem untill recently (which was strange)
so I hold my hands up and say … my fault!
Thanks for taking the time to look into this.
Peter