Performance Data value is 0.00

Icinga Web 2
1

Hi there,

Got a bit of a strange issue and I’m not sure where to troubleshoot - I have a check command which was outputting performance data up until the start of this month - I only realised this now but I have not made any major changes, so I am unsure of why it has occurred.

Manually running the check command:

[root@satellite local]# '/usr/lib64/nagios/plugins/check_snmp_eaton_ups.py' '-C' 'public' '-H' '10.0.0.1' '-V' '1' '-t' 'input_voltage'

OK - Input Voltage is 244 VAC | 'input_voltage'=244VAC;;;;

Plugin output from Icinga Web:

Plugin Output

OK - Output Voltage is 229 VAC

Performance data

Label Value
output_voltage 0.00

I have other checks in the same python script that are outputting correctly - e.g:

[root@satellite local]# '/usr/lib64/nagios/plugins/check_snmp_eaton_ups.py' '-C' 'public' '-H' '10.2.60.100' '-V' '1' '-t' 'input_frequency'

OK - Input Frequency is 49 Hz | 'input_frequency'=49Hz;;;;

Plugin Output

OK - Input Frequency is 49 Hz

Performance data

Label Value
input_frequency 49.00

Service settings:

enable_active_checks true
enable_event_handler true
enable_flapping false
enable_notifications true
enable_passive_checks false
enable_perfdata true

What is interesting too is that we are exporting perfdata to OpenTSDB and we’re not even seeing the 0.00 result logged - it’s as if it’s not processing the performance data at all.

Debug log when running one of the problem checks (hostnames/IPs have been redacted):

[2019-09-25 11:37:18 +1000] notice/JsonRpcConnection: Received 'event::SetForceNextCheck' message from 'master1'

[2019-09-25 11:37:18 +1000] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from 'master1'

[2019-09-25 11:37:18 +1000] debug/CheckerComponent: Scheduling info for checkable 'UPS!Eaton UPS Environment Temperature' (2019-09-25 11:37:18 +1000): Object 'UPS!Eaton UPS Environment Temperature', Next Check: 2019-09-25 11:37:18 +1000(1.56938e+09).

[2019-09-25 11:37:18 +1000] debug/CheckerComponent: Executing check for 'UPS!Eaton UPS Environment Temperature'

[2019-09-25 11:37:18 +1000] debug/Checkable: Update checkable 'UPS!Eaton UPS Environment Temperature' with check interval '300' from last check time at 2019-09-25 11:36:53 +1000 (1.56938e+09) to next check time at 2019-09-25 11:42:10 +1000(1.56938e+09).

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::SetForceNextCheck' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::SetNextCheck' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::SetForceNextCheck' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'master1'

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::SetNextCheck' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Sending message 'event::SetNextCheck' to 'master1'

[2019-09-25 11:37:18 +1000] notice/Process: Running command '/usr/lib64/nagios/plugins/check_snmp_eaton_ups.py' '-C' 'public' '-H' '10.0.0.1' '-V' '1' '-t' 'environment_temperature': PID 21560

[2019-09-25 11:37:18 +1000] debug/CheckerComponent: Check finished for object 'UPS!Eaton UPS Environment Temperature'

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::SetNextCheck' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Sending message 'event::SetNextCheck' to 'master1'

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::CheckResult' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Sending message 'event::CheckResult' to 'master1'

[2019-09-25 11:37:18 +1000] notice/Process: PID 21560 ('/usr/lib64/nagios/plugins/check_snmp_eaton_ups.py' '-C' 'public' '-H' '10.0.0.1' '-V' '1' '-t' 'environment_temperature') terminated with exit code 0

[2019-09-25 11:37:18 +1000] debug/Checkable: Update checkable 'UPS!Eaton UPS Environment Temperature' with check interval '300' from last check time at 2019-09-25 11:37:18 +1000 (1.56938e+09) to next check time at 2019-09-25 11:42:10 +1000(1.56938e+09).

[2019-09-25 11:37:18 +1000] debug/DbEvents: add checkable check history for 'UPS!Eaton UPS Environment Temperature'

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::SetNextCheck' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Sending message 'event::SetNextCheck' to 'master1'

[2019-09-25 11:37:18 +1000] notice/ApiListener: Relaying 'event::CheckResult' message

[2019-09-25 11:37:18 +1000] notice/ApiListener: Sending message 'event::CheckResult' to 'master1'

I guess the last thing to note is that this is running from a satellite node connected to two masters - but all other checks seem to be recording performance data - it just seems to be the checks within this script, but nothing has really changed from a configuration perspective.

Any pointers would be much appreciated :slight_smile:

2

VAC & hz are not a valid uom. See http://nagios-plugins.org/doc/guidelines.html#AEN200

1 Like
3

Future supported UOMs will be documented here: https://icinga.com/docs/icinga2/latest/doc/05-service-monitoring/#unit-of-measurement-uom

I’ve ported the old plugin guideline HTML into a more readable documentation, allowing us to enhance this even further.

2 Likes
4

I had written up a big wall of text with SQL outputs, etc but I just discovered this in the logs:

[2019-09-25 17:07:25 +1000] warning/OpenTsdbWriter: **Ignoring invalid perfdata value** : output_voltage=239VAC;;;;

[2019-09-25 17:07:35 +1000] warning/OpenTsdbWriter: **Ignoring invalid perfdata value** : output_voltage=230VAC;;;;

I’ll have to investigate why it suddenly stopped working - it mustn’t have been putting in that UoM earlier… very strange!

5

Ah - I worked out why it was working.

We modified the script to remove the UoM at the end of the perfdata - however I never committed this to our git repo, and we recently implemented GitFS in our Salt master to sync our check scripts from git… so it must’ve overwritten the modifications I made…

Dumb mistake!

EDIT:

Aaaaand fixed:

                  @@ -264,7 +264,7 @@
                       the_helper.add_metric(
                           label=the_helper.options.type,
                           value=the_snmp_value,
                  -        uom='degree')
                  +        uom='')
1 Like