Howto send checkcommand service.output to a graphite server

Icinga 2
,
1

I have enabled the graphite integration on the icinga2 distributed monitor and the central and distributed icinga2 servers. I used the documention at Features - Icinga 2. once enabled and configured to send to the graphite host and one of the relay LINE_RECEIVER_PORT. However I don’t see any icinga metrics produced. Also really what I want is to take the output from a checkcommand that basically when seccessful sends graphite plain text format messages on each line with the following format

metricnamelevel1.metricnamelevel2.metrcname mtricvalue epochtimestamp
metricnamelevel1.metricnamelevel2.metrcname mtricvalue epochtimestamp
…
metricnamelevel1.metricnamelevel2.metrcname mtricvalue epochtimestamp

Once the output comes into icinga loop through the output and send the data to the graphite server that was configured as the graphiteWriter

2

Hi @jimm1139 ,
This almost certainly won’t work the way you describe it. Icinga2 will write only the performance data of a check to graphite, not the human readable part ( see https://www.ietf.org/archive/id/draft-kaestle-monitoring-plugins-interface-03.html for how to produce performance data).
Also, you only need to enable the graphite writer on the main Icinga server, since this is the one which receives ALL the check results.
In /etc/icinga2/features-enabled/graphite-writer.conf there should be the configuration for the targeted Graphite writer, is that correct?

Is there something in the icinga2 log about failing to write the data?

3

@lorenz What I wanted was to have the output of an nrpe or some other check command which contain plaintext graphite formatted data sent to a graphite server. I have created a GraphiteWriter.py that runs from within the nrpe client plugin directory to accomplish this. this now sends the output from the check command that generates application or system related metrics to the graphite host based on the region and also as part of the output that is sent back to the icinga2 satellite that sent the nrpe remote check to the icinga / nrpe agent machine. This is how I got this to work

4

Well, probably not the setup I would have recommended, but if it works for you :slight_smile:

5

@lerenz. I am always open to finding better approaches for this if you would like to share your recommendations for this that would be great

6

If you take a look at the link I posted, there is a “standardized” way to hand performance data to Icinga ( The Monitoring Plugins Interface ).
Normally I would use this to transport performance data to the master and write it to a graphite using the Graphite Writer feature.

7

@lorenz Ah I read that further and see that performance metrics need to be added to the output <HUMAN READABLE OUTPUT (ADDITIONAL INFO)> | METRIC_LABEL1=METRIC_VALUE[;warn[;crit[;min[;max]]]]

Then read the following specific for nagios which has examples for sending multiple metrics in the perfdata at Performance Data

So the example I have would be something like the folloiwng format
metrics for process_jmx_metrics retreival is OK |
<graphite_metric_path_1>=metric_value[<metric_unit_of_measure],
<graphite_metric_path_2>=metric_value[<metric_unit_of_measure],
…,
<graphite_metric_path_X>=metric_value[<metric_unit_of_measure]

Also see in the icinga-core code repository there is the following config settings related to the perfdata

process_performance_data=1
host_perfdata_command=process-host-perfdata
service_perfdata_command=process-service-perfdata
host_perfdata_file=/tmp/host-perfdata
service_perfdata_file=/tmp/service-perfdata
host_perfdata_file_template=[HOSTPERFDATA]\t$TIMET$\t$HOSTNAME$\t$HOSTEXECUTIONTIME$\t$HOSTOUTPUT$\t$HOSTPERFDATA$
service_perfdata_file_template=[SERVICEPERFDATA]\t$TIMET$\t$HOSTNAME$\t$SERVICEDESC$\t$SERVICEEXECUTIONTIME$\t$SERVICELATENCY$\t$SERVICEOUTPUT$\t$SERVICEPERFDATA$
host_perfdata_file_mode=a
service_perfdata_file_mode=a
host_perfdata_file_processing_interval=0
service_perfdata_file_processing_interval=0
host_perfdata_file_processing_command=process-host-perfdata-file
service_perfdata_file_processing_command=process-service-perfdata-file
host_perfdata_process_empty_results=1
service_perfdata_process_empty_results=1

I assume it would be possible to send more than a single performance metric from the output of a single command , Are my assumptions correct?

Thanks for pointing me in the correct direction this should definitely be implemented this way