Graphing issue with graphite

pombaer · August 30, 2024, 5:04am

Hello, i have an graphing issue with graphite in icinga2. I am using an external check plugin which reports metrics in performance date containing brackets, this seams to break something. I guess the problem is the icinga2 graphite module, but there are no configuration settings to fix the issue.
I already described it on github

github.com/Icinga/icingaweb2-module-graphite

Metric rewrite issue in graphite module:

opened 02:09PM - 29 Aug 24 UTC

pombaer

Hello, i have an issue with performance Data that contain round brakets (). I ha…ve a third party check command which returns brackets in the performance data, i made a simple dummy check for reproduction: ``` apply Service "graphite_1" { import "generic-service" display_name = "graphite_1" check_command = "dummy" vars.dummy_state = 0 vars.dummy_text = "OK fix perfdata 1 | 'value (disk)'=83338133504B;19198167449.600002;9599083724.8000011;0;95990837248" assign where host.name == "localhost" } ``` When applying this Check to "localhost" metric is written as ` "icinga2.localhost.services.graphite_dummy_1.dummy.perfdata.value_(disk).value" ` on disk by whisper, and is also displayed in graphite dashboard in this format. The issue is that icinga2 graphite module is looking for "value_disk_.value" for the metric, the brackets are replaced to single "_" character, i found the request string in Apache2 request string, so i expect replacement is done in icinga2 module graphite. I am not sure how i can deal with the issue. In Graphite (cacher/relay/wisper) the brackets seam to be valid characters, otherwise i expect they should be replaeced before writing on disk. In Icinga2 Graphite Module there is no configuration for valid Characters. For graphite_web the characters look also valid, an valid result is returned: ``` /metrics/expand?query=icinga2.localhost.services.graphite_dummy_1.dummy.perfdata.value_(disk).value {"results": ["icinga2.localhost.services.graphite_dummy_1.dummy.perfdata.value_(disk).value"]} ``` I am using: ``` icinga2: 2.12.1 php: 8.2.23 graphite module: 1.2.4 carbon, whisper, graphite_web from "https://github.com/graphite-project" master branch. ``` May you have some hints for me how to solve the problem, thanks in advance!!

In short, the metric data is correctly written to carbon and is available in the graphite_web dashboard interface. The metric name contain the brackets ‘()’ and spaces are replaced by “" characters. So everything looks ok.
Requests from Icinga2 to grafana_web in Apache LOG do not contain the brackets, they are replaced by "” characters, multiple replaced characters result also in a single “_” character.
Excuse the cross posting, but it’s pretty important for me to fix this.
I know i could write a wrapper, but this would be the last option for me.
Thanks in advance.

pombaer · September 6, 2024, 5:10am

I solved it using carbon-c-relay. It looks like carbon-relay should replase () to _ but does not.