FYI: Grafana upgrade (v6 to v7) breaks grafana integration with icingaweb2

Today’s update on my Icinga2 server resulted in Grafana being updated from 6.7.0-beta1 to 7.0.0-beta1. Unfortunately, this broke the Grafana integration to Icinga2 and I had to install the following libraries to get it going again:

sudo apt install libx11-xcb1 libxcomposite1 libxcursor1 libxdamage1 libxi6 libxtst6 libnss3 libxss1 libxrandr2 libasound2 libatk1.0-0 libatk-bridge2.0-0 libpangocairo-1.0-0 libgtk-3-0

That last one - libgtk-3-0 (I was doing the upgrades one by one as I checked the logs on the server) - brought with it numerous desktop related packages. Not sure who Grafana is targeting the latest updates for but installing desktop packages seems rather odd.

Cheers,
ak.

UPDATE. I’ve updated the list as I missed a few more:

1 Like

Congrats,
you did not read the changelog of Grafana. If you did you would have noticed that they finaly droped PhantomJS as renderer and now only use the renderer plugin that need chrome headless (or something similar).

And further if you read the troubleshoot docuof the renderer plugin you see exactly which package for which distri is needed.

Beside that if you use beta versions without knowing what you do, you should rethink your workflow.

Regards,
Carsten

I should feel blessed to be congratulated - but then I realize you are mocking me - how sad.

Anyway, all’s well that ends well.

PS: I see there’s a cultural difference. Maybe you were not making a mockery of me, but I’ve felt that way here just like the last time I posted something in the Github issues page. Or maybe I just need to grow a thicker skin.

PS2: I did search for the error that appeared in icingaweb2 but it did not take me to that troubleshooting document page you pasted above. Maybe the error message needs to be captured and linked to that document for those with a similar predicament / work flow. But, it is now.

Iam not mocking you, but be careful what and how you write such things. Give the users the correct information and not only the half truth. Other user may even less expirenced then you when they read your post and i will get a lot of issues for the grafana module (yes i wrote it) becuase they think there is something wrong with the module.

Read the changelog before you install new major release of whatever software you use. On top you read this

Maybe I’m just a lazy sysadmin, but I’m not sure I’d be inclined to know how each and every package works on the servers I manage. I trust that the package maintainers will know when a breaking change (one that breaks their package) is on its way, that they will maybe update their packages prior, so that when fools such as myself come along and deploy without reading each and every changelog, they’d at least get a message saying why the breakage is there and what to do to mitigate it.

For instance, even if I had read the changelog and seen that PhantomJS was being removed, there’s no way I’d have guessed that it would have an impact on IcingaWeb2.

Anyway, I may change my work flow but don’t trust that everyone of us will.

Hi Carsten,

I really appreciate your work with the Icingaweb2Grafana-Plugin and it is a huge upgrade to icingaweb2.
Also after the Grafana update, your module works properly, if you install the grafana-image-renderer correctly. Only if you enabled the link to Grafana, you will be redirected to the whole dashboard and not to the panel. Can you confirm this behaviour?

So far so good, but the image-rendering with the new Grafana plugin is horrible slow and there are also issues: https://github.com/grafana/grafana-image-renderer/issues/124

Therefore also the images on icingaweb2 only appear after 5-8sec when you open a host/service in detail using proxy/indirect proxy mode. I know thats a problem of the Grafana plugin, but I can’t upgrade my Grafana server, because it’s not reasonable and I would annoy our icinga users if they had to wait longer than 5sec to get an image.

So I tested the iFrame mode of your plugin, which would be a great alternative, if there was a link to the Grafana panel. Also as you mentioned in your docs, the icingaweb2 page refresh is very annoying because it distort the screen every 10sec.

Do you have any ideas or best practice how we can use your awesome icingaweb2-plugin to get satisfying results again?

Regards,
Philipp

Hello,

you can speed up a lot of graphs if you use constants and not variables with queries in your dashboards used by the grafana module. Even under heavy load my grafana instance (with renderer plugin) takes only 1 sec to show the graphs.

Regards,
Carsten

Thanks Carsten, just tried your suggestion, but the rendering time is still the same. Moreover I don’t want to modify all my dashboards.
What do you think about the option to add a link to the iframe-mode?

Hi,

I needed to add GF_RENDERER_PLUGIN_IGNORE_HTTPS_ERRORS=true to /etc/defaults/grafana-server on Ubuntu, otherwise https://localhost throws an Certifikate error if a local image renderer is used.

As a “feature” the new renderer is slower then the old one…

1 Like

Hi
I had to use chrome standalone instead of the chrome packaged in plugin-linux-x64-glibc and add GF_RENDERER_PLUGIN_IGNORE_HTTPS_ERRORS=true to /etc/defaults/grafana-server as described above. I use centos.
Now, the graphs are displayed correctly in icinga web 2 but I had the problem, that the link from the graph to grafana didn’t work. It made a redirect to localhost:3000 and ended in a error message with ’ PR_END_OF_FILE_ERROR’, probalbly a security/certificate problem.
I patched /usr/share/icingaweb2/modules/grafana/library/Grafana/ProvidedHook/Grapher.php

# diff Grapher.php Grapher.php.orig
571c571
<                     $html .= '<a href="%s://%s/d/%s/%s?var-hostname=%s&var-service=%s&var-command=%s%s&from=%s&to=%s&orgId=%s&viewPanel=%s" target="_blank">%s</a>';
---
>                     $html .= '<a href="%s://%s/d/%s/%s?var-hostname=%s&var-service=%s&var-command=%s%s&from=%s&to=%s&orgId=%s&panelId=%s&fullscreen" target="_blank">%s</a>';
590c590
<                     $html .= '<a href="%s://%s/dashboard/%s/%s?var-hostname=%s&var-service=%s&var-command=%s%s&from=%s&to=%s&orgId=%s&viewPanel=%s" target="_blank">%s</a>';
---
>                     $html .= '<a href="%s://%s/dashboard/%s/%s?var-hostname=%s&var-service=%s&var-command=%s%s&from=%s&to=%s&orgId=%s&panelId=%s&fullscreen" target="_blank">%s</a>';

Without &fullscreen and with viewPanel= instead of panelId= it seems to work now.