Icingaweb 2 --> Grafana not working after update --> cannot fetch graph with curl

stefan · January 29, 2025, 12:37pm

Hello Icinga Community,

we want to ask you for help
Our company is using icinga2, icingadb,icingaweb2 up to date community edition (ubuntu 22.04 LTS repo) with grafana module.
After updating we are not able to see our graphs anymore. We get timeout error “cannot fetch graph with curl”. Increasing timeout does not help. If we copy the link from the graph its showing the graph in Grafana Server UI.

Einvironment:

Icinga Web 2 Version 2.12.2
PHP-Version 8.1.2-1ubuntu2.20
Git Commit Datum 2024-11-13

Geladene Bibliotheken

|icinga/icinga-php-thirdparty|0.12.1|
|icinga/icinga-php-library|0.14.1|

Geladene Module

businessprocess||2.5.0
company||1.0.0
|customdashboards||1.0.2
|director||1.10.2
|fileshipper||1.1.0
|grafana||2.0.3
|icingadb||1.1.1
|incubator||0.19.0
|ipl||0.0.0
|pdfexport||0.11.0
|reactbundle||0.7.0
|reporting||1.0.2

Web browser used: Opera, Chrome, Firefox, Edge
Icinga 2 version: icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.4-1)
PHP version used: PHP 8.1.2-1ubuntu2.20 (cli) (built: Dec 3 2024 20:14:35) (NTS)
Server operating system and version: Ubuntu 22.04.5 LTS (jammy) PRO

bberg · January 29, 2025, 3:04pm

In one of the last grafana releases they had a small breaking change which forces the grafana-module users to do some config changes.
Netways forked the grafana module and updated it accordingly, including necessary changes on the grafana-dashboards: GitHub - NETWAYS/icingaweb2-module-grafana: Grafana module for Icinga Web 2 (supports InfluxDB & Graphite)

You also might want to check /var/log/grafana/grafana.log for any error messages in there.

stefan · January 30, 2025, 8:21am

Thanks for your help.
We replaced the plugin. But nothing changed. We still get timeout. If we click on the link the correct graph is shown.

TLS is set to accept insecure, so we can ignore those warnings.
Grafana Logfile shows following error:

logger=rendering renderer=plugin t=2025-01-30T09:16:46.424664414+01:00 level=info msg=Rendering path=“d-solo/cdgbfilzwbitcd/basic-metrics?var-hostname=SERVERNAME&var-service=Linux-Festplatte&var-command=disk&panelId=2&orgId=1&width=640&height=500&theme=light&from=now-15m&to=now” userID=4
t=2025-01-30T09:16:46.830283783+01:00 level=info msg=“http: TLS handshake error from [::1]:36062: remote error: tls: unknown certificate”
t=2025-01-30T09:16:46.891199627+01:00 level=info msg=“http: TLS handshake error from [::1]:36088: remote error: tls: unknown certificate”
t=2025-01-30T09:16:46.972357829+01:00 level=info msg=“http: TLS handshake error from [::1]:36098: remote error: tls: unknown certificate”
t=2025-01-30T09:16:46.972732268+01:00 level=info msg=“http: TLS handshake error from [::1]:36138: remote error: tls: unknown certificate”
t=2025-01-30T09:16:46.977880761+01:00 level=info msg=“http: TLS handshake error from [::1]:36116: remote error: tls: unknown certificate”
t=2025-01-30T09:16:46.994630641+01:00 level=info msg=“http: TLS handshake error from [::1]:36108: remote error: tls: unknown certificate”
t=2025-01-30T09:16:46.995369719+01:00 level=info msg=“http: TLS handshake error from [::1]:36124: remote error: tls: unknown certificate”
t=2025-01-30T09:16:47.01712769+01:00 level=info msg=“http: TLS handshake error from [::1]:36154: remote error: tls: unknown certificate”
t=2025-01-30T09:16:48.157815807+01:00 level=info msg=“http: TLS handshake error from [::1]:36192: remote error: tls: unknown certificate”
logger=context userId=4 orgId=1 uname=sa-1-icingaweb t=2025-01-30T09:16:48.196203479+01:00 level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=[::1] time_ms=1 duration=1.645614ms size=0 referer= handler=/api/live/ws status_source=server
logger=plugin.grafana-image-renderer t=2025-01-30T09:16:49.078640855+01:00 level=error msg=“Browser request failed” failure=net::ERR_ABORTED url=“https://localhost:3000/api/ds/query?ds_type=influxdb&requestId=SQR100” method=POST
logger=rendering t=2025-01-30T09:16:51.386335466+01:00 level=error msg=“Failed to delete render key” error=“context canceled”
logger=context userId=4 orgId=1 uname=sa-1-icingaweb t=2025-01-30T09:16:51.386415512+01:00 level=error msg=“Rendering failed.” error=“rpc error: code = Canceled desc = context canceled”
logger=context userId=4 orgId=1 uname=sa-1-icingaweb t=2025-01-30T09:16:51.387271273+01:00 level=error msg=“Request error” error=“Context.HTML - Error rendering template: error. You may need to build frontend assets \n template: error:16:42: executing "error" at <.Assets.Dark>: can’t evaluate field Assets in type struct { Title string; AppTitle string; AppSubUrl string; ThemeType string; ErrorMsg error }” stack=“github.com/grafana/grafana/pkg/web/context.go:112 (0x7aae6c4)\ngithub.com/grafana/grafana/pkg/services/contexthandler/model/model.go:59 (0x7ab9f68)\ngithub.com/grafana/grafana/pkg/api/render.go:106 (0xa73d92a)\ngithub.com/grafana/grafana/pkg/api/response/web_hack.go:40 (0x7ac126f)\nnet/http/server.go:2220 (0x7aaf818)\ngithub.com/grafana/grafana/pkg/web/macaron.go:131 (0x7aaf7d5)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/validate_action_url.go:51 (0x812a23a)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/services/contexthandler/contexthandler.go:147 (0xa7494ea)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/render.go:51 (0x7ab0349)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/macaron.go:137 (0x7aaf8b4)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/csrf/csrf.go:66 (0x9b8d0af)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/recovery.go:180 (0xa71e98b)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/loggermw/logger.go:72 (0x9b8e03e)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/request_metrics.go:82 (0x8128c98)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/request_tracing.go:102 (0xa71f8bc)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/middleware/requestmeta/request_metadata.go:66 (0xa747b01)\nnet/http/server.go:2220 (0x6608308)\ngithub.com/grafana/grafana/pkg/web/context.go:52 (0x7aae05a)\ngithub.com/grafana/grafana/pkg/web/router.go:155 (0x7ab1b8d)\ngithub.com/grafana/grafana/pkg/web/router.go:221 (0x7ab2694)\ngithub.com/grafana/grafana/pkg/web/macaron.go:174 (0x7aafcfc)\nnet/http/server.go:3210 (0x662886d)\nnet/http/server.go:2092 (0x6606caf)\nruntime/asm_amd64.s:1700 (0x6286280)\n”
logger=context userId=4 orgId=1 uname=sa-1-icingaweb t=2025-01-30T09:16:51.387338503+01:00 level=error msg=“Request Completed” method=GET path=/render/d-solo/cdgbfilzwbitcd/basic-metrics status=500 remote_addr=IPADDRESS time_ms=4981 duration=4.981314514s size=343 referer= handler=/render/* status_source=server

bberg · January 30, 2025, 8:35am

Okay, what Version of Grafana are you running?

stefan · February 3, 2025, 12:45pm

Sorry for the delay, i was ill…
we are using Version 11.5.0 (commit: f7a938db9ad71c1558e93d8e29e69f42c8a5f50b, branch: HEAD)

lorenz · February 4, 2025, 12:26pm

Does it work, if you use iframe mode instead Indirect Proxy? Might be related to that horrible imagerenderer plugin.

stefan · February 5, 2025, 7:31am

IFrame Mode is working fine, but all users have to login to grafana and i cant use service account.

Workaround: enabled anonymous access in grafana server with read role. Its working until next security audit

picoroma · February 8, 2025, 7:01pm

I have also ISSUE with renfering GRAFANA Graph into ICINGA.
My problem is described here:

Still have not found a solution