Grafana Panels Not Updating with Fresh Icinga2 Data via InfluxDB – Stale Metrics?

Icinga 2
,
1

Hello

I have integrated Icinga2 with Grafana using InfluxDB as the backend & most of the setup works well. :slightly_smiling_face:However; I have noticed that some panels on my Grafana dashboards are showing stale data; even though the Icinga2 metrics are fresh in the InfluxDB when I query them directly. :upside_down_face:

The panels update only after a long delay / sometimes not until I manually refresh the dashboard. This becomes problematic for real-time monitoring & alert visualization. :thinking:

I suspect it might be related to how Grafana caches or queries data from InfluxDB. I have already checked time ranges, panel settings & query intervals; but the issue persists intermittently. :innocent: Could this be a templating or variable refresh issue in Grafana? Or does anyone know if Icinga2-InfluxDB setups require special tweaks to keep Grafana panels in sync? :thinking:

I have checked InfluxDB data source | Grafana documentation documentation guide for reference. Since I am still understanding what is Grafana and how it handles data visualization timing across data sources; I would appreciate any suggestions also pointers to best practices. :innocent:

A working config example for a reliable Icinga2 → InfluxDB → Grafana pipeline would be extremely helpful.

Thank you !! :slightly_smiling_face:

2

Welcome to the Icinga Community and thanks for posting.

Since the data is present in the InfluxDB, this does not seem to be an Icinga 2 issue - fortunately.

However, which data seems to arrive late in the Grafana dashboard? Could you please share the relevant queries with us and check how long they take if you execute them manually? Maybe some queries are getting a bit too expensive.

Are you using some Grafana dashboard you found on the web or did you build on by yourself?

In general, I know of no special cases for plotting Icinga 2 performance data in Grafana via InfluxDB. When I used it in the past, it “just worked”.

Maybe share some details about your setup?

3

I see no such issue with my setup. It was generated via @linuxfabrik’s LFOps.