Performance data is discarded from Graphite after 6 hours

[stdevel (Christian Stankowic)]

Hello community,
I’m currently stumbling upon issues with visualizing performance data from Icinga2 with Graphite + Grafana.

Problem:
Performance data older than 6 hours are deleted from Graphite. Leveraging performance data up to 6 hours of age works perfectly.
I’m using thes storage-schemas.conf from the official Icinga2 vagrant boxes.

Setup / Envrionment:
Icinga2 is running on a virtualized Debian Jessie machine.
Beside Icinga2, also Docker CE is installed. Graphite and Grafana are running in containers.

storage-schemas.conf:

[carbon]
pattern = ^carbon\.
retentions = 10s:6h,1m:90d

[default_1min_for_1day]
pattern = .*
retentions = 10s:6h,1m:6d,10m:1800d

[icinga2_internals]
pattern = ^icinga2\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 5m:7d

[icinga2_default]
pattern = ^icinga2\.
retentions = 5m:10d,30m:90d,360m:4y

docker-compose.yml for Graphite / Grafana:

version: '2'

services:
  graphite:
    image: graphiteapp/graphite-statsd
    restart: always
    ports:
      - "80:80"
      - "2003:2003"
      - "2004:2004"
      - "8125:8125/udp"
      - "8126:8126"
    volumes:
      - storage:/opt/graphite/storage
      - ./storage-schemas.conf:/opt/graphite/conf/storage-schemas.conf:ro

  grafana:
    image: grafana/grafana
    restart: always
    ports:
      - "3000:3000"

volumes:
  storage:

Versions:
Icinga 2 - 2.10.4-1.jessie
Docker Community Edition - 18.06.3~ce~3-0~debian
Graphite container image - 1d7b81067e37 (March 2019)
Grafana container image - d9bdb6044027 (April 2019)

Do you have any hints how to process performance data that is older than 6 hours?
Thanks a lot in advance!

Best wishes,
Christian.

[log1c]

Quoting from another thread

@blakehartshorn

In storage-schemas.conf in your carbon config folder, first match wins. So in this case, your default_1min_for_1day entry is hitting before your Icinga entries. You’ll want to move those up.

Something that might catch you off guard if you’re new to graphite is that your whisper database will not update the retention automatically as that’s set in the metadata of the whisper files. You can see this by using whisper-info.py on any given value.wsp. If it’s a new setup, I’d just go ahead and start over. Otherwise, the whisper-resize.py script is available.

Not sure if this is the entire reason why the perf data are only kept for 6 hours, but it’s a start.

[blakehartshorn]

Yeah it looks like a few things here.

Your default is definitely hitting before Icinga is, but the 6h entry I see is regex’d for carbon, so that wouldn’t apply to Icinga’s perfdata.

Did you, by chance, copy/paste some entries, modify them, reload carbon, and haven’t resized the whisper files? Either way, definitely move that Icinga entry up, I’d say go ahead and nuke your icinga2 folder in the whisper storage since it’ll be gone in 6 hours anyway, start carbon back up and try again.

Use whisper-dump.py and whisper-info.py to make sure your whisper files are generating right.

[dnsmichi (Michael Friedrich)]

Also, don’t use the ones from the Graphite box, that’s just a demo. Refer to what’s written in the documentation.

Cheers,
Michael