# Performance data is discarded from Graphite after 6 hours

**URL:** <https://community.icinga.com/t/performance-data-is-discarded-from-graphite-after-6-hours/1148>\
**Category:** Visualization, Metrics & Reporting\
**Tags:** grafana, graphite\
**Created:** [May 8, 2019, 12:15pm UTC](https://community.icinga.com/t/performance-data-is-discarded-from-graphite-after-6-hours/1148 "2019-05-08T12:15:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![stdevel](https://community.icinga.com/user_avatar/community.icinga.com/stdevel/32/1343_2.png) [@stdevel](https://community.icinga.com/u/stdevel)\
**Post date:** [May 8, 2019, 12:16pm UTC](https://community.icinga.com/t/performance-data-is-discarded-from-graphite-after-6-hours/1148/1 "2019-05-08T12:16:00Z")

</div>

Hello community,  
I’m currently stumbling upon issues with visualizing performance data from Icinga2 with Graphite + Grafana.

**Problem:**  
Performance data older than 6 hours are deleted from Graphite. Leveraging performance data up to 6 hours of age works perfectly.  
I’m using thes `storage-schemas.conf` from the official Icinga2 vagrant boxes.

**Setup / Envrionment:**  
Icinga2 is running on a virtualized Debian Jessie machine.  
Beside Icinga2, also Docker CE is installed. Graphite and Grafana are running in containers.

`storage-schemas.conf`:

```auto
[carbon]
pattern = ^carbon\.
retentions = 10s:6h,1m:90d

[default_1min_for_1day]
pattern = .*
retentions = 10s:6h,1m:6d,10m:1800d

[icinga2_internals]
pattern = ^icinga2\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 5m:7d

[icinga2_default]
pattern = ^icinga2\.
retentions = 5m:10d,30m:90d,360m:4y

```

`docker-compose.yml` for Graphite / Grafana:

```auto
version: '2'

services:
  graphite:
    image: graphiteapp/graphite-statsd
    restart: always
    ports:
      - "80:80"
      - "2003:2003"
      - "2004:2004"
      - "8125:8125/udp"
      - "8126:8126"
    volumes:
      - storage:/opt/graphite/storage
      - ./storage-schemas.conf:/opt/graphite/conf/storage-schemas.conf:ro

  grafana:
    image: grafana/grafana
    restart: always
    ports:
      - "3000:3000"

volumes:
  storage:

```

**Versions:**  
Icinga 2 - 2.10.4-1.jessie  
Docker Community Edition - 18.06.3~ce~3-0~debian  
Graphite container image - 1d7b81067e37 (_March 2019_)  
Grafana container image - d9bdb6044027 (_April 2019_)

Do you have any hints how to process performance data that is older than 6 hours?  
Thanks a lot in advance!

Best wishes,  
Christian.

---

<div class="post-metadata">

**Author:** ![log1c](https://community.icinga.com/user_avatar/community.icinga.com/log1c/32/2071_2.png) [@log1c](https://community.icinga.com/u/log1c)\
**Post date:** [May 9, 2019, 2:54pm UTC](https://community.icinga.com/t/performance-data-is-discarded-from-graphite-after-6-hours/1148/2 "2019-05-09T14:54:45Z")

</div>

Quoting from another thread

> [@Graphite won't keep history](https://community.icinga.com/t/graphite-wont-keep-history/813/8):
>
> In storage-schemas.conf in your carbon config folder, first match wins. So in this case, your default\_1min\_for\_1day entry is hitting before your Icinga entries. You’ll want to move those up.
> 
> Something that might catch you off guard if you’re new to graphite is that your whisper database will not update the retention automatically as that’s set in the metadata of the whisper files. You can see this by using whisper-info.py on any given value.wsp. If it’s a new setup, I’d just go ahead and start over. Otherwise, the whisper-resize.py script is available.

Not sure if this is the entire reason why the perf data are only kept for 6 hours, but it’s a start.

---

<div class="post-metadata">

**Author:** ![blakehartshorn](https://community.icinga.com/user_avatar/community.icinga.com/blakehartshorn/32/599_2.png) [@blakehartshorn](https://community.icinga.com/u/blakehartshorn)\
**Post date:** [May 9, 2019, 2:59pm UTC](https://community.icinga.com/t/performance-data-is-discarded-from-graphite-after-6-hours/1148/3 "2019-05-09T14:59:56Z")

</div>

Yeah it looks like a few things here.

Your default is definitely hitting before Icinga is, but the 6h entry I see is regex’d for carbon, so that wouldn’t apply to Icinga’s perfdata.

Did you, by chance, copy/paste some entries, modify them, reload carbon, and haven’t resized the whisper files? Either way, definitely move that Icinga entry up, I’d say go ahead and nuke your icinga2 folder in the whisper storage since it’ll be gone in 6 hours anyway, start carbon back up and try again.

Use whisper-dump.py and whisper-info.py to make sure your whisper files are generating right.

---

<div class="post-metadata">

**Author:** ![dnsmichi](https://community.icinga.com/user_avatar/community.icinga.com/dnsmichi/32/7721_2.png) [@dnsmichi](https://community.icinga.com/u/dnsmichi)\
**Post date:** [May 9, 2019, 3:24pm UTC](https://community.icinga.com/t/performance-data-is-discarded-from-graphite-after-6-hours/1148/4 "2019-05-09T15:24:41Z")

</div>

Also, don’t use the ones from the Graphite box, that’s just a demo. Refer to what’s written in the [documentation](https://icinga.com/docs/icinga2/latest/doc/14-features/#graphite-carbon-cache-writer).

Cheers,  
Michael