Fehler bei Graphite in der Historie

For sure I already thought about during writing.

So the problem of the original poster is that graphite is not showing data after two days for checks with a check interval of 5 minutes while it works fine with 90 seconds as interval.

Thankfully the problem is easily solved when you look at the posted configuration.

First we need to look which pattern is matching so which section is responsible, in this case ^icinga2. which results in section [icinga_default] and by this retention of 1m:2d,5m:10d,30m:90d,360m:4y. This means data will be stored with 1 minute accuracy for 2 days, then aggregated to 5 minutes for 10 days, 30 minutes for 90 days and 360 minutes for 4 years.

Now for the checks that work which are 90 seconds and will generate 3 metrics within 5 minutes. So while it works it is not ideal as we will only get 3 metrics instead of 5. This is 60% or a factor of 0.6. In the default aggregation configuration an aggregation by average is defined and with xFilesFactor=0.5 meaning a minimum of 50% of data is required for an aggregation to happen.

The not working ones are only filing one timeslot of 5 because they are checked every 5 minutes so it is only 20% or a factor of 0.2 which will result in no aggregation for the second interval and all later ones. And this results in having data only for the first 2 days!

What has to be changed is the configuration of the retention to match the check interval in Icinga! Furthermore to the first one matching the interval, further ones have to be a multiple of the first one.

So two changes for this specific case are needed. A new section for the 5 minute checks is created which requires a matching pattern like name of the service or check command instead of the placeholder 5MINUTECHECK. And the default section was adjusted.

...
[icinga_5m]
pattern = ^icinga2\..*\.5MINUTECHECK
retentions = 5m:10d,30m:90d,360m:4y


[icinga_default]
pattern = ^icinga2\.
retentions = 90s:2d,6m:10d,30m:90d,360m:4y
...

Furthermore you have to change the aggregation of the existing whisper files using whisper-resize / whisper-auto-resize or remove the files and let them be recreated for the change having an effect. (Thanks for pointing out @homerjay)

2 Likes