Continuous increase of storage since Upgrade 2.13.7 --> 2.14.0

Icinga 2
1
  • Version used (icinga2 --version)
    r2.14.0-1
  • Operating System and version
    rhel 8.8
  • Enabled features (icinga2 feature list)
    Enabled features: api checker ido-mysql influxdb2 mainlog notification
  • Icinga Web 2 version and modules (System - About)
    image
    image422×545 14.7 KB
  • Config validation (icinga2 daemon -C)
[2023-10-30 14:09:28 +0100] information/cli: Icinga application loader (version: r2.14.0-1)
[2023-10-30 14:09:28 +0100] information/cli: Loading configuration file(s).
[2023-10-30 14:09:28 +0100] information/ConfigItem: Committing config item(s).
[2023-10-30 14:09:28 +0100] information/ApiListener: My API identity: vie02s810.vie01.local
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 309 Comments.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 57 Users.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 19 UserGroups.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 2 TimePeriods.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 14 ServiceGroups.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 335 Zones.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 9461 Services.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 8 NotificationCommands.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 21 Notifications.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 582 Hosts.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 89 HostGroups.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 333 Endpoints.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 66 Downtimes.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 3 ApiUsers.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 328 CheckCommands.
[2023-10-30 14:09:28 +0100] information/ConfigItem: Instantiated 1 Influxdb2Writer.
[2023-10-30 14:09:28 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-10-30 14:09:28 +0100] information/cli: Finished validating the configuration file(s).

Hi I noticed a distinct continuous increase of the influxdb since my upgrade from icinga 2.13.7 to 2.14… Here a snippet of the relevant updates made:
icinga2-2.14.0-1.el8.x86_64
icinga2-2.13.7-1.el8.x86_64
icinga2-bin-2.14.0-1.el8.x86_64
icinga2-bin-2.13.7-1.el8.x86_64
icinga2-common-2.14.0-1.el8.x86_64
icinga2-common-2.13.7-1.el8.x86_64
icinga2-ido-mysql-2.14.0-1.el8.x86_64
icinga2-ido-mysql-2.13.7-1.el8.x86_64

According to Grafana the continuous increase started exactly when the upper packages were updated.

image
image2516×1029 199 KB

The influxdb version and the influx settings like retention policy or the like stayed the same. Nothing has been changed on the system regarding influxdb.
Update: I rechecked the retention policy of the icinga-bucket which is mainly a 1:1 mapping of metrics for 365 days and all data is removed after 365 days as expected/configured.

Can someone guide me to find the reason why the influxdb is significantly increasing? Maybe before I run out of storage :slight_smile:

2

I’d check the logs before the upgrade to see if there were some perfdata discarded due to incorrect format. And/or this bug fix which now produces more or larger time series.

3

Hi, unfortunately I could not find any logs before this event on 20.09. However I had this problem once and this was fixed as far as I can remember. Apart of that I found a debug.log from August and there were no logs regarding incorrect format (which however is not really representative since these are logs from 5minutes).

Regarding the second bug I doubt that this explains a increase of 25% of storage-space.

So far I compared some time-series data before and after this event and they all seemed ident, so if something in the time series has been changed then all time-series data in the bucket must have been changed at once but the increase on the 20.09 would be massive in such a case if you put the continuous increase of storage (from last month) in relation.

Here the influxdb2.conf

/**
 * The Influxdb2Writer type writes check result metrics and
 * performance data to an InfluxDB v2 HTTP API
 */

object Influxdb2Writer "influxdb2" {
  host = "127.0.0.1"
  port = 8086
  organization = "ORG"
  bucket = "icinga/default"
  auth_token = "ddddddddddd
  host_template = {
    measurement = "$host.check_command$"
    tags = {
      hostname = "$host.name$"
      alias = "$host.display_name$"
    }
  }
  service_template = {
    measurement = "$service.check_command$"
    tags = {
      hostname = "$host.name$"
      service = "$service.name$"
      alias = "$host.display_name$"
    }
  }
}

Until now the data is still increasing at the same pace. Thank you for your hints!

4

Obviously same Problem which has been fixed in 2.14.2 → see High InfluxDB disk usage after upgrade to 2.14.0