Performance data for some services is not recorded in graphite

Hello,

(please see below for environment information)

we are having the problem that for some services the performance data is not recorded in graphite (and thus no graph shown in icingaweb2, just two pie charts), but for other services it is. We cannot discern any reason for this.

Output from a service failing to record data:

[
    "mem_usage=73%;75;90;0",
    "cpu_load=44%;90;98;0"
]

Output from a service that is working fine:

[
    "battery=100%;;;0;100",
    "load=42%;;;0;100"
]

What else may be wrong? What are we missing?

Thanks for any help or pointers,
Jörg

root@monitor:/etc/icinga2# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.3-1)

Copyright (c) 2012-2024 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 24.04.1 LTS (Noble Numbat)
  Kernel: Linux
  Kernel version: 6.8.0-48-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 13.2.0
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 3.0.13 30 Jan 2024

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
root@monitor:/etc/icinga2# icinga feature list
Command 'icinga' not found, did you mean:
  command 'icinga2' from deb icinga2-bin (2.13.8-1)
Try: apt install <deb name>
root@monitor:/etc/icinga2# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf influxdb influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker graphite icingadb mainlog notification
root@monitor:/etc/icinga2# icinga2 daemon -C
[2024-11-20 14:08:23 +0100] information/cli: Icinga application loader (version: r2.14.3-1)
[2024-11-20 14:08:23 +0100] information/cli: Loading configuration file(s).
[2024-11-20 14:08:23 +0100] information/ConfigItem: Committing config item(s).
[2024-11-20 14:08:23 +0100] information/ApiListener: My API identity: monitor
[2024-11-20 14:08:23 +0100] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/conf.d/services.conf: 34:1-34:21) for type 'Service' does not match anywhere!
[2024-11-20 14:08:23 +0100] warning/ApplyRule: Apply rule 'check-ad-replication' (in /etc/icinga2/conf.d/services.conf: 144:1-144:36) for type 'Service' does not match anywhere!
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 2 NotificationCommands.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 114 Notifications.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 2 HostGroups.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 32 Hosts.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 Downtime.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 3 Comments.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 IcingaDB.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 3 Zones.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 User.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 Endpoint.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 262 CheckCommands.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 GraphiteWriter.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 3 ServiceGroups.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2024-11-20 14:08:23 +0100] information/ConfigItem: Instantiated 91 Services.
[2024-11-20 14:08:23 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2024-11-20 14:08:23 +0100] information/cli: Finished validating the configuration file(s).
root@monitor:~# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf influxdb influxdb2 journald livestatus opentsdb perfdata syslog
Enabled features: api checker graphite icingadb mainlog notification

Sorry for the noise. Turns out it was unrelated to icinga2. The carbon-cache daemon stopped running because of a system misconfiguration, so the data for the new services was never recorded.

1 Like