Icinga2 + Graphite showing empty graphs

Hello everyone,
I am try install module graphite (ver 1.1.0) for showing performance data. But graphs for ping service is empty.

icinga2 object list --type GraphiteWriter

Object ‘graphite’ of type ‘GraphiteWriter’:
% declared in ‘/etc/icinga2/features-enabled/graphite.conf’, lines 1:0-1:31

  • __name = “graphite”
  • enable_ha = false
  • enable_send_metadata = false
  • enable_send_thresholds = true
    % = modified in ‘/etc/icinga2/features-enabled/graphite.conf’, lines 4:3-4:31
  • host = “127.0.0.1”
    % = modified in ‘/etc/icinga2/features-enabled/graphite.conf’, lines 2:3-2:20
  • host_name_template = “icinga2.$host.name$.host.$host.check_command$”
  • name = “graphite”
  • package = “_etc”
  • port = “2003”
    % = modified in ‘/etc/icinga2/features-enabled/graphite.conf’, lines 3:3-3:13
  • service_name_template = “icinga2.$host.name$.services.$service.name$.$service.check_command$”
  • source_location
    • first_column = 0
    • first_line = 1
    • last_column = 31
    • last_line = 1
    • path = “/etc/icinga2/features-enabled/graphite.conf”
  • templates = [ “graphite” ]
    % = modified in ‘/etc/icinga2/features-enabled/graphite.conf’, lines 1:0-1:31
  • type = “GraphiteWriter”
  • zone = “”

Thanks
Jozef

Hi Jozef,

do other graphs appear?
Did you have an look at this amazing guide?

Greetings.

1 Like

Hi Homerjay,

Thanks for the link. This is my first installation.
I have now added services disk and load. The Graphite module still does not display new services neither ping.

Jozef

if use mysqld:
try:
mybe modify /etc/my.cnf

sudo sed -i ‘/mb4/s@^@#@g’ /etc/my.cnf
sudo service mysqld restart

Hi jqrung,

your command nothing modify in /etc/my.cnf

[root@lmaster ~]# cp /etc/my.cnf /etc/my.cnf.bck
[root@lmaster ~]# sudo sed -i ‘/mb4/s@^@#@g’ /etc/my.cnf
[root@lmaster ~]# diff /etc/my.cnf /etc/my.cnf.bck
[root@lmaster ~]#

Jozef

Hi Homerjay,

I am got to step by step use guide, but after define service i has problem with starts carbon-relay service. Detailed info in my comment. A distro agnostic guide to Graphite with venv (examples: RHEL8, Debian10, Ubuntu18)

Jozef

sorry i do not say clear
my mysql is Compile and install

if you yum or apt install
Not check here
dpkg -L mysql…
rpm -qL mysql…

find your configure

i question same as you , mysqld
just like this
image

and

if you mouse right click

https://icingaserver.kaka.com/icingaweb2/graphite/graph/service?host.name=icingaserver.kaka.com&service.name=users&service.check_command=users&template=users&start=-3600&end=&width=545&height=220&cachebuster=104061463062337&legend=1

image

check sql error

SQLSTATE[42000]: Syntax error or access violation: 1253 COLLATION ‘latin1_general_ci’ is not valid for CHARACTER SET 'utf8mb4

if you have this error
so modify mysql configure
sudo sed -i ‘/mb4/s@^@#@g’ /yourpath/my.cnf
sudo service mysqld restart
image

Hi again.

Can you please post:

  • all corresponding config-files (as used in the guide)
    without comments and without your secrets
    (please formatted as code - for readability)
  • your OS

Greetings.

Hi jqrun

Thank you your answer and your recommande.

I think I have another problem. My problem is that I can’t start the carbon-relay service.

Hi Homerjay,

==================================================================================================

[root@lmaster ~]# uname -a
Linux lmaster.localdomain 4.18.0-147.5.1.el8_1.x86_64 #1 SMP Wed Feb 5 02:00:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@lmaster ~]# cat /etc/rele
CentOS Linux release 8.1.1911 (Core)
Derived from Red Hat Enterprise Linux 8.1 (Source)
NAME=“CentOS Linux”
VERSION=“8 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“8”
PLATFORM_ID=“platform:el8”
PRETTY_NAME=“CentOS Linux 8 (Core)”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:8”
HOME_URL=“https://www.centos.org/
BUG_REPORT_URL=“https://bugs.centos.org/

CENTOS_MANTISBT_PROJECT=“CentOS-8”
CENTOS_MANTISBT_PROJECT_VERSION=“8”
REDHAT_SUPPORT_PRODUCT=“centos”
REDHAT_SUPPORT_PRODUCT_VERSION=“8”

CentOS Linux release 8.1.1911 (Core)
CentOS Linux release 8.1.1911 (Core)
cpe:/o:centos:centos:8
[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# cat /opt/graphite/conf/carbon.conf |grep -v ^#
[cache]

DATABASE = whisper
ENABLE_LOGROTATION = True
USER = carbon
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500

MAX_CREATES_PER_MINUTE = 50
MIN_TIMESTAMP_RESOLUTION = 1

LINE_RECEIVER_INTERFACE = 0.0.0.0
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_MAX_LENGTH = 1048576

USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0

LOG_UPDATES = False
LOG_CREATES = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False

CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False

WHISPER_FALLOCATE_CREATE = True
WHISPER_LOCK_WRITES = False

GRAPHITE_URL = http://127.0.0.1:8000/graphite

RELAY_METHOD = rules
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004
MAX_QUEUE_SIZE = 10000
MAX_DATAPOINTS_PER_MESSAGE = 500
QUEUE_LOW_WATERMARK_PCT = 0.8
TIME_TO_DEFER_SENDING = 0.0001
USE_FLOW_CONTROL = True

USE_RATIO_RESET=False
MIN_RESET_STAT_FLOW=1000
MIN_RESET_RATIO=0.9
MIN_RESET_INTERVAL=121

[aggregator]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
FORWARD_ALL = True
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5

[cache:a]
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_PORT = 7002

[cache:b]
LINE_RECEIVER_PORT = 2103
PICKLE_RECEIVER_PORT = 2104
CACHE_QUERY_PORT = 7102

[relay]
LINE_RECEIVER_INTERFACE = 127.0.0.1 # you can use 127.0.0.1 if Icinga is on the same server
LINE_RECEIVER_PORT = 2013
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 1
DYNAMIC_ROUTER = True
DESTINATIONS = 127.0.0.1:2004:a, 127.0.0.1:2104:b
DESTINATION_PROTOCOL = pickle
DESTINATION_TRANSPORT = none
USER = carbon
QUEUE_LOW_WATERMARK_PCT = 0.8
TIME_TO_DEFER_SENDING = 0.0001
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_MAX_LENGTH = 1048576

[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# cat /opt/graphite/conf/storage-schemas.conf |grep -v ^#

[icinga-hosts]
pattern = ^icinga2…*.host.
retentions = 5m:4w

[icinga-services]
pattern = ^icinga2…*.services.
retentions = 5m:12w,15m:26w,30m:1y

[carbon]
pattern = ^carbon.
retentions = 60:90d

[default]
pattern = .*
retentions = 60s:1d,5m:30d,1h:3y
[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# cat /opt/graphite/webapp/graphite/local_settings.py |grep -v ^#

SECRET_KEY = ‘imalittleteapotshortandstout’
DEFAULT_XFILES_FACTOR = 0
URL_PREFIX = ‘/graphite’

CARBONLINK_HOSTS = [“127.0.0.1:7002:a”, “127.0.0.1:7102:b”]
CARBONLINK_TIMEOUT = 1.0
CARBONLINK_RETRY_DELAY = 15 # Seconds to blacklist a failed remote server

REPLICATION_FACTOR = 1
FUNCTION_PLUGINS = []

[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# cat /etc/systemd/system/carbon-cache@.service
[Unit]
Description=carbon-cache instance %i (graphite)

[Service]
Environment=PATH=/opt/graphite/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Environment=VIRTUAL_ENV=/opt/graphite
User=carbon
Group=carbon
ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-cache-%i.pid
ExecStart=/opt/graphite/bin/carbon-cache.py --instance=%i start --pidfile=/opt/graphite/storage/carbon-cache-%i.pid
Type=forking
PIDFile=/opt/graphite/storage/carbon-cache-%i.pid
LimitNOFILE=128000

[Install]
WantedBy=multi-user.target
[root@lmaster ~]#

==================================================================================================
[root@lmaster ~]# cat /etc/systemd/system/graphite.service
[Unit]
Description = Graphite

[Service]
Environment=PATH=/opt/graphite/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Environment=PYTHONPATH=/opt/graphite/webapp:/opt/graphite
Environment=VIRTUAL_ENV=/opt/graphite
User = carbon
Group = carbon
WorkingDirectory = /opt/graphite/webapp
PIDFile = /var/run/graphite/graphite.pid
ExecStart = /opt/graphite/bin/gunicorn --bind 127.0.0.1:8000 graphite.wsgi
ExecReload = /bin/kill -s HUP $MAINPID
ExecStop = /bin/kill -s TERM $MAINPID

[Install]
WantedBy = multi-user.target
[root@lmaster ~]#

==================================================================================================
[root@lmaster ~]# cat /etc/systemd/system/carbon-relay.service
[Unit]
Description=Graphite Carbon Relay
After=network.target

[Service]
Environment=PATH=/opt/graphite/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
Environment=PYTHONPATH=/opt/graphite/webapp:/opt/graphite
Environment=VIRTUAL_ENV=/opt/graphite
Type=forking
StandardOutput=syslog
StandardError=syslog
ExecStart=/opt/graphite/bin/carbon-relay.py --config=/opt/graphite/conf/carbon.conf --pidfile=/var/run/carbon-relay.pid start
ExecReload=/bin/kill -USR1 $MAINPID
PIDFile=/var/run/carbon-relay.pid
Restart=always

[Install]
WantedBy=multi-user.target
[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# cat /etc/icinga2/features-enabled/graphite.conf
/**

  • The GraphiteWriter type writes check result metrics and
  • performance data to a graphite tcp socket.
    */

object GraphiteWriter “graphite” {
host = “127.0.0.1”
port = 2013
}
[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# find / -name value.wsp
[root@lmaster ~]#

==================================================================================================

[root@lmaster ~]# ps aux |grep carbon
carbon 805 0.0 2.0 102736 16880 ? Ss 06:01 0:03 /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn --bind 127.0.0.1:8000 graphite.wsgi
carbon 873 0.1 3.1 279108 26688 ? Sl 06:01 0:16 /opt/graphite/bin/python3 /opt/graphite/bin/carbon-cache.py --instance=b start --pidfile=/opt/graphite/storage/carbon-cache-b.pid
carbon 875 0.1 2.9 279108 24628 ? Sl 06:01 0:15 /opt/graphite/bin/python3 /opt/graphite/bin/carbon-cache.py --instance=a start --pidfile=/opt/graphite/storage/carbon-cache-a.pid
carbon 6752 0.1 5.3 491200 45388 ? Sl 09:01 0:04 /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn --bind 127.0.0.1:8000 graphite.wsgi
root 8398 0.0 0.1 12108 968 pts/1 R+ 10:01 0:00 grep --color=auto carbon
[root@lmaster ~]#

==================================================================================================

Info about wrong start service carbon-relay see here

Jozef

Hi Jozef.

I took your config-files and started a new installation process in a VM (Ubuntu, not Debian).
All services run without an error.

The only thing that had to be changed was
the /opt/graphite/webapp/graphite/local_settings.py, before the migration:

PYTHONPATH=/opt/graphite/webapp django-admin.py migrate

The quotation marks had to be changed, but I assume that this just happened by copy&paste.

Maybe anyone else has a hint for you.

Greetings,

Hi Homerjay,

sorry I don’t understand your recommande.

Do you means add line “PYTHONPATH=/opt/graphite/webapp django-admin.py migrate” in to the script /opt/graphite/webapp/graphite/local_settings.py ?

I am tryed this but see SyntaxError in line 14 on the last line …

(graphite) [carbon@lmaster ~]$ PYTHONPATH=/opt/graphite/webapp django-admin.py migrate --settings=graphite.settings --run-syncdb
Traceback (most recent call last):
File “/opt/graphite/bin/django-admin.py”, line 5, in
management.execute_from_command_line()
File “/opt/graphite/webapp/django/core/management/init.py”, line 401, in execute_from_command_line
utility.execute()
File “/opt/graphite/webapp/django/core/management/init.py”, line 345, in execute
settings.INSTALLED_APPS
File “/opt/graphite/webapp/django/conf/init.py”, line 76, in getattr
self._setup(name)
File “/opt/graphite/webapp/django/conf/init.py”, line 63, in _setup
self._wrapped = Settings(settings_module)
File “/opt/graphite/webapp/django/conf/init.py”, line 142, in init
mod = importlib.import_module(self.SETTINGS_MODULE)
File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 678, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/opt/graphite/webapp/graphite/settings.py”, line 226, in
globals().update(import_module(SETTINGS_MODULE).dict)
File “/usr/lib64/python3.6/importlib/init.py”, line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 994, in _gcd_import
File “”, line 971, in _find_and_load
File “”, line 955, in _find_and_load_unlocked
File “”, line 665, in _load_unlocked
File “”, line 674, in exec_module
File “”, line 781, in get_code
File “”, line 741, in source_to_code
File “”, line 219, in _call_with_frames_removed
File “/opt/graphite/webapp/graphite/local_settings.py”, line 14
PYTHONPATH=/opt/graphite/webapp django-admin.py migrate
^
SyntaxError: invalid syntax
(graphite) [carbon@lmaster ~]$

Your test start the service carbon-relay is really run?
Because I am have constantly this state …

[root@lmaster ~]# systemctl status carbon-relay.service
● carbon-relay.service - Graphite Carbon Relay
Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2020-05-23 17:38:47 CEST; 24s ago
Process: 13987 ExecStart=/opt/graphite/bin/carbon-relay.py --config=/opt/graphite/conf/carbon.conf --pidfile=/var/run/carbon-relay.pid start >

May 23 17:38:47 lmaster.localdomain systemd[1]: carbon-relay.service: Failed with result ‘exit-code’.
May 23 17:38:47 lmaster.localdomain systemd[1]: Failed to start Graphite Carbon Relay.
May 23 17:38:47 lmaster.localdomain systemd[1]: carbon-relay.service: Service RestartSec=100ms expired, scheduling restart.
May 23 17:38:47 lmaster.localdomain systemd[1]: carbon-relay.service: Scheduled restart job, restart counter is at 5.
May 23 17:38:47 lmaster.localdomain systemd[1]: Stopped Graphite Carbon Relay.
May 23 17:38:47 lmaster.localdomain systemd[1]: carbon-relay.service: Start request repeated too quickly.
May 23 17:38:47 lmaster.localdomain systemd[1]: carbon-relay.service: Failed with result ‘exit-code’.
May 23 17:38:47 lmaster.localdomain systemd[1]: Failed to start Graphite Carbon Relay.
ESCOC

Hi Jozef.

Yes, I mean the

/opt/graphite/webapp/graphite/local_settings.py

Use this (the only difference are the single quotation marks):

SECRET_KEY = 'imalittleteapotshortandstout'
DEFAULT_XFILES_FACTOR = 0
URL_PREFIX = '/graphite'

CARBONLINK_HOSTS = ["127.0.0.1:7002:a", "127.0.0.1:7102:b"]
CARBONLINK_TIMEOUT = 1.0
CARBONLINK_RETRY_DELAY = 15 # Seconds to blacklist a failed remote server

REPLICATION_FACTOR = 1
FUNCTION_PLUGINS = []

Then run:

PYTHONPATH=/opt/graphite/webapp django-admin.py migrate

again. This process has to complete successfully.

I only used your configs and the services all started correctly.
The only thing which didn’t work was the db-migration process.

Hope this helps.

Greetings.