Sorry for my last post without any details. I was a little bit frustrated and didn’t thought about.
Let me give you some more details because I don’t how to find out what is missing or going worng.
OS is Raspbian Buster.
I am using Icinga2 with Icinga Web 2 (V. 2.6.2). I installed Graphite (V. 1.1.0) and everything is running on NGINX.
I installed graphite-web and graphite-carbon via apt. I got the module “graphite” via Github. For NGINX uwsgi is installed.
Icvinga2 is running well. I got alarms, performance data … everything I need. Only graphite is showing “No data for graph” or is serving empty graphs.
No I looked at different logfiles but there are not hints whats going wrong.
I give you data of my logfiles:
/var/log/nginx/graphite-access.log:
IP - - [06/Nov/2019:19:50:16 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:16 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:26 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value HTTP/1.1" 200 160 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:26 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:26 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:37 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value HTTP/1.1" 200 160 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:37 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:37 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:48 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value HTTP/1.1" 200 160 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:48 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:48 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:58 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value HTTP/1.1" 200 160 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:58 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:50:58 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:51:09 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value HTTP/1.1" 200 160 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:51:09 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:51:09 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:53:28 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value HTTP/1.1" 200 160 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:53:29 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
IP - - [06/Nov/2019:19:53:29 +0100] "GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value HTTP/1.1" 200 15 "-" "icingaweb2-module-graphite"
The corresponging error.log:
EMPTY!!!
The NGINX error.log:
EMPTY!!!
The carbon listener.log (I did some restarts during the day):
06/11/2019 16:47:27 :: MetricLineReceiver connection with 127.0.0.1:48586 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
06/11/2019 16:47:33 :: MetricLineReceiver connection with 127.0.0.1:50652 established
06/11/2019 16:47:46 :: MetricLineReceiver connection with 127.0.0.1:50652 closed cleanly
06/11/2019 16:47:47 :: MetricLineReceiver connection with 127.0.0.1:50658 established
06/11/2019 17:10:39 :: MetricLineReceiver connection with 127.0.0.1:50658 closed cleanly
06/11/2019 17:10:40 :: MetricLineReceiver connection with 127.0.0.1:50938 established
06/11/2019 17:14:36 :: MetricLineReceiver connection with 127.0.0.1:50938 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
06/11/2019 17:14:40 :: MetricLineReceiver connection with 127.0.0.1:50968 established
06/11/2019 17:28:11 :: MetricLineReceiver connection with 127.0.0.1:50968 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
06/11/2019 17:28:20 :: MetricLineReceiver connection with 127.0.0.1:51146 established
06/11/2019 18:21:01 :: MetricLineReceiver connection with 127.0.0.1:51146 closed cleanly
06/11/2019 18:21:02 :: MetricLineReceiver connection with 127.0.0.1:52198 established
06/11/2019 18:21:06 :: MetricLineReceiver connection with 127.0.0.1:52198 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
06/11/2019 18:21:12 :: MetricLineReceiver connection with 127.0.0.1:52206 established
06/11/2019 18:28:50 :: MetricLineReceiver connection with 127.0.0.1:52206 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
06/11/2019 18:28:57 :: MetricLineReceiver connection with 127.0.0.1:52242 established
06/11/2019 19:58:19 :: MetricLineReceiver connection with 127.0.0.1:52242 closed cleanly
06/11/2019 19:58:20 :: MetricLineReceiver connection with 127.0.0.1:53420 established
06/11/2019 20:03:18 :: MetricLineReceiver connection with 127.0.0.1:53420 lost: Connection to the other side was lost in a non-clean fashion: Connection lost.
06/11/2019 20:03:30 :: MetricLineReceiver connection with 127.0.0.1:53436 established
The console.log of carbon:
06/11/2019 20:03:18 :: (TCP Port 2003 Closed)
06/11/2019 20:03:18 :: Stopping factory <carbon.protocols.CarbonReceiverFactory object at 0xb553cc90>
06/11/2019 20:03:18 :: (TCP Port 7002 Closed)
06/11/2019 20:03:18 :: Stopping factory <twisted.internet.protocol.ServerFactory object at 0xb579e070>
06/11/2019 20:03:19 :: Main loop terminated.
06/11/2019 20:03:19 :: Warning: No permission to delete pid file
06/11/2019 20:03:19 :: Server Shut Down.
06/11/2019 20:03:20 :: Using sorted write strategy for cache
06/11/2019 20:03:20 :: Enabling Whisper fallocate support
06/11/2019 20:03:20 :: /etc/carbon/storage-aggregation.conf not found or wrong perms, ignoring.
06/11/2019 20:03:20 :: twistd 18.9.0 (/usr/bin/python3 3.7.3) starting up.
06/11/2019 20:03:20 :: reactor class: twisted.internet.epollreactor.EPollReactor.
06/11/2019 20:03:20 :: ServerFactory starting on 7002
06/11/2019 20:03:20 :: Starting factory <twisted.internet.protocol.ServerFactory object at 0xb5533210>
06/11/2019 20:03:20 :: set uid/gid 112/118
06/11/2019 20:03:20 :: Installing SIG_IGN for SIGHUP
06/11/2019 20:03:20 :: CarbonReceiverFactory starting on 2003
06/11/2019 20:03:20 :: Starting factory <carbon.protocols.CarbonReceiverFactory object at 0xb5556b70>
06/11/2019 20:03:20 :: CarbonReceiverFactory starting on 2004
06/11/2019 20:03:20 :: Starting factory <carbon.protocols.CarbonReceiverFactory object at 0xb5556b30>
The uwsgi graphite.log:
[ pid: 30492|app: 0|req: 688/1525] IP () {30 vars in 635 bytes} [Wed Nov 6 19:50:16 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 689/1526] IP () {30 vars in 643 bytes} [Wed Nov 6 19:50:16 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 690/1527] IP () {30 vars in 627 bytes} [Wed Nov 6 19:50:26 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value => generated 160 bytes in 8 msecs (HTTP/1.1 200) 4 headers in 115 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 691/1528] IP () {30 vars in 635 bytes} [Wed Nov 6 19:50:26 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 692/1529] IP () {30 vars in 643 bytes} [Wed Nov 6 19:50:26 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 693/1530] IP () {30 vars in 627 bytes} [Wed Nov 6 19:50:37 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value => generated 160 bytes in 6 msecs (HTTP/1.1 200) 4 headers in 115 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 694/1531] IP () {30 vars in 635 bytes} [Wed Nov 6 19:50:37 2019] GET /metrics/expand?query=icinga2.itunesserver_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 695/1532] IP () {30 vars in 643 bytes} [Wed Nov 6 19:50:37 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 194/1533] IP () {30 vars in 627 bytes} [Wed Nov 6 19:50:48 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value => generated 160 bytes in 9 msecs (HTTP/1.1 200) 4 headers in 115 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 696/1534] IP () {30 vars in 635 bytes} [Wed Nov 6 19:50:48 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 7 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 697/1535] IP () {30 vars in 643 bytes} [Wed Nov 6 19:50:48 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 7 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 195/1536] IP () {30 vars in 627 bytes} [Wed Nov 6 19:50:58 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value => generated 160 bytes in 6 msecs (HTTP/1.1 200) 4 headers in 115 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 196/1537] IP () {30 vars in 635 bytes} [Wed Nov 6 19:50:58 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 197/1538] IP () {30 vars in 643 bytes} [Wed Nov 6 19:50:58 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 198/1539] IP () {30 vars in 627 bytes} [Wed Nov 6 19:51:09 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value => generated 160 bytes in 4 msecs (HTTP/1.1 200) 4 headers in 115 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 199/1540] IP () {30 vars in 635 bytes} [Wed Nov 6 19:51:09 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 200/1541] IP () {30 vars in 643 bytes} [Wed Nov 6 19:51:09 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 3 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 201/1542] IP () {30 vars in 627 bytes} [Wed Nov 6 19:53:28 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.value => generated 160 bytes in 7 msecs (HTTP/1.1 200) 4 headers in 115 bytes (1 switches on core 0)
[pid: 30492|app: 0|req: 698/1543] IP () {30 vars in 635 bytes} [Wed Nov 6 19:53:28 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.value => generated 15 bytes in 11 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
[pid: 30490|app: 0|req: 202/1544] IP () {30 vars in 643 bytes} [Wed Nov 6 19:53:29 2019] GET /metrics/expand?query=icinga2.server_lan.services.Drive_Z%3A_-_Avg__Second_Per_Data_Request.nscp-local-counter.perfdata.%2A.%2A.%2A.value => generated 15 bytes in 10 msecs (HTTP/1.1 200) 4 headers in 114 bytes (1 switches on core 0)
This is my storage_schema.conf:
[carbon]
pattern = ^carbon.
retentions = 60s:90d
[icinga2_internals]
pattern = ^icinga2..* (max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 5m:7d
[icinga2_default]
pattern = ^icinga2.
retentions = 1m:2d,5m:10d,30m:90d,60m:180d,360m:4y
#retentions = 1m:2d
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d
This is my /etc/icingaweb2/modules/graphite/config.ini:
[graphite]
url = “http://IP:9002”
insecure = “0”
This is an example where the data of whisper is located:
/var/lib/graphite/whisper/icinga2/server_lan/services/Drive_Z:_-_Avg__Second_Per_Data_Request/nscp-local-counter/perfdata/Datendurchsatz_NAS_Multimedia_value# ls
crit.wsp value.wsp warn.wsp
Do you need more informations, configs or logs? Please ask.
Do you have any suggestions what is going I try everything you ask me to 
Thank you for your help,
Jens