Hi all,
I have a problem with my graphite installation on a monitoring system.
# du -h -d1 /var/lib/graphite/storage/whisper/
19G /var/lib/graphite/storage/whisper/_tagged
2,2G /var/lib/graphite/storage/whisper/icinga2
42M /var/lib/graphite/storage/whisper/carbon
21G /var/lib/graphite/storage/whisper/
I used this How To to install graphite, but only set up a single instance carbon-cache-a:
The log folder is also quite big
# du -h /var/log/graphite/
116M /var/log/graphite/carbon-cache-a
118M /var/log/graphite/
total 118372
drwxr-xr-x 2 carbon carbon 4096 Mär 17 00:02 ./
drwxr-xr-x 3 carbon carbon 4096 Mär 17 13:08 ../
-rwxr-xr-x 1 carbon carbon 631400 Mär 17 15:53 console.log*
-rwxr-xr-x 1 carbon carbon 265867 Jan 15 13:22 console.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon 481219 Jan 22 23:55 console.log.2020_1_22*
-rwxr-xr-x 1 carbon carbon 614186 Jan 23 23:56 console.log.2020_1_23*
-rwxr-xr-x 1 carbon carbon 619359 Jan 24 23:58 console.log.2020_1_24*
-rwxr-xr-x 1 carbon carbon 354852 Jan 25 23:58 console.log.2020_1_25*
-rwxr-xr-x 1 carbon carbon 226789 Jan 26 22:38 console.log.2020_1_26*
-rwxr-xr-x 1 carbon carbon 637406 Jan 27 23:57 console.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon 778103 Jan 28 23:55 console.log.2020_1_28*
-rwxr-xr-x 1 carbon carbon 623876 Jan 29 23:54 console.log.2020_1_29*
-rwxr-xr-x 1 carbon carbon 526360 Jan 30 23:45 console.log.2020_1_30*
-rwxr-xr-x 1 carbon carbon 485714 Jan 31 23:50 console.log.2020_1_31*
-rwxr-xr-x 1 carbon carbon 84049 Feb 1 23:51 console.log.2020_2_1*
-rwxr-xr-x 1 carbon carbon 624972 Feb 10 23:56 console.log.2020_2_10*
-rwxr-xr-x 1 carbon carbon 698377 Feb 11 23:57 console.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon 623148 Feb 12 23:59 console.log.2020_2_12*
-rwxr-xr-x 1 carbon carbon 508709 Feb 13 23:59 console.log.2020_2_13*
-rwxr-xr-x 1 carbon carbon 544326 Feb 14 23:39 console.log.2020_2_14*
-rwxr-xr-x 1 carbon carbon 90660 Feb 15 23:58 console.log.2020_2_15*
-rwxr-xr-x 1 carbon carbon 103565 Feb 16 23:27 console.log.2020_2_16*
-rwxr-xr-x 1 carbon carbon 634430 Feb 17 23:57 console.log.2020_2_17*
-rwxr-xr-x 1 carbon carbon 858033 Feb 18 23:58 console.log.2020_2_18*
-rwxr-xr-x 1 carbon carbon 850113 Feb 19 23:59 console.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon 105022 Feb 2 23:26 console.log.2020_2_2*
-rwxr-xr-x 1 carbon carbon 756954 Feb 20 23:59 console.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon 749170 Feb 21 23:58 console.log.2020_2_21*
-rwxr-xr-x 1 carbon carbon 335498 Feb 22 23:50 console.log.2020_2_22*
-rwxr-xr-x 1 carbon carbon 98436 Feb 23 23:59 console.log.2020_2_23*
-rwxr-xr-x 1 carbon carbon 578560 Feb 24 23:56 console.log.2020_2_24*
-rwxr-xr-x 1 carbon carbon 676484 Feb 25 23:56 console.log.2020_2_25*
-rwxr-xr-x 1 carbon carbon 706348 Feb 26 23:56 console.log.2020_2_26*
-rwxr-xr-x 1 carbon carbon 640541 Feb 27 23:56 console.log.2020_2_27*
-rwxr-xr-x 1 carbon carbon 547282 Feb 28 23:56 console.log.2020_2_28*
-rwxr-xr-x 1 carbon carbon 366896 Feb 29 23:56 console.log.2020_2_29*
-rwxr-xr-x 1 carbon carbon 542356 Feb 3 23:55 console.log.2020_2_3*
-rwxr-xr-x 1 carbon carbon 692199 Feb 4 23:59 console.log.2020_2_4*
-rwxr-xr-x 1 carbon carbon 742028 Feb 5 23:58 console.log.2020_2_5*
-rwxr-xr-x 1 carbon carbon 601008 Feb 6 23:57 console.log.2020_2_6*
-rwxr-xr-x 1 carbon carbon 581650 Feb 7 23:58 console.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon 616688 Feb 8 23:59 console.log.2020_2_8*
-rwxr-xr-x 1 carbon carbon 341315 Feb 9 23:40 console.log.2020_2_9*
-rwxr-xr-x 1 carbon carbon 342850 Mär 1 23:41 console.log.2020_3_1*
-rwxr-xr-x 1 carbon carbon 935135 Mär 10 23:59 console.log.2020_3_10*
-rwxr-xr-x 1 carbon carbon 666995 Mär 11 23:45 console.log.2020_3_11*
-rwxr-xr-x 1 carbon carbon 546863 Mär 12 23:56 console.log.2020_3_12*
-rwxr-xr-x 1 carbon carbon 622233 Mär 13 23:59 console.log.2020_3_13*
-rwxr-xr-x 1 carbon carbon 203523 Mär 14 23:41 console.log.2020_3_14*
-rwxr-xr-x 1 carbon carbon 100114 Mär 15 23:59 console.log.2020_3_15*
-rwxr-xr-x 1 carbon carbon 694881 Mär 16 23:58 console.log.2020_3_16*
-rwxr-xr-x 1 carbon carbon 614848 Mär 2 23:56 console.log.2020_3_2*
-rwxr-xr-x 1 carbon carbon 709941 Mär 3 23:56 console.log.2020_3_3*
-rwxr-xr-x 1 carbon carbon 782593 Mär 4 23:59 console.log.2020_3_4*
-rwxr-xr-x 1 carbon carbon 657198 Mär 5 23:56 console.log.2020_3_5*
-rwxr-xr-x 1 carbon carbon 503557 Mär 6 23:51 console.log.2020_3_6*
-rwxr-xr-x 1 carbon carbon 82166 Mär 7 23:57 console.log.2020_3_7*
-rwxr-xr-x 1 carbon carbon 92794 Mär 8 23:59 console.log.2020_3_8*
-rwxr-xr-x 1 carbon carbon 749406 Mär 9 23:59 console.log.2020_3_9*
-rwxr-xr-x 1 carbon carbon 249 Mär 3 14:42 listener.log*
-rwxr-xr-x 1 carbon carbon 695 Jan 15 13:22 listener.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon 249 Jan 27 14:51 listener.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon 249 Feb 11 14:07 listener.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon 249 Feb 19 07:51 listener.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon 249 Feb 20 14:09 listener.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon 249 Feb 7 09:00 listener.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon 49 Mär 4 07:45 query.log*
-rwxr-xr-x 1 carbon carbon 233 Jan 15 13:22 query.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon 101 Jan 21 14:36 query.log.2020_1_21*
-rwxr-xr-x 1 carbon carbon 135 Jan 27 14:49 query.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon 101 Feb 11 14:11 query.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon 184 Feb 19 07:55 query.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon 184 Feb 20 14:09 query.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon 49 Feb 6 09:19 query.log.2020_2_6*
-rwxr-xr-x 1 carbon carbon 184 Feb 7 09:02 query.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon 135 Mär 3 14:41 query.log.2020_3_3*
-rwxr-xr-x 1 carbon carbon 1149816 Mär 17 15:57 tagdb.log*
-rwxr-xr-x 1 carbon carbon 85072 Jan 15 23:57 tagdb.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon 281138 Jan 16 23:58 tagdb.log.2020_1_16*
-rwxr-xr-x 1 carbon carbon 467140 Jan 17 23:59 tagdb.log.2020_1_17*
-rwxr-xr-x 1 carbon carbon 508112 Jan 18 23:59 tagdb.log.2020_1_18*
-rwxr-xr-x 1 carbon carbon 507305 Jan 19 23:59 tagdb.log.2020_1_19*
-rwxr-xr-x 1 carbon carbon 525421 Jan 20 23:59 tagdb.log.2020_1_20*
-rwxr-xr-x 1 carbon carbon 945805 Jan 21 23:59 tagdb.log.2020_1_21*
-rwxr-xr-x 1 carbon carbon 1336854 Jan 22 23:59 tagdb.log.2020_1_22*
-rwxr-xr-x 1 carbon carbon 1369146 Jan 23 23:59 tagdb.log.2020_1_23*
-rwxr-xr-x 1 carbon carbon 1436514 Jan 24 23:59 tagdb.log.2020_1_24*
-rwxr-xr-x 1 carbon carbon 1266295 Jan 25 23:59 tagdb.log.2020_1_25*
-rwxr-xr-x 1 carbon carbon 1254153 Jan 26 23:59 tagdb.log.2020_1_26*
-rwxr-xr-x 1 carbon carbon 1600309 Jan 27 23:59 tagdb.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon 1617157 Jan 28 23:59 tagdb.log.2020_1_28*
-rwxr-xr-x 1 carbon carbon 1609863 Jan 29 23:59 tagdb.log.2020_1_29*
-rwxr-xr-x 1 carbon carbon 1592980 Jan 30 23:59 tagdb.log.2020_1_30*
-rwxr-xr-x 1 carbon carbon 1565435 Jan 31 23:59 tagdb.log.2020_1_31*
-rwxr-xr-x 1 carbon carbon 1407715 Feb 1 23:59 tagdb.log.2020_2_1*
-rwxr-xr-x 1 carbon carbon 1643714 Feb 10 23:59 tagdb.log.2020_2_10*
-rwxr-xr-x 1 carbon carbon 1630925 Feb 11 23:59 tagdb.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon 1616394 Feb 12 23:59 tagdb.log.2020_2_12*
-rwxr-xr-x 1 carbon carbon 1593108 Feb 13 23:59 tagdb.log.2020_2_13*
-rwxr-xr-x 1 carbon carbon 1612013 Feb 14 23:59 tagdb.log.2020_2_14*
-rwxr-xr-x 1 carbon carbon 1436231 Feb 15 23:59 tagdb.log.2020_2_15*
-rwxr-xr-x 1 carbon carbon 1433211 Feb 16 23:59 tagdb.log.2020_2_16*
-rwxr-xr-x 1 carbon carbon 1623547 Feb 17 23:59 tagdb.log.2020_2_17*
-rwxr-xr-x 1 carbon carbon 1668697 Feb 18 23:59 tagdb.log.2020_2_18*
-rwxr-xr-x 1 carbon carbon 1659977 Feb 19 23:59 tagdb.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon 1405163 Feb 2 23:59 tagdb.log.2020_2_2*
-rwxr-xr-x 1 carbon carbon 1662482 Feb 20 23:59 tagdb.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon 1648012 Feb 21 23:59 tagdb.log.2020_2_21*
-rwxr-xr-x 1 carbon carbon 1431667 Feb 22 23:59 tagdb.log.2020_2_22*
-rwxr-xr-x 1 carbon carbon 1469925 Feb 23 23:59 tagdb.log.2020_2_23*
-rwxr-xr-x 1 carbon carbon 1673293 Feb 24 23:59 tagdb.log.2020_2_24*
-rwxr-xr-x 1 carbon carbon 1654439 Feb 25 23:59 tagdb.log.2020_2_25*
-rwxr-xr-x 1 carbon carbon 1674167 Feb 26 23:59 tagdb.log.2020_2_26*
-rwxr-xr-x 1 carbon carbon 1652779 Feb 27 23:59 tagdb.log.2020_2_27*
-rwxr-xr-x 1 carbon carbon 1636852 Feb 28 23:59 tagdb.log.2020_2_28*
-rwxr-xr-x 1 carbon carbon 1478787 Feb 29 23:59 tagdb.log.2020_2_29*
-rwxr-xr-x 1 carbon carbon 1580752 Feb 3 23:59 tagdb.log.2020_2_3*
-rwxr-xr-x 1 carbon carbon 1591792 Feb 4 23:59 tagdb.log.2020_2_4*
-rwxr-xr-x 1 carbon carbon 1591451 Feb 5 23:59 tagdb.log.2020_2_5*
-rwxr-xr-x 1 carbon carbon 1621344 Feb 6 23:59 tagdb.log.2020_2_6*
-rwxr-xr-x 1 carbon carbon 1585619 Feb 7 23:59 tagdb.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon 1445692 Feb 8 23:59 tagdb.log.2020_2_8*
-rwxr-xr-x 1 carbon carbon 1440083 Feb 9 23:59 tagdb.log.2020_2_9*
-rwxr-xr-x 1 carbon carbon 1473057 Mär 1 23:59 tagdb.log.2020_3_1*
-rwxr-xr-x 1 carbon carbon 1710197 Mär 10 23:59 tagdb.log.2020_3_10*
-rwxr-xr-x 1 carbon carbon 1656068 Mär 11 23:59 tagdb.log.2020_3_11*
-rwxr-xr-x 1 carbon carbon 1664228 Mär 12 23:59 tagdb.log.2020_3_12*
-rwxr-xr-x 1 carbon carbon 1657353 Mär 13 23:59 tagdb.log.2020_3_13*
-rwxr-xr-x 1 carbon carbon 1493323 Mär 14 23:59 tagdb.log.2020_3_14*
-rwxr-xr-x 1 carbon carbon 1497172 Mär 15 23:59 tagdb.log.2020_3_15*
-rwxr-xr-x 1 carbon carbon 1693392 Mär 16 23:59 tagdb.log.2020_3_16*
-rwxr-xr-x 1 carbon carbon 1665188 Mär 2 23:59 tagdb.log.2020_3_2*
-rwxr-xr-x 1 carbon carbon 1668424 Mär 3 23:59 tagdb.log.2020_3_3*
-rwxr-xr-x 1 carbon carbon 1698673 Mär 4 23:59 tagdb.log.2020_3_4*
-rwxr-xr-x 1 carbon carbon 1674007 Mär 5 23:59 tagdb.log.2020_3_5*
-rwxr-xr-x 1 carbon carbon 1637025 Mär 6 23:59 tagdb.log.2020_3_6*
-rwxr-xr-x 1 carbon carbon 1482534 Mär 7 23:59 tagdb.log.2020_3_7*
-rwxr-xr-x 1 carbon carbon 1483121 Mär 8 23:59 tagdb.log.2020_3_8*
-rwxr-xr-x 1 carbon carbon 1719663 Mär 9 23:59 tagdb.log.2020_3_9*
carbon.conf
# grep "^[^#;]" conf/carbon.conf
[cache]
LOCAL_DATA_DIR = /var/lib/graphite/storage/whisper/
LOG_DIR = /var/log/graphite/
STORAGE_DIR = /var/lib/graphite/storage
DATABASE = whisper
ENABLE_LOGROTATION = True
USER = carbon
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50
MIN_TIMESTAMP_RESOLUTION = 1
LINE_RECEIVER_INTERFACE = 0.0.0.0
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_MAX_LENGTH = 1048576
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
USE_FLOW_CONTROL = True
LOG_UPDATES = False
LOG_CREATES = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False
CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
WHISPER_LOCK_WRITES = True
GRAPHITE_URL = http://127.0.0.1:8000/graphite
[cache:a]
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_PORT = 7002
[relay]
LINE_RECEIVER_INTERFACE = 127.0.0.1
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
PICKLE_RECEIVER_MAX_LENGTH = 1048576
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004:a
DESTINATION_PROTOCOL = pickle
DESTINATION_TRANSPORT = none
DYNAMIC_ROUTER = True
MAX_QUEUE_SIZE = 10000
MAX_DATAPOINTS_PER_MESSAGE = 500
USER = carbon
QUEUE_LOW_WATERMARK_PCT = 0.8
TIME_TO_DEFER_SENDING = 0.0001
USE_FLOW_CONTROL = True
USE_RATIO_RESET=False
MIN_RESET_STAT_FLOW=1000
MIN_RESET_RATIO=0.9
MIN_RESET_INTERVAL=121
[aggregator]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
FORWARD_ALL = True
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5
Services are running.
Interesting though, that Graphite-Web can’t be reached.
Graphs are displayed though…
# systemctl status carbon-cache@a
â—Ź carbon-cache@a.service - carbon-cache instance a (graphite)
Loaded: loaded (/etc/systemd/system/carbon-cache@.service; indirect; vendor preset: enabled)
Active: active (running) since Tue 2020-03-03 14:42:45 CET; 2 weeks 0 days ago
Process: 1351 ExecStart=/opt/graphite/bin/carbon-cache.py --instance=a start --pidfile=/opt/graphite/storage/carb
Process: 1180 ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-cache-a.pid (code=exited, status=0/SUCCESS)
Main PID: 2028 (carbon-cache.py)
Tasks: 3 (limit: 4594)
CGroup: /system.slice/system-carbon\x2dcache.slice/carbon-cache@a.service
└─2028 /opt/graphite/bin/python3 /opt/graphite/bin/carbon-cache.py --instance=a start --pidfile=/opt/gra
Mär 03 14:42:03 SERVER systemd[1]: Starting carbon-cache instance a (graphite)...
Mär 03 14:42:45 SERVER systemd[1]: Started carbon-cache instance a (graphite).
# systemctl status carbon-relay.service
â—Ź carbon-relay.service - Graphite Carbon Relay
Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-03-03 14:42:44 CET; 2 weeks 0 days ago
Process: 989 ExecStart=/opt/graphite/bin/carbon-relay.py --config=/opt/graphite/conf/carbon.conf --pidfile=/var/r
Main PID: 2021 (carbon-relay.py)
Tasks: 1 (limit: 4594)
CGroup: /system.slice/carbon-relay.service
└─2021 /opt/graphite/bin/python3 /opt/graphite/bin/carbon-relay.py --config=/opt/graphite/conf/carbon.co
Mär 03 14:42:01 SERVER systemd[1]: Starting Graphite Carbon Relay...
Mär 03 14:42:44 SERVER systemd[1]: Started Graphite Carbon Relay.
# systemctl status graphite
â—Ź graphite.service - Graphite
Loaded: loaded (/etc/systemd/system/graphite.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-03-03 14:42:03 CET; 2 weeks 0 days ago
Main PID: 1143 (gunicorn)
Tasks: 6 (limit: 4594)
CGroup: /system.slice/graphite.service
├─1143 /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn --bind 0.0.0.0:8000 graphite.wsgi
└─1910 /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn --bind 0.0.0.0:8000 graphite.wsgi
Mär 03 14:42:03 SERVER systemd[1]: Started Graphite.
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1143] [INFO] Starting gunicorn 20.0.4
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1143] [INFO] Listening at: http://0.0.0.0
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1143] [INFO] Using worker: sync
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1910] [INFO] Booting worker with pid: 191
tagdb.log
09/03/2020 23:58:26 :: Error tagging icinga2.somehostname.services.memory.nwc_health.perfdata.Processor_usage.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:58:32 :: Error tagging icinga2.somehostname.services.load.kemp_loadmaster.perfdata.Load-5.value: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:58:44 :: Error tagging icinga2.somehostname.services.swap.swap-windows.perfdata.swap.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:58:52 :: Error tagging icinga2.somehostname.services.memory.memory-windows.perfdata.memory.max: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:06 :: Error tagging icinga2.somehostname.services.disk-D.disk-windows.perfdata.D:_.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:18 :: Error tagging icinga2.somehostname.host.icmp.perfdata.pl.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:27 :: Error tagging icinga2.somehostname.services.memory.nwc_health.perfdata.MEMPOOL_MSGLYR_2_usage.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:37 :: Error tagging icinga2.somehostname.services.swap.swap-windows.perfdata.swap.value: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:47 :: Error tagging icinga2.somehostname.host.icmp.perfdata.rta.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:53 :: Error tagging icinga2.somehostname.services.volume-usage.netapp_cdot_volume_status.perfdata.Vol_CIFS_CIFS_root.check_cdot_volume_usage.inode_used.warn, icinga2.NetApp_xyz-xyz.services.volume-usage.netapp_cdot_volume_status.perfdata.Vol_FC_FC_data2_SATA.check_cdot_volume_usage.snap_used.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:59 :: Error tagging icinga2.somehostname.host.icmp.perfdata.rta.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
Consol log seems to only contain messages about two checks, that are monitoring exchange queues:
14/03/2020 22:50:49 :: Error parsing metric icinga2.SERVER-EXCH02-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH02-x_31=0;20;40;0SERVER-EXCH02-x_Submission.warn: Cannot parse path icinga2.SERVER-EXCH02-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH02-x_31=0;20;40;0SERVER-EXCH02-x_Submission.warn, invalid segment 20
14/03/2020 23:11:10 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.value: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.value, invalid segment 20
14/03/2020 23:11:10 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.crit: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.crit, invalid segment 20
14/03/2020 23:11:10 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.warn: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.warn, invalid segment 20
14/03/2020 23:41:46 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.value: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.value, invalid segment 20
14/03/2020 23:41:46 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.crit: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.crit, invalid segment 20
14/03/2020 23:41:46 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.warn: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.warn, invalid segment 20
Did I **** something up during the installation?
Seems like this is going on since the beginning, at least the logs go back that far.
Can I safely delete the _tagged folder? (Before/after fixing the problem ofc ;))