Graphite: Data piles up in _tagged folder

Hi all,

I have a problem with my graphite installation on a monitoring system.

# du -h -d1 /var/lib/graphite/storage/whisper/
19G     /var/lib/graphite/storage/whisper/_tagged
2,2G    /var/lib/graphite/storage/whisper/icinga2
42M     /var/lib/graphite/storage/whisper/carbon
21G     /var/lib/graphite/storage/whisper/

I used this How To to install graphite, but only set up a single instance carbon-cache-a:

The log folder is also quite big

# du -h /var/log/graphite/
116M    /var/log/graphite/carbon-cache-a
118M    /var/log/graphite/
total 118372
drwxr-xr-x 2 carbon carbon    4096 Mär 17 00:02 ./
drwxr-xr-x 3 carbon carbon    4096 Mär 17 13:08 ../
-rwxr-xr-x 1 carbon carbon  631400 Mär 17 15:53 console.log*
-rwxr-xr-x 1 carbon carbon  265867 Jan 15 13:22 console.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon  481219 Jan 22 23:55 console.log.2020_1_22*
-rwxr-xr-x 1 carbon carbon  614186 Jan 23 23:56 console.log.2020_1_23*
-rwxr-xr-x 1 carbon carbon  619359 Jan 24 23:58 console.log.2020_1_24*
-rwxr-xr-x 1 carbon carbon  354852 Jan 25 23:58 console.log.2020_1_25*
-rwxr-xr-x 1 carbon carbon  226789 Jan 26 22:38 console.log.2020_1_26*
-rwxr-xr-x 1 carbon carbon  637406 Jan 27 23:57 console.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon  778103 Jan 28 23:55 console.log.2020_1_28*
-rwxr-xr-x 1 carbon carbon  623876 Jan 29 23:54 console.log.2020_1_29*
-rwxr-xr-x 1 carbon carbon  526360 Jan 30 23:45 console.log.2020_1_30*
-rwxr-xr-x 1 carbon carbon  485714 Jan 31 23:50 console.log.2020_1_31*
-rwxr-xr-x 1 carbon carbon   84049 Feb  1 23:51 console.log.2020_2_1*
-rwxr-xr-x 1 carbon carbon  624972 Feb 10 23:56 console.log.2020_2_10*
-rwxr-xr-x 1 carbon carbon  698377 Feb 11 23:57 console.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon  623148 Feb 12 23:59 console.log.2020_2_12*
-rwxr-xr-x 1 carbon carbon  508709 Feb 13 23:59 console.log.2020_2_13*
-rwxr-xr-x 1 carbon carbon  544326 Feb 14 23:39 console.log.2020_2_14*
-rwxr-xr-x 1 carbon carbon   90660 Feb 15 23:58 console.log.2020_2_15*
-rwxr-xr-x 1 carbon carbon  103565 Feb 16 23:27 console.log.2020_2_16*
-rwxr-xr-x 1 carbon carbon  634430 Feb 17 23:57 console.log.2020_2_17*
-rwxr-xr-x 1 carbon carbon  858033 Feb 18 23:58 console.log.2020_2_18*
-rwxr-xr-x 1 carbon carbon  850113 Feb 19 23:59 console.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon  105022 Feb  2 23:26 console.log.2020_2_2*
-rwxr-xr-x 1 carbon carbon  756954 Feb 20 23:59 console.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon  749170 Feb 21 23:58 console.log.2020_2_21*
-rwxr-xr-x 1 carbon carbon  335498 Feb 22 23:50 console.log.2020_2_22*
-rwxr-xr-x 1 carbon carbon   98436 Feb 23 23:59 console.log.2020_2_23*
-rwxr-xr-x 1 carbon carbon  578560 Feb 24 23:56 console.log.2020_2_24*
-rwxr-xr-x 1 carbon carbon  676484 Feb 25 23:56 console.log.2020_2_25*
-rwxr-xr-x 1 carbon carbon  706348 Feb 26 23:56 console.log.2020_2_26*
-rwxr-xr-x 1 carbon carbon  640541 Feb 27 23:56 console.log.2020_2_27*
-rwxr-xr-x 1 carbon carbon  547282 Feb 28 23:56 console.log.2020_2_28*
-rwxr-xr-x 1 carbon carbon  366896 Feb 29 23:56 console.log.2020_2_29*
-rwxr-xr-x 1 carbon carbon  542356 Feb  3 23:55 console.log.2020_2_3*
-rwxr-xr-x 1 carbon carbon  692199 Feb  4 23:59 console.log.2020_2_4*
-rwxr-xr-x 1 carbon carbon  742028 Feb  5 23:58 console.log.2020_2_5*
-rwxr-xr-x 1 carbon carbon  601008 Feb  6 23:57 console.log.2020_2_6*
-rwxr-xr-x 1 carbon carbon  581650 Feb  7 23:58 console.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon  616688 Feb  8 23:59 console.log.2020_2_8*
-rwxr-xr-x 1 carbon carbon  341315 Feb  9 23:40 console.log.2020_2_9*
-rwxr-xr-x 1 carbon carbon  342850 Mär  1 23:41 console.log.2020_3_1*
-rwxr-xr-x 1 carbon carbon  935135 Mär 10 23:59 console.log.2020_3_10*
-rwxr-xr-x 1 carbon carbon  666995 Mär 11 23:45 console.log.2020_3_11*
-rwxr-xr-x 1 carbon carbon  546863 Mär 12 23:56 console.log.2020_3_12*
-rwxr-xr-x 1 carbon carbon  622233 Mär 13 23:59 console.log.2020_3_13*
-rwxr-xr-x 1 carbon carbon  203523 Mär 14 23:41 console.log.2020_3_14*
-rwxr-xr-x 1 carbon carbon  100114 Mär 15 23:59 console.log.2020_3_15*
-rwxr-xr-x 1 carbon carbon  694881 Mär 16 23:58 console.log.2020_3_16*
-rwxr-xr-x 1 carbon carbon  614848 Mär  2 23:56 console.log.2020_3_2*
-rwxr-xr-x 1 carbon carbon  709941 Mär  3 23:56 console.log.2020_3_3*
-rwxr-xr-x 1 carbon carbon  782593 Mär  4 23:59 console.log.2020_3_4*
-rwxr-xr-x 1 carbon carbon  657198 Mär  5 23:56 console.log.2020_3_5*
-rwxr-xr-x 1 carbon carbon  503557 Mär  6 23:51 console.log.2020_3_6*
-rwxr-xr-x 1 carbon carbon   82166 Mär  7 23:57 console.log.2020_3_7*
-rwxr-xr-x 1 carbon carbon   92794 Mär  8 23:59 console.log.2020_3_8*
-rwxr-xr-x 1 carbon carbon  749406 Mär  9 23:59 console.log.2020_3_9*
-rwxr-xr-x 1 carbon carbon     249 Mär  3 14:42 listener.log*
-rwxr-xr-x 1 carbon carbon     695 Jan 15 13:22 listener.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon     249 Jan 27 14:51 listener.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon     249 Feb 11 14:07 listener.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon     249 Feb 19 07:51 listener.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon     249 Feb 20 14:09 listener.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon     249 Feb  7 09:00 listener.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon      49 Mär  4 07:45 query.log*
-rwxr-xr-x 1 carbon carbon     233 Jan 15 13:22 query.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon     101 Jan 21 14:36 query.log.2020_1_21*
-rwxr-xr-x 1 carbon carbon     135 Jan 27 14:49 query.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon     101 Feb 11 14:11 query.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon     184 Feb 19 07:55 query.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon     184 Feb 20 14:09 query.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon      49 Feb  6 09:19 query.log.2020_2_6*
-rwxr-xr-x 1 carbon carbon     184 Feb  7 09:02 query.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon     135 Mär  3 14:41 query.log.2020_3_3*
-rwxr-xr-x 1 carbon carbon 1149816 Mär 17 15:57 tagdb.log*
-rwxr-xr-x 1 carbon carbon   85072 Jan 15 23:57 tagdb.log.2020_1_15*
-rwxr-xr-x 1 carbon carbon  281138 Jan 16 23:58 tagdb.log.2020_1_16*
-rwxr-xr-x 1 carbon carbon  467140 Jan 17 23:59 tagdb.log.2020_1_17*
-rwxr-xr-x 1 carbon carbon  508112 Jan 18 23:59 tagdb.log.2020_1_18*
-rwxr-xr-x 1 carbon carbon  507305 Jan 19 23:59 tagdb.log.2020_1_19*
-rwxr-xr-x 1 carbon carbon  525421 Jan 20 23:59 tagdb.log.2020_1_20*
-rwxr-xr-x 1 carbon carbon  945805 Jan 21 23:59 tagdb.log.2020_1_21*
-rwxr-xr-x 1 carbon carbon 1336854 Jan 22 23:59 tagdb.log.2020_1_22*
-rwxr-xr-x 1 carbon carbon 1369146 Jan 23 23:59 tagdb.log.2020_1_23*
-rwxr-xr-x 1 carbon carbon 1436514 Jan 24 23:59 tagdb.log.2020_1_24*
-rwxr-xr-x 1 carbon carbon 1266295 Jan 25 23:59 tagdb.log.2020_1_25*
-rwxr-xr-x 1 carbon carbon 1254153 Jan 26 23:59 tagdb.log.2020_1_26*
-rwxr-xr-x 1 carbon carbon 1600309 Jan 27 23:59 tagdb.log.2020_1_27*
-rwxr-xr-x 1 carbon carbon 1617157 Jan 28 23:59 tagdb.log.2020_1_28*
-rwxr-xr-x 1 carbon carbon 1609863 Jan 29 23:59 tagdb.log.2020_1_29*
-rwxr-xr-x 1 carbon carbon 1592980 Jan 30 23:59 tagdb.log.2020_1_30*
-rwxr-xr-x 1 carbon carbon 1565435 Jan 31 23:59 tagdb.log.2020_1_31*
-rwxr-xr-x 1 carbon carbon 1407715 Feb  1 23:59 tagdb.log.2020_2_1*
-rwxr-xr-x 1 carbon carbon 1643714 Feb 10 23:59 tagdb.log.2020_2_10*
-rwxr-xr-x 1 carbon carbon 1630925 Feb 11 23:59 tagdb.log.2020_2_11*
-rwxr-xr-x 1 carbon carbon 1616394 Feb 12 23:59 tagdb.log.2020_2_12*
-rwxr-xr-x 1 carbon carbon 1593108 Feb 13 23:59 tagdb.log.2020_2_13*
-rwxr-xr-x 1 carbon carbon 1612013 Feb 14 23:59 tagdb.log.2020_2_14*
-rwxr-xr-x 1 carbon carbon 1436231 Feb 15 23:59 tagdb.log.2020_2_15*
-rwxr-xr-x 1 carbon carbon 1433211 Feb 16 23:59 tagdb.log.2020_2_16*
-rwxr-xr-x 1 carbon carbon 1623547 Feb 17 23:59 tagdb.log.2020_2_17*
-rwxr-xr-x 1 carbon carbon 1668697 Feb 18 23:59 tagdb.log.2020_2_18*
-rwxr-xr-x 1 carbon carbon 1659977 Feb 19 23:59 tagdb.log.2020_2_19*
-rwxr-xr-x 1 carbon carbon 1405163 Feb  2 23:59 tagdb.log.2020_2_2*
-rwxr-xr-x 1 carbon carbon 1662482 Feb 20 23:59 tagdb.log.2020_2_20*
-rwxr-xr-x 1 carbon carbon 1648012 Feb 21 23:59 tagdb.log.2020_2_21*
-rwxr-xr-x 1 carbon carbon 1431667 Feb 22 23:59 tagdb.log.2020_2_22*
-rwxr-xr-x 1 carbon carbon 1469925 Feb 23 23:59 tagdb.log.2020_2_23*
-rwxr-xr-x 1 carbon carbon 1673293 Feb 24 23:59 tagdb.log.2020_2_24*
-rwxr-xr-x 1 carbon carbon 1654439 Feb 25 23:59 tagdb.log.2020_2_25*
-rwxr-xr-x 1 carbon carbon 1674167 Feb 26 23:59 tagdb.log.2020_2_26*
-rwxr-xr-x 1 carbon carbon 1652779 Feb 27 23:59 tagdb.log.2020_2_27*
-rwxr-xr-x 1 carbon carbon 1636852 Feb 28 23:59 tagdb.log.2020_2_28*
-rwxr-xr-x 1 carbon carbon 1478787 Feb 29 23:59 tagdb.log.2020_2_29*
-rwxr-xr-x 1 carbon carbon 1580752 Feb  3 23:59 tagdb.log.2020_2_3*
-rwxr-xr-x 1 carbon carbon 1591792 Feb  4 23:59 tagdb.log.2020_2_4*
-rwxr-xr-x 1 carbon carbon 1591451 Feb  5 23:59 tagdb.log.2020_2_5*
-rwxr-xr-x 1 carbon carbon 1621344 Feb  6 23:59 tagdb.log.2020_2_6*
-rwxr-xr-x 1 carbon carbon 1585619 Feb  7 23:59 tagdb.log.2020_2_7*
-rwxr-xr-x 1 carbon carbon 1445692 Feb  8 23:59 tagdb.log.2020_2_8*
-rwxr-xr-x 1 carbon carbon 1440083 Feb  9 23:59 tagdb.log.2020_2_9*
-rwxr-xr-x 1 carbon carbon 1473057 Mär  1 23:59 tagdb.log.2020_3_1*
-rwxr-xr-x 1 carbon carbon 1710197 Mär 10 23:59 tagdb.log.2020_3_10*
-rwxr-xr-x 1 carbon carbon 1656068 Mär 11 23:59 tagdb.log.2020_3_11*
-rwxr-xr-x 1 carbon carbon 1664228 Mär 12 23:59 tagdb.log.2020_3_12*
-rwxr-xr-x 1 carbon carbon 1657353 Mär 13 23:59 tagdb.log.2020_3_13*
-rwxr-xr-x 1 carbon carbon 1493323 Mär 14 23:59 tagdb.log.2020_3_14*
-rwxr-xr-x 1 carbon carbon 1497172 Mär 15 23:59 tagdb.log.2020_3_15*
-rwxr-xr-x 1 carbon carbon 1693392 Mär 16 23:59 tagdb.log.2020_3_16*
-rwxr-xr-x 1 carbon carbon 1665188 Mär  2 23:59 tagdb.log.2020_3_2*
-rwxr-xr-x 1 carbon carbon 1668424 Mär  3 23:59 tagdb.log.2020_3_3*
-rwxr-xr-x 1 carbon carbon 1698673 Mär  4 23:59 tagdb.log.2020_3_4*
-rwxr-xr-x 1 carbon carbon 1674007 Mär  5 23:59 tagdb.log.2020_3_5*
-rwxr-xr-x 1 carbon carbon 1637025 Mär  6 23:59 tagdb.log.2020_3_6*
-rwxr-xr-x 1 carbon carbon 1482534 Mär  7 23:59 tagdb.log.2020_3_7*
-rwxr-xr-x 1 carbon carbon 1483121 Mär  8 23:59 tagdb.log.2020_3_8*
-rwxr-xr-x 1 carbon carbon 1719663 Mär  9 23:59 tagdb.log.2020_3_9*
carbon.conf
# grep "^[^#;]" conf/carbon.conf
[cache]
LOCAL_DATA_DIR = /var/lib/graphite/storage/whisper/
LOG_DIR        = /var/log/graphite/
STORAGE_DIR = /var/lib/graphite/storage
DATABASE = whisper
ENABLE_LOGROTATION = True
USER = carbon
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 500
MAX_CREATES_PER_MINUTE = 50
MIN_TIMESTAMP_RESOLUTION = 1
LINE_RECEIVER_INTERFACE = 0.0.0.0
ENABLE_UDP_LISTENER = False
UDP_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_MAX_LENGTH = 1048576
USE_INSECURE_UNPICKLER = False
CACHE_QUERY_INTERFACE = 0.0.0.0
USE_FLOW_CONTROL = True
LOG_UPDATES = False
LOG_CREATES = False
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = False
CACHE_WRITE_STRATEGY = sorted
WHISPER_AUTOFLUSH = False
WHISPER_FALLOCATE_CREATE = True
WHISPER_LOCK_WRITES = True
GRAPHITE_URL = http://127.0.0.1:8000/graphite
[cache:a]
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_PORT = 7002
[relay]
LINE_RECEIVER_INTERFACE = 127.0.0.1
LINE_RECEIVER_PORT = 2013
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2014
PICKLE_RECEIVER_MAX_LENGTH = 1048576
RELAY_METHOD = consistent-hashing
REPLICATION_FACTOR = 1
DESTINATIONS = 127.0.0.1:2004:a
DESTINATION_PROTOCOL = pickle
DESTINATION_TRANSPORT = none
DYNAMIC_ROUTER = True
MAX_QUEUE_SIZE = 10000
MAX_DATAPOINTS_PER_MESSAGE = 500
USER = carbon
QUEUE_LOW_WATERMARK_PCT = 0.8
TIME_TO_DEFER_SENDING = 0.0001
USE_FLOW_CONTROL = True
USE_RATIO_RESET=False
MIN_RESET_STAT_FLOW=1000
MIN_RESET_RATIO=0.9
MIN_RESET_INTERVAL=121
[aggregator]
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2023
PICKLE_RECEIVER_INTERFACE = 0.0.0.0
PICKLE_RECEIVER_PORT = 2024
FORWARD_ALL = True
DESTINATIONS = 127.0.0.1:2004
REPLICATION_FACTOR = 1
MAX_QUEUE_SIZE = 10000
USE_FLOW_CONTROL = True
MAX_DATAPOINTS_PER_MESSAGE = 500
MAX_AGGREGATION_INTERVALS = 5

Services are running.
Interesting though, that Graphite-Web can’t be reached.
Graphs are displayed though…

# systemctl status carbon-cache@a
● carbon-cache@a.service - carbon-cache instance a (graphite)
   Loaded: loaded (/etc/systemd/system/carbon-cache@.service; indirect; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-03 14:42:45 CET; 2 weeks 0 days ago
  Process: 1351 ExecStart=/opt/graphite/bin/carbon-cache.py --instance=a start --pidfile=/opt/graphite/storage/carb
  Process: 1180 ExecStartPre=/bin/rm -f /opt/graphite/storage/carbon-cache-a.pid (code=exited, status=0/SUCCESS)
 Main PID: 2028 (carbon-cache.py)
    Tasks: 3 (limit: 4594)
   CGroup: /system.slice/system-carbon\x2dcache.slice/carbon-cache@a.service
           └─2028 /opt/graphite/bin/python3 /opt/graphite/bin/carbon-cache.py --instance=a start --pidfile=/opt/gra

Mär 03 14:42:03 SERVER systemd[1]: Starting carbon-cache instance a (graphite)...
Mär 03 14:42:45 SERVER systemd[1]: Started carbon-cache instance a (graphite).


# systemctl status carbon-relay.service
● carbon-relay.service - Graphite Carbon Relay
   Loaded: loaded (/etc/systemd/system/carbon-relay.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-03 14:42:44 CET; 2 weeks 0 days ago
  Process: 989 ExecStart=/opt/graphite/bin/carbon-relay.py --config=/opt/graphite/conf/carbon.conf --pidfile=/var/r
 Main PID: 2021 (carbon-relay.py)
    Tasks: 1 (limit: 4594)
   CGroup: /system.slice/carbon-relay.service
           └─2021 /opt/graphite/bin/python3 /opt/graphite/bin/carbon-relay.py --config=/opt/graphite/conf/carbon.co

Mär 03 14:42:01 SERVER systemd[1]: Starting Graphite Carbon Relay...
Mär 03 14:42:44 SERVER systemd[1]: Started Graphite Carbon Relay.

# systemctl status graphite
● graphite.service - Graphite
   Loaded: loaded (/etc/systemd/system/graphite.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-03 14:42:03 CET; 2 weeks 0 days ago
 Main PID: 1143 (gunicorn)
    Tasks: 6 (limit: 4594)
   CGroup: /system.slice/graphite.service
           ├─1143 /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn --bind 0.0.0.0:8000 graphite.wsgi
           └─1910 /opt/graphite/bin/python3 /opt/graphite/bin/gunicorn --bind 0.0.0.0:8000 graphite.wsgi

Mär 03 14:42:03 SERVER systemd[1]: Started Graphite.
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1143] [INFO] Starting gunicorn 20.0.4
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1143] [INFO] Listening at: http://0.0.0.0
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1143] [INFO] Using worker: sync
Mär 03 14:42:22 SERVER gunicorn[1143]: [2020-03-03 14:42:22 +0100] [1910] [INFO] Booting worker with pid: 191
tagdb.log
09/03/2020 23:58:26 :: Error tagging icinga2.somehostname.services.memory.nwc_health.perfdata.Processor_usage.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:58:32 :: Error tagging icinga2.somehostname.services.load.kemp_loadmaster.perfdata.Load-5.value: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:58:44 :: Error tagging icinga2.somehostname.services.swap.swap-windows.perfdata.swap.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:58:52 :: Error tagging icinga2.somehostname.services.memory.memory-windows.perfdata.memory.max: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:06 :: Error tagging icinga2.somehostname.services.disk-D.disk-windows.perfdata.D:_.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:18 :: Error tagging icinga2.somehostname.host.icmp.perfdata.pl.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:27 :: Error tagging icinga2.somehostname.services.memory.nwc_health.perfdata.MEMPOOL_MSGLYR_2_usage.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:37 :: Error tagging icinga2.somehostname.services.swap.swap-windows.perfdata.swap.value: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:47 :: Error tagging icinga2.somehostname.host.icmp.perfdata.rta.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:53 :: Error tagging icinga2.somehostname.services.volume-usage.netapp_cdot_volume_status.perfdata.Vol_CIFS_CIFS_root.check_cdot_volume_usage.inode_used.warn, icinga2.NetApp_xyz-xyz.services.volume-usage.netapp_cdot_volume_status.perfdata.Vol_FC_FC_data2_SATA.check_cdot_volume_usage.snap_used.crit: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries
09/03/2020 23:59:59 :: Error tagging icinga2.somehostname.host.icmp.perfdata.rta.warn: Error response 500 from http://127.0.0.1:8000/graphite/tags/tagMultiSeries

Consol log seems to only contain messages about two checks, that are monitoring exchange queues:

14/03/2020 22:50:49 :: Error parsing metric icinga2.SERVER-EXCH02-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH02-x_31=0;20;40;0SERVER-EXCH02-x_Submission.warn: Cannot parse path icinga2.SERVER-EXCH02-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH02-x_31=0;20;40;0SERVER-EXCH02-x_Submission.warn, invalid segment 20
14/03/2020 23:11:10 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.value: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.value, invalid segment 20
14/03/2020 23:11:10 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.crit: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.crit, invalid segment 20
14/03/2020 23:11:10 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.warn: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_22616=0;20;40;0SERVER-EXCH01-x_22622=0;20;40;0SERVER-EXCH01-x_22628=0;20;40;0SERVER-EXCH01-x_Submission.warn, invalid segment 20
14/03/2020 23:41:46 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.value: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.value, invalid segment 20
14/03/2020 23:41:46 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.crit: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.crit, invalid segment 20
14/03/2020 23:41:46 :: Error parsing metric icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.warn: Cannot parse path icinga2.SERVER-EXCH01-x.services.Exchange_Queue_Health.powershell_check.perfdata.SERVER-EXCH01-x_23066=0;20;40;0SERVER-EXCH01-x_Submission.warn, invalid segment 20

Did I **** something up during the installation?
Seems like this is going on since the beginning, at least the logs go back that far.

Can I safely delete the _tagged folder? (Before/after fixing the problem ofc ;))

I haven’t messed with Graphite’s tagging feature really. Were you using it for something? For graphing in Icinga it’s probably overkill, but it does require some config on both the carbon and graphite sides.

So if I follow, Icingaweb’s graphite module is successfully pulling the graph data, but trying to access the web portal manually isn’t? What are you seeing when you try to hit the webpage?

Nope, I’m not (willingly) using the tagging feature.
I also don’t see anything related in the conf file or does this feature needs to be configred somewhere else?
Could this get enable by “some kind of strange performance data” :D?

Graphs in Icinga Web 2 are displayed, yes.
If I call IP:8000/graphite I don’t get anything, like if there is nothing there at all

Do you basically have a black headerbar and a white page? That’s because gunicorn isn’t serving the static content. It needs apache or nginx sitting in front of it.

In Graphite’s local_settings.py, is anything uncommented in TagDB Settings?

No, everything commented out.

grep "^[^#;]" /opt/graphite/webapp/graphite/local_settings.py
SECRET_KEY = 'asdf'
TIME_ZONE = 'Europe/Berlin'
DEFAULT_XFILES_FACTOR = 0
URL_PREFIX = '/graphite'
STORAGE_DIR = '/var/lib/graphite/storage'
LOG_DIR = '/var/log/graphite'
CARBONLINK_HOSTS = ["127.0.0.1:7002:a"]
CARBONLINK_TIMEOUT = 1.0
CARBONLINK_RETRY_DELAY = 15 # Seconds to blacklist a failed remote server
REPLICATION_FACTOR = 1
FUNCTION_PLUGINS = []

The file /opt/graphite/webapp/graphite/settings.py has nothing commented out though. Is this file relevant? Or is this ignored/overruled by local_settings.py?

Not even that. Nothing is found.

http://monitoringserver:8000/graphite
image

I also had to input the login credentials in the graphite module, though I have never done this before in other setups.

Do I need any extra steps for apache2 compared to nginx used in the HowTo?

This is going to sound silly, but…

try /graphite/

1 Like

:open_mouth:
That gave me insight to a “nice” error log:

Traceback (most recent call last):
  File "/opt/graphite/webapp/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/graphite/webapp/django/db/backends/sqlite3/base.py", line 383, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: no such table: auth_user

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/graphite/webapp/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/opt/graphite/webapp/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/graphite/webapp/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/graphite/webapp/graphite/composer/views.py", line 27, in composer
    profile = getProfile(request)
  File "/opt/graphite/webapp/graphite/user_util.py", line 35, in getProfile
    return default_profile()
  File "/opt/graphite/webapp/graphite/user_util.py", line 51, in default_profile
    'password': '!'})
  File "/opt/graphite/webapp/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/graphite/webapp/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/opt/graphite/webapp/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/opt/graphite/webapp/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/opt/graphite/webapp/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/graphite/webapp/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/graphite/webapp/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/opt/graphite/webapp/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/graphite/webapp/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/graphite/webapp/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/graphite/webapp/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/graphite/webapp/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/graphite/webapp/django/db/backends/sqlite3/base.py", line 383, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: auth_user

Seems like something didn’t work when creating the db, I think I saw this some time ago when I stated playing around with graphite.

Rerunning

removes the error and displays the graphite web header:

But I don’t think this has anything to do with the tag files…

Yeah I’m not sure why you’re getting tag files unless they defaulted something just recently. I double checked my most up to date production server on 1.1.6 that I just installed a few months ago and I don’t have the _tagged folder.

I’m curious reading more into this though, https://graphite.readthedocs.io/en/latest/tags.html

Are you on Icinga 2.11 yet? I haven’t upgraded. The doc makes it look like the format of the input determines if you have tagged files or not. Can you do icinga2 object list --type GraphiteWriter and see if there’s anything new in there?

Object 'graphite' of type 'GraphiteWriter':
  % declared in '/etc/icinga2/features-enabled/graphite.conf', lines 6:1-6:32
  * __name = "graphite"
  * enable_ha = false
  * enable_send_metadata = false
  * enable_send_thresholds = true
    % = modified in '/etc/icinga2/features-enabled/graphite.conf', lines 9:3-9:31
  * host = "127.0.0.1"
    % = modified in '/etc/icinga2/features-enabled/graphite.conf', lines 7:3-7:20
  * host_name_template = "icinga2.$host.name$.host.$host.check_command$"
  * name = "graphite"
  * package = "_etc"
  * port = "2013"
    % = modified in '/etc/icinga2/features-enabled/graphite.conf', lines 8:3-8:13
  * service_name_template = "icinga2.$host.name$.services.$service.name$.$service.check_command$"
  * source_location
    * first_column = 1
    * first_line = 6
    * last_column = 32
    * last_line = 6
    * path = "/etc/icinga2/features-enabled/graphite.conf"
  * templates = [ "graphite" ]
    % = modified in '/etc/icinga2/features-enabled/graphite.conf', lines 6:1-6:32
  * type = "GraphiteWriter"
  * zone = ""
cat /etc/icinga2/features-enabled/graphite.conf
/**
 * The GraphiteWriter type writes check result metrics and
 * performance data to a graphite tcp socket.
 */

object GraphiteWriter "graphite" {
  host = "127.0.0.1"
  port = 2013
  enable_send_thresholds = true
}

I tracked down the services that produce the _tagged folder. At least in the last 15 min after I stopped those two checks, the folder didn’t appear again.
Strange though, because it is a simple powershell script, running on the windows server, that checks the Exchange queues.
The perfdata label has a \ in it, but that has never been a problem.

Only difference to other setups running that check is that in the problematic setup the check is executed by the Icinga Agent on the Windows host.

I’m more curious about the settings.py under /opt/graphite/webapp/graphite
I haven’t touched this, but it has so many uncommented options.
edit: scratch that, I just saw the line, that the changes should be done in the local_settings.py file
# DO NOT MODIFY THIS FILE DIRECTLY - use local_settings.py instead

Here is the whole history of the carbon user, after installing graphite using your HowTo

carbon@SERVER:~$ history
    1  source bin/activate
    2  pip install --upgrade pip wheel setuptools gunicorn whisper
    3  pip install carbon --install-option="--prefix=/opt/graphite" --install-option="--install-lib=/opt/graphite/lib"
    4  pip install graphite-web --install-option="--prefix=/opt/graphite" --install-option="--install-lib=/opt/graphite/webapp"
    5  cd /opt/graphite/conf
    6  cp carbon.conf.example carbon.conf
    7  vi carbon.conf
    8  vi carbon.conf
    9  cp storage-schemas.conf.example storage-schemas.conf
   10  vi storage-schemas.conf
   11  cp storage-aggregation.conf.example storage-aggregation.conf
   12  vi storage-aggregation.conf
   13  vi /opt/graphite/webapp/graphite/local_settings.py
   14  cd /opt/graphite/webapp/graphite
   15  ll
   16  ls -la
   17  cp local_settings.py.example local_settings.py
   18  vi /opt/graphite/webapp/graphite/local_settings.py
   19  cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/webapp/graphite.wsgi
   20  PYTHONPATH=/opt/graphite/webapp django-admin.py migrate --settings=graphite.settings --run-syncdb
   21  env
   22  exit
1 Like

That is a good find that it be service formatting specific. Reading Graphite’s docs, it seems like I was wrong about any configuration needing to be done on the carbon end for tagging to work; it’s just about what you feed it. That check might just need to be modified. The default Graphite that ships with CentOS7 is 0.9, so you might be the first person to have noticed such a thing.

1 Like

This is a good hint. My other setups that are using graphite use the Ubuntu packages.
Those install v1.0.2, so a pre-tag version.

This problematic system is the first I installed with your HowTo and v1.1.x

Will check the PowerShell script and see if changeing the output can be easily done.

Thanks so far for your help!

I found the problem…

Whatever made me do it (quite some time ago…):
But I replaced all “spaces” in the perfdata output of the script with “nothing”, thus the output of the perfdata was just one big line. And this matches exactly to the tagging feature of the newer graphite versions.

So I just deleted the line that replaces the spaces in the string, boom. It works.

Layer 8 problem.

But thank you for your help @blakehartshorn!