Hi everyone,
I have an icinga2 cluster, and this cluster will ship the monitor status to nagvis.
Because of large topology, i can not using livestatus as backend in nagvis because it took too long to load object and service status. Therefore, i using ido-db as backend for nagvis. I notice that icinga2 lagging to update data to ido-db; I got the log as:
# tail -f /var/log/icinga2/icinga2.log | grep "information/IdoMysqlConnection"
[2023-10-24 12:09:57 +0200] information/IdoMysqlConnection: Pending queries: 10934 (Input: 1095/s; Output: 2/s)
[2023-10-24 12:10:07 +0200] information/IdoMysqlConnection: Pending queries: 130169 (Input: 11916/s; Output: 0/s)
[2023-10-24 12:10:17 +0200] information/IdoMysqlConnection: Pending queries: 280994 (Input: 14778/s; Output: 0/s)
[2023-10-24 12:10:27 +0200] information/IdoMysqlConnection: Pending queries: 435796 (Input: 15474/s; Output: 0/s)
[2023-10-24 12:10:37 +0200] information/IdoMysqlConnection: Pending queries: 576218 (Input: 13742/s; Output: 0/s)
[2023-10-24 12:10:47 +0200] information/IdoMysqlConnection: Pending queries: 745182 (Input: 16571/s; Output: 0/s)
[2023-10-24 12:10:57 +0200] information/IdoMysqlConnection: Pending queries: 912380 (Input: 16412/s; Output: 0/s)
[2023-10-24 12:11:07 +0200] information/IdoMysqlConnection: Pending queries: 1076174 (Input: 16078/s; Output: 0/s)
[2023-10-24 12:11:17 +0200] information/IdoMysqlConnection: Pending queries: 1197123 (Input: 11740/s; Output: 0/s)
[2023-10-24 12:11:27 +0200] information/IdoMysqlConnection: Pending queries: 1359834 (Input: 15901/s; Output: 0/s)
[2023-10-24 12:11:37 +0200] information/IdoMysqlConnection: Pending queries: 1522992 (Input: 15959/s; Output: 0/s)
[2023-10-24 12:11:47 +0200] information/IdoMysqlConnection: Pending queries: 1656494 (Input: 13098/s; Output: 64/s)
[2023-10-24 12:11:57 +0200] information/IdoMysqlConnection: Pending queries: 1659399 (Input: 281/s; Output: 64/s)
[2023-10-24 12:12:07 +0200] information/IdoMysqlConnection: Pending queries: 1661286 (Input: 185/s; Output: 64/s)
[2023-10-24 12:12:19 +0200] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2023-10-24 12:12:27 +0200] information/IdoMysqlConnection: Pending queries: 61826 (Input: 167/s; Output: 3/s)
[2023-10-24 12:12:37 +0200] information/IdoMysqlConnection: Pending queries: 218455 (Input: 15661/s; Output: 1/s)
[2023-10-24 12:12:47 +0200] information/IdoMysqlConnection: Pending queries: 405520 (Input: 18277/s; Output: 0/s)
[2023-10-24 12:12:57 +0200] information/IdoMysqlConnection: Pending queries: 588864 (Input: 17938/s; Output: 0/s)
[2023-10-24 12:13:07 +0200] information/IdoMysqlConnection: Pending queries: 771337 (Input: 18247/s; Output: 0/s)
[2023-10-24 12:13:17 +0200] information/IdoMysqlConnection: Pending queries: 963735 (Input: 18837/s; Output: 0/s)
[2023-10-24 12:13:27 +0200] information/IdoMysqlConnection: Pending queries: 1120830 (Input: 15334/s; Output: 0/s)
[2023-10-24 12:13:37 +0200] information/IdoMysqlConnection: Pending queries: 1294893 (Input: 17401/s; Output: 0/s)
The pending queries increased very fast. But I not really find what is relationship between this pending queries and the lagging update of ido data.
my only solution workaround now is to restart icinga2 service. But, does anyone know how to fix the root cause?