Problem with IDO during reloads

lamaral · April 17, 2019, 12:08pm

Hello gents,

Our Icinga2 setup has been struggling quite a lot during reloads.
The IDO queue grows and takes a while to come back to normal.
We are running version r2.10.3-1 with two masters, a few zones with two satellites per zone and the database is running on a separate machine.

icinga2 daemon -C output follows:

[2019-04-15 09:15:10 +0000] information/cli: Icinga application loader (version: r2.10.3-1)
[2019-04-15 09:15:10 +0000] information/cli: Loading configuration file(s).
[2019-04-15 09:15:11 +0000] information/ConfigItem: Committing config item(s).
[2019-04-15 09:15:11 +0000] information/ApiListener: My API identity: XXXX
[2019-04-15 09:15:21 +0000] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 0, rate: 17.6/s (1056/min 1056/5min 1056/15min);
[2019-04-15 09:15:21 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2019-04-15 09:15:21 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2019-04-15 09:15:21 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 172130 Services.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 LivestatusListener.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 7330 Hosts.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 42 Downtimes.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 7 Zones.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 11 Endpoints.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 15 Users.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 66 CheckCommands.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2019-04-15 09:15:29 +0000] information/ConfigItem: Instantiated 1 TimePeriod.
[2019-04-15 09:15:29 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2019-04-15 09:15:29 +0000] information/cli: Finished validating the configuration file(s).

The log output below was retrieved using grep "IdoPgsqlConnection" /var/log/icinga2/icinga2.log:

[2019-04-15 08:33:56 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 4, rate: 1188.67/s (71320/min 391663/5min 1191831/15min);
[2019-04-15 08:34:49 +0000] information/IdoPgsqlConnection: 'ido-pgsql' resumed.
[2019-04-15 08:34:49 +0000] information/IdoPgsqlConnection: PGSQL IDO instance id: 1 (schema version: '1.14.3')
[2019-04-15 08:34:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 8363, rate:  0/s (0/min 0/5min 0/15min); empty in 9 seconds
[2019-04-15 08:35:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 22919, rate:  0/s (0/min 0/5min 0/15min); empty in 15 seconds
[2019-04-15 08:35:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 37518, rate:  0/s (0/min 0/5min 0/15min); empty in 25 seconds
[2019-04-15 08:35:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 51325, rate:  0/s (0/min 0/5min 0/15min); empty in 37 seconds
[2019-04-15 08:35:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 65368, rate:  0/s (0/min 0/5min 0/15min); empty in 46 seconds
[2019-04-15 08:35:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 79425, rate:  0/s (0/min 0/5min 0/15min); empty in 56 seconds
[2019-04-15 08:35:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 91720, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 14 seconds
[2019-04-15 08:36:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 105068, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 18 seconds
[2019-04-15 08:36:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 118518, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 28 seconds
[2019-04-15 08:36:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 131481, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 41 seconds
[2019-04-15 08:36:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 144432, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 51 seconds
[2019-04-15 08:36:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 158023, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 56 seconds
[2019-04-15 08:36:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 171796, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 4 seconds
[2019-04-15 08:37:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 184775, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 22 seconds
[2019-04-15 08:37:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 197787, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 32 seconds
[2019-04-15 08:37:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 211137, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 38 seconds
[2019-04-15 08:37:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 224452, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 48 seconds
[2019-04-15 08:37:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 238699, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 47 seconds
[2019-04-15 08:37:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 251962, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 9 seconds
[2019-04-15 08:38:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 265266, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 19 seconds
[2019-04-15 08:38:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 278578, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 29 seconds
[2019-04-15 08:38:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 291675, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 42 seconds
[2019-04-15 08:38:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 305006, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 48 seconds
[2019-04-15 08:38:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 319555, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 39 seconds
[2019-04-15 08:38:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 332984, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 7 seconds
[2019-04-15 08:39:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 344571, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 57 seconds
[2019-04-15 08:39:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 357704, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 32 seconds
[2019-04-15 08:39:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 372453, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 12 seconds
[2019-04-15 08:39:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 387676, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 14 seconds
[2019-04-15 08:39:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 402269, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 35 seconds
[2019-04-15 08:39:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 416904, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 44 seconds
[2019-04-15 08:40:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 431589, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 53 seconds
[2019-04-15 08:40:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 445680, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 16 seconds
[2019-04-15 08:40:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 459641, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 29 seconds
[2019-04-15 08:40:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 473390, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 44 seconds
[2019-04-15 08:40:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 486213, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 19 seconds
[2019-04-15 08:40:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 499364, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 19 seconds
[2019-04-15 08:41:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 513129, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 12 seconds
[2019-04-15 08:41:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 526160, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 43 seconds
[2019-04-15 08:41:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 539172, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 54 seconds
[2019-04-15 08:41:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 552268, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 1 second
[2019-04-15 08:41:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 565767, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 59 seconds
[2019-04-15 08:41:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 579226, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 10 seconds
[2019-04-15 08:42:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 592441, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 28 seconds
[2019-04-15 08:42:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 605804, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 33 seconds
[2019-04-15 08:42:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 619035, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 47 seconds
[2019-04-15 08:42:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 632580, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 46 seconds
[2019-04-15 08:42:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 646394, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 47 seconds
[2019-04-15 08:42:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 659533, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 21 seconds
[2019-04-15 08:43:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 672922, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 22 seconds
[2019-04-15 08:43:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 686339, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 31 seconds
[2019-04-15 08:43:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 699614, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 46 seconds
[2019-04-15 08:43:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 713641, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 28 seconds
[2019-04-15 08:43:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 727837, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 32 seconds
[2019-04-15 08:43:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 740820, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 30 seconds
[2019-04-15 08:44:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 752683, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 34 seconds
[2019-04-15 08:44:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 766376, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 19 seconds
[2019-04-15 08:44:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 781381, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 40 seconds
[2019-04-15 08:44:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 796548, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 45 seconds
[2019-04-15 08:44:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 811155, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 15 seconds
[2019-04-15 08:44:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 825778, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 24 seconds
[2019-04-15 08:45:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 840183, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 43 seconds
[2019-04-15 08:45:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 854356, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 2 seconds
[2019-04-15 08:45:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 868412, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 17 seconds
[2019-04-15 08:45:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 881474, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 14 seconds
[2019-04-15 08:45:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 894447, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 29 seconds
[2019-04-15 08:45:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 907896, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 15 seconds
[2019-04-15 08:46:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 921171, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 33 seconds
[2019-04-15 08:46:14 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 934282, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 52 seconds
[2019-04-15 08:46:24 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 947803, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 40 seconds
[2019-04-15 08:46:34 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 960961, rate:  0/s (0/min 0/5min 0/15min); empty in 12 minutes and 10 seconds
[2019-04-15 08:46:44 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 974164, rate:  0/s (0/min 0/5min 0/15min); empty in 12 minutes and 17 seconds
[2019-04-15 08:46:54 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 987763, rate:  0/s (0/min 0/5min 0/15min); empty in 12 minutes and 6 seconds
[2019-04-15 08:47:04 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in 13 minutes and 36 seconds
[2019-04-15 08:47:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:47:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:47:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:47:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:47:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:48:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:48:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:48:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:48:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:48:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:48:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:49:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:49:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:49:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:49:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:49:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:49:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:50:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:50:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:50:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:50:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:50:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:50:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:51:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:51:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:51:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:51:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:51:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:51:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:52:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:52:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:52:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:52:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:52:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:52:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:53:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:53:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:53:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:53:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:53:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:53:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:54:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:54:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:54:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:54:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:54:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:54:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:55:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 08:55:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate:  0/s (0/min 0/5min 0/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:01:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 999997, rate: 0.516667/s (31/min 31/5min 31/15min); empty in less than 1 millisecond
[2019-04-15 09:01:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 267.183/s (16031/min 16031/5min 16031/15min); empty in 38 days, 13 hours, 36 minutes and 47 seconds
[2019-04-15 09:01:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 999999, rate: 539.783/s (32387/min 32387/5min 32387/15min); empty in less than 1 millisecond
[2019-04-15 09:01:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 999998, rate: 797.517/s (47851/min 47851/5min 47851/15min); empty in less than 1 millisecond
[2019-04-15 09:01:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1049.35/s (62961/min 62961/5min 62961/15min); empty in 57 days, 20 hours, 21 minutes and 20 seconds
[2019-04-15 09:02:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1280.78/s (76847/min 76847/5min 76847/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:02:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1494.35/s (89661/min 90552/5min 90552/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:02:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1423.33/s (85400/min 102779/5min 102779/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:02:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1324.57/s (79474/min 113301/5min 113301/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:02:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1238/s (74280/min 123403/5min 123403/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:02:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1202.43/s (72146/min 136255/5min 136255/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:03:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1189.07/s (71344/min 149427/5min 149427/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:03:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1141.13/s (68468/min 160379/5min 160379/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:03:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 999999, rate: 1147.02/s (68821/min 172320/5min 172320/15min); empty in less than 1 millisecond
[2019-04-15 09:03:37 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1135.2/s (68112/min 182349/5min 182349/15min); empty in 115 days, 17 hours, 16 minutes and 38 seconds
[2019-04-15 09:03:47 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 1036.25/s (62175/min 186566/5min 186566/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:03:57 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 919.3/s (55158/min 192477/5min 192477/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:04:07 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 824.6/s (49476/min 199993/5min 199993/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:04:17 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 1000000, rate: 800.85/s (48051/min 209588/5min 209588/15min); empty in infinite time, your task handler isn't able to keep up
[2019-04-15 09:04:27 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 999997, rate: 839.167/s (50350/min 223819/5min 223819/15min); empty in less than 1 millisecond

The load on the masters and on the DB server are fine and there is no iowait.

Is there something that could be tweaked to improve this problem?

dnsmichi · April 17, 2019, 1:50pm

130k services and PostgreSQL is something we’ve never tested nor optimized. I’d assume that MySQL is faster. Nevertheless, which hardware is in the background?

lamaral · April 17, 2019, 2:21pm

Masters are running on VMs with 8 vCPUs and 16GB of RAM each, while the database is running with 8 vCPUs and 8GB of RAM.

Each machine in a different hypervisor, all M620s with SSD disks.

I’ll give it a try with MySQL.

lamaral · April 17, 2019, 4:25pm

I have got a similar behavior with MySQL.

[2019-04-17 16:10:37 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2019-04-17 16:10:37 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2019-04-17 16:10:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 21165, rate:  0/s (0/min 0/5min 0/15min); empty in 9 seconds
[2019-04-17 16:10:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 67588, rate:  0/s (0/min 0/5min 0/15min); empty in 14 seconds
[2019-04-17 16:11:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 115946, rate:  0/s (0/min 0/5min 0/15min); empty in 23 seconds
[2019-04-17 16:11:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 163167, rate:  0/s (0/min 0/5min 0/15min); empty in 34 seconds
[2019-04-17 16:11:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 209556, rate:  0/s (0/min 0/5min 0/15min); empty in 45 seconds
[2019-04-17 16:11:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 253077, rate:  0/s (0/min 0/5min 0/15min); empty in 58 seconds
[2019-04-17 16:11:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 295933, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 9 seconds
[2019-04-17 16:11:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 341752, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 14 seconds
[2019-04-17 16:12:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 387808, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 24 seconds
[2019-04-17 16:12:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 435524, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 31 seconds
[2019-04-17 16:12:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 483027, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 42 seconds
[2019-04-17 16:12:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 526541, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes
[2019-04-17 16:12:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 571074, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 8 seconds
[2019-04-17 16:12:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 613533, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 24 seconds
[2019-04-17 16:13:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 697832, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 44 seconds
[2019-04-17 16:13:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 740047, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 55 seconds
[2019-04-17 16:13:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 781101, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 10 seconds
[2019-04-17 16:13:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 822195, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 19 seconds
[2019-04-17 16:14:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 901594, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 56 seconds
[2019-04-17 16:14:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 976345, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 20 seconds
[2019-04-17 16:14:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1011803, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 45 seconds
[2019-04-17 16:14:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1047398, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 54 seconds
[2019-04-17 16:14:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1081282, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 19 seconds
[2019-04-17 16:15:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1112507, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 56 seconds
[2019-04-17 16:15:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1142119, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 25 seconds
[2019-04-17 16:15:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1171335, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 40 seconds
[2019-04-17 16:15:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1197658, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 34 seconds
[2019-04-17 16:15:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1223024, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 2 seconds
[2019-04-17 16:15:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1252636, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 3 seconds
[2019-04-17 16:16:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1292069, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 27 seconds
[2019-04-17 16:16:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1333606, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 20 seconds
[2019-04-17 16:16:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1376671, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 19 seconds
[2019-04-17 16:16:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1419305, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 32 seconds
[2019-04-17 16:16:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1507765, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 37 seconds
[2019-04-17 16:17:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1553141, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 42 seconds
[2019-04-17 16:17:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1598891, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 49 seconds
[2019-04-17 16:17:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1644279, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 2 seconds
[2019-04-17 16:17:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1689154, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 16 seconds
[2019-04-17 16:17:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1733623, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 29 seconds
[2019-04-17 16:17:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1776316, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 56 seconds
[2019-04-17 16:18:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1817225, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 24 seconds
[2019-04-17 16:18:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1859916, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 15 seconds
[2019-04-17 16:18:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1905387, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 59 seconds
[2019-04-17 16:18:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1950068, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 16 seconds
[2019-04-17 16:18:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 1992972, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 44 seconds
[2019-04-17 16:18:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2035462, rate:  0/s (0/min 0/5min 0/15min); empty in 7 minutes and 59 seconds
[2019-04-17 16:19:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2077799, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 10 seconds
[2019-04-17 16:19:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2118021, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 46 seconds
[2019-04-17 16:19:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2158875, rate:  0/s (0/min 0/5min 0/15min); empty in 8 minutes and 48 seconds
[2019-04-17 16:19:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2198743, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 11 seconds
[2019-04-17 16:19:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2238544, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 22 seconds
[2019-04-17 16:19:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2277960, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 37 seconds
[2019-04-17 16:20:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2317093, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 52 seconds
[2019-04-17 16:20:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2355748, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 9 seconds
[2019-04-17 16:20:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2395024, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 9 seconds
[2019-04-17 16:20:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2435910, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 55 seconds
[2019-04-17 16:20:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2477928, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 49 seconds
[2019-04-17 16:20:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2520642, rate:  0/s (0/min 0/5min 0/15min); empty in 9 minutes and 50 seconds
[2019-04-17 16:21:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2563172, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 2 seconds
[2019-04-17 16:21:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2605791, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 11 seconds
[2019-04-17 16:21:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2649686, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 3 seconds
[2019-04-17 16:21:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2692182, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 33 seconds
[2019-04-17 16:21:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2736760, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 13 seconds
[2019-04-17 16:21:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2782711, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 5 seconds
[2019-04-17 16:22:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2828820, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 13 seconds
[2019-04-17 16:22:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2875432, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 16 seconds
[2019-04-17 16:22:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2922098, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 26 seconds
[2019-04-17 16:22:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 2967781, rate:  0/s (0/min 0/5min 0/15min); empty in 10 minutes and 49 seconds
[2019-04-17 16:22:42 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 3013282, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 2 seconds
[2019-04-17 16:22:52 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 3057787, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 27 seconds
[2019-04-17 16:23:02 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 3100067, rate:  0/s (0/min 0/5min 0/15min); empty in 12 minutes and 13 seconds
[2019-04-17 16:23:12 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 3143617, rate:  0/s (0/min 0/5min 0/15min); empty in 12 minutes and 1 second
[2019-04-17 16:23:22 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 3188618, rate:  0/s (0/min 0/5min 0/15min); empty in 11 minutes and 48 seconds
[2019-04-17 16:23:32 +0000] information/WorkQueue: #7 (IdoMysqlConnection, ido-mysql) items: 3201440, rate: 420.95/s (25257/min 25257/5min 25257/15min); empty in 41 minutes and 36 seconds

tredel · April 18, 2019, 3:28am

Hi Luiz,

I setup a monitoring for a customer a few months ago with about 155k services running on r2.10.1. The highest number of queued items I’ve ever seen on this was about 250.000.

Seeing your WorkQueue constantly growing with a rate of 0/s is a bit “suspicious”. Seems like IDO is paused on this node.

Could you please provide this log during a reload from both of your master nodes? Seems like the other master node is actively writing to the database during the timeframe of the log.

Please also have a look at the CPU usage and the iowait of both Icinga 2 masters and the database servers during the reload. As I needed a much bigger server sizing to run smooth for my customer setup it’s still possible that you have a performance bottleneck on your servers.

lamaral · April 18, 2019, 2:40pm

In order to rule out this problem of the two masters trying to write at the same time, I have removed one of them for now.

I increased CPUs to 12 also.

I actually forgot to mention on first post that I am pushing my configuration through API using a config package. All config files are being pushed at once, to the same package.

After some more debugging and doing performance analysis, I found out using mpstat -P ALL 1 and pidstat 1 that there is one Icinga2 process using a single CPU 100%. As soon as the CPU “cools down”, the WorkQueue starts writing to database.

pidstat 1 output:

Average:      UID       PID    %usr %system  %guest    %CPU   CPU  Command
Average:      107     24902  100.00   16.79    0.00  100.00     -  icinga2

mpstat -P ALL 1 output:

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
Average:     all   35.42    0.00    2.84    0.09    0.00    0.13    0.02    0.00    0.00   61.50
Average:       0   33.94    0.00    3.77    0.99    0.00    1.53    0.04    0.00    0.00   59.73
Average:       1   38.00    0.00    3.19    0.00    0.00    0.01    0.02    0.00    0.00   58.78
Average:       2   44.60    0.00    2.97    0.00    0.00    0.00    0.04    0.00    0.00   52.40
Average:       3   28.08    0.00    5.01    0.00    0.00    0.00    0.01    0.00    0.00   66.90
Average:       4   32.18    0.01    2.21    0.04    0.00    0.00    0.01    0.00    0.00   65.55
Average:       5   30.19    0.00    1.73    0.00    0.00    0.00    0.01    0.00    0.00   68.07
Average:       6   32.00    0.00    2.27    0.00    0.00    0.01    0.03    0.00    0.00   65.69
Average:       7   61.91    0.00    3.60    0.00    0.00    0.00    0.01    0.00    0.00   34.48
Average:       8   30.38    0.00    2.92    0.00    0.00    0.00    0.01    0.00    0.00   66.69
Average:       9   34.79    0.00    2.55    0.00    0.00    0.00    0.01    0.00    0.00   62.65
Average:      10   30.44    0.00    2.16    0.03    0.00    0.00    0.01    0.00    0.00   67.36
Average:      11   28.64    0.00    1.73    0.00    0.00    0.00    0.01    0.00    0.00   69.62

In the instantaneous measurements, the Icinga2 process was showing up on CPU 7.

These are graphs for the database and master servers: