Icinga2 not writing to IDO

Icinga 2
1

I’m running icinga2 in HA setup. After DB server (postgresql) server crash icinga2 daemon is unable to write to the DB.

IDO connection: ido-pgsql
qlConnection, ido-pgsql) items: 2601, rate:  0/s (0/min 0/5min 0/15min); empty in 9 seconds
qlConnection, ido-pgsql) items: 3614, rate:  0/s (0/min 0/5min 0/15min); empty in 35 seconds
qlConnection, ido-pgsql) items: 4843, rate:  0/s (0/min 0/5min 0/15min); empty in 39 seconds
qlConnection, ido-pgsql) items: 5844, rate:  0/s (0/min 0/5min 0/15min); empty in 58 seconds
qlConnection, ido-pgsql) items: 6917, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 4 seconds
qlConnection, ido-pgsql) items: 7584, rate:  0/s (0/min 0/5min 0/15min); empty in 1 minute and 53 seconds
qlConnection, ido-pgsql) items: 7985, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 19 seconds
qlConnection, ido-pgsql) items: 8462, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 57 seconds
qlConnection, ido-pgsql) items: 8929, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 11 seconds
qlConnection, ido-pgsql) items: 9308, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 5 seconds
qlConnection, ido-pgsql) items: 9725, rate:  0/s (0/min 0/5min 0/15min); empty in 3 minutes and 53 seconds
qlConnection, ido-pgsql) items: 10124, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 13 seconds
qlConnection, ido-pgsql) items: 10487, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 48 seconds
qlConnection, ido-pgsql) items: 10834, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 12 seconds
qlConnection, ido-pgsql) items: 11183, rate:  0/s (0/min 0/5min 0/15min); empty in 5 minutes and 20 seconds
qlConnection, ido-pgsql) items: 11464, rate:  0/s (0/min 0/5min 0/15min); empty in 6 minutes and 47 seconds
qlConnection, ido-pgsql) items: 11921, rate:  0/s (0/min 0/5min 0/15min); empty in 4 minutes and 20 seconds
qlConnection, ido-pgsql) items: 12872, rate:  0/s (0/min 0/5min 0/15min); empty in 2 minutes and 15 seconds

The update time seems to be updated on master restart, but no other data (service updates) being written.

icinga2_data=# select status_update_time from icinga_programstatus;
 status_update_time  
---------------------
 2020-07-28 05:01:05

I don’t see any other errors in logs. The web interface says:

Monitoring backend 'backends' is not running.

2

Any hints in icinga.log(s)?

3

The replicated log got somehow corrupted:

[2020-07-28 03:50:19 +0000] information/ApiListener: Replayed 816 messages.
[2020-07-28 03:50:19 +0000] warning/ApiListener: Unexpected end-of-file for cluster log: /var/lib/icinga2/api/log/current
Context:
        (0) Replaying log for Endpoint...

I’ve stopped both master instances, deleted cluster log /var/lib/icinga2/api/log/current and started again masters. After a while icinga2 managed to recover.