Icinga2 HA master setup does not failover nor write to IDO

Icinga 2
1

Hello,
currently I do have the situation that with a Icinga2 (v. 2.13.2-1) HA master setup, the master does not correctly failover to the second master (here: sa01) if the first master (here: ma01) will get unresponsive (e.g. icinga2 service stopped).

  • The second master does not write to the IDO, the first does.
  • The first master is setup as a master, the second one as a satellite (as the documentation suggests)
  • Both nodes have the same features enabled.
  • PSQL is configured on both nodes the same way

Here is some information about the setup:

zones.conf on ma01 (MASTER)

object Endpoint "otn-ac-monq-ma01.localdomain" {
  // That's us
}

object Endpoint "otn-ac-monq-sa01.localdomain" {
  host = "otn-ac-monq-sa01.localdomain" // Actively connect to the secondary master
}

object Zone "master" {
  endpoints = [ "otn-ac-monq-ma01.localdomain", "otn-ac-monq-sa01.localdomain" ]
}

zones.conf on sa01 (SATELLITE) → should take over for ma01 in case of outage

object Endpoint "otn-ac-monq-ma01.localdomain" {
// The first master already connects to us
}

object Zone "master" {
        endpoints = [ "otn-ac-monq-ma01.localdomain", "otn-ac-monq-sa01.localdomain" ]
}

object Endpoint "otn-ac-monq-sa01.localdomain" {
// That's us
}

debug.log of sa01 (SATELLITE) when icinga2 service of ma01 got shutdown at 14:54:47


[2022-02-14 14:54:29 +0100] notice/ApiListener: Relaying 'event::SetLastCheckStarted' message
[2022-02-14 14:54:29 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:54:29 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:54:29 +0100] notice/Process: Running command '/usr/lib64/nagios/plugins/check_ping' '-4' '-H' 'otn-ac-monq-sa02.localdomain' '-c' '200,15%' '-w' '100,5%': PID 19910
[2022-02-14 14:54:30 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0.0333333
[2022-02-14 14:54:30 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/14 14:54:29
[2022-02-14 14:54:30 +0100] notice/JsonRpcConnection: Received 'event::SetLastCheckStarted' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:30 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:30 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:30 +0100] notice/JsonRpcConnection: Received 'event::CheckResult' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:32 +0100] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2022-02-14 14:54:33 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:33 +0100] notice/JsonRpcConnection: Received 'event::Heartbeat' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:33 +0100] notice/Process: PID 19910 ('/usr/lib64/nagios/plugins/check_ping' '-4' '-H' 'otn-ac-monq-sa02.localdomain' '-c' '200,15%' '-w' '100,5%') terminated with exit code 0
[2022-02-14 14:54:33 +0100] notice/ApiListener: Sending message 'event::CheckResult' to 'otn-ac-monq-ma01.localdomain'
[2022-02-14 14:54:33 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:33 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:54:33 +0100] notice/JsonRpcConnection: Received 'event::CheckResult' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:33 +0100] debug/Checkable: Update checkable 'otn-ac-monq-sa02.localdomain!ping4onZone' with check interval '20' from last check time at 2022-02-14 14:54:33 +0100 (1.64485e+09) to next check time at 2022-02-14 14:54:53 +0100 (1.64485e+09).
[2022-02-14 14:54:33 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:54:33 +0100] notice/ApiListener: Relaying 'event::CheckResult' message
[2022-02-14 14:54:35 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa01.localdomain'.
[2022-02-14 14:54:35 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-14 14:54:35 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa01.localdomain' because that's us.
[2022-02-14 14:54:35 +0100] notice/ApiListener: Current zone master: otn-ac-monq-ma01.localdomain
[2022-02-14 14:54:35 +0100] notice/ApiListener: Connected endpoints: otn-ac-monq-ma01.localdomain (1)
[2022-02-14 14:54:35 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/14 14:54:33
[2022-02-14 14:54:35 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0.05
[2022-02-14 14:54:38 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:40 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0.05
[2022-02-14 14:54:40 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/14 14:54:33
[2022-02-14 14:54:43 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:45 +0100] notice/JsonRpcConnection: Received 'event::SetLastCheckStarted' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:45 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:45 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:45 +0100] notice/JsonRpcConnection: Received 'event::CheckResult' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-14 14:54:45 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-14 14:54:45 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa01.localdomain'.
[2022-02-14 14:54:45 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0.05
[2022-02-14 14:54:45 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/14 14:54:45
[2022-02-14 14:54:45 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa01.localdomain' because that's us.
[2022-02-14 14:54:45 +0100] notice/ApiListener: Current zone master: otn-ac-monq-ma01.localdomain
[2022-02-14 14:54:45 +0100] notice/ApiListener: Connected endpoints: otn-ac-monq-ma01.localdomain (1)
[2022-02-14 14:54:47 +0100] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'otn-ac-monq-ma01.localdomain': Error: short read

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000AFA8B7 in /usr/lib64/icinga2/sbin/icinga2
 5# 0x0000000000B06A89 in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/libboost_context.so.1.69.0
[2022-02-14 14:54:47 +0100] warning/JsonRpcConnection: API client disconnected for identity 'otn-ac-monq-ma01.localdomain'
[2022-02-14 14:54:47 +0100] warning/ApiListener: Removing API client for endpoint 'otn-ac-monq-ma01.localdomain'. 0 API clients left.
[2022-02-14 14:54:47 +0100] debug/EndpointDbObject: update is_connected=0 for endpoint 'otn-ac-monq-ma01.localdomain'
[2022-02-14 14:54:50 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0.0333333
[2022-02-14 14:54:55 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-14 14:54:55 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa01.localdomain'.
[2022-02-14 14:54:55 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0.0333333
[2022-02-14 14:54:55 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa01.localdomain' because that's us.
[2022-02-14 14:54:55 +0100] notice/ApiListener: Current zone master: otn-ac-monq-sa01.localdomain
[2022-02-14 14:54:55 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-14 14:54:55 +0100] debug/CheckerComponent: Scheduling info for checkable 'otn-ac-monq-sa02.localdomain!ping4onZone' (2022-02-14 14:54:53 +0100): Object 'otn-ac-monq-sa02.localdomain!ping4onZone', Next Check: 2022-02-14 14:54:53 +0100(1.64485e+09).
[2022-02-14 14:54:55 +0100] debug/CheckerComponent: Executing check for 'otn-ac-monq-sa02.localdomain!ping4onZone'
[2022-02-14 14:54:55 +0100] debug/Checkable: Update checkable 'otn-ac-monq-sa02.localdomain!ping4onZone' with check interval '20' from last check time at 2022-02-14 14:54:33 +0100 (1.64485e+09) to next check time at 2022-02-14 14:55:15 +0100 (1.64485e+09).
[2022-02-14 14:54:55 +0100] notice/ApiListener: Relaying 'event::SetLastCheckStarted' message
[2022-02-14 14:54:55 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:54:55 +0100] notice/Process: Running command '/usr/lib64/nagios/plugins/check_ping' '-4' '-H' 'otn-ac-monq-sa02.localdomain' '-c' '200,15%' '-w' '100,5%': PID 19912
[2022-02-14 14:54:55 +0100] debug/CheckerComponent: Check finished for object 'otn-ac-monq-sa02.localdomain!ping4onZone'
[2022-02-14 14:54:59 +0100] notice/Process: PID 19912 ('/usr/lib64/nagios/plugins/check_ping' '-4' '-H' 'otn-ac-monq-sa02.localdomain' '-c' '200,15%' '-w' '100,5%') terminated with exit code 0
[2022-02-14 14:54:59 +0100] debug/Checkable: Update checkable 'otn-ac-monq-sa02.localdomain!ping4onZone' with check interval '20' from last check time at 2022-02-14 14:54:59 +0100 (1.64485e+09) to next check time at 2022-02-14 14:55:19 +0100 (1.64485e+09).
[2022-02-14 14:54:59 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:54:59 +0100] notice/ApiListener: Relaying 'event::CheckResult' message
[2022-02-14 14:55:00 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 2; Checks/s: 0.05
[2022-02-14 14:55:02 +0100] debug/CheckerComponent: Scheduling info for checkable 'otn-ac-monq-sa02.localdomain' (2022-02-14 14:55:02 +0100): Object 'otn-ac-monq-sa02.localdomain', Next Check: 2022-02-14 14:55:02 +0100(1.64485e+09).
[2022-02-14 14:55:02 +0100] debug/CheckerComponent: Executing check for 'otn-ac-monq-sa02.localdomain'
[2022-02-14 14:55:02 +0100] debug/Checkable: Update checkable 'otn-ac-monq-sa02.localdomain' with check interval '300' from last check time at 2022-02-14 14:50:06 +0100 (1.64485e+09) to next check time at 2022-02-14 15:00:02 +0100 (1.64485e+09).
[2022-02-14 14:55:02 +0100] notice/ApiListener: Relaying 'event::SetLastCheckStarted' message
[2022-02-14 14:55:02 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2022-02-14 14:55:02 +0100] notice/Process: Running command '/usr/lib64/nagios/plugins/check_ping' '-H' 'otn-ac-monq-sa02.localdomain' '-c' '5000,100%' '-w' '3000,80%': PID 19914
[2022-02-14 14:55:02 +0100] debug/CheckerComponent: Check finished for object 'otn-ac-monq-sa02.localdomain'
2

Hello @lobr!

Please share your ido-pgsql.conf from both nodes.

Best,
AK

3

@Al2Klimov

object IdoPgsqlConnection "ido-pgsql" {
  user = "icinga"
  password = "12345"
  host = "otn-ac-monq-dvip.localdomain"
  database = "icinga"
  enable_ha=true
}
4

And it’s the same on both nodes?

5

@Al2Klimov
Yes, it’s on both nodes.

6

Please share those logs.

7

Especially “IDO” stuff.

8

I managed the second master write to the database by reinstalling the whole setup for the second master.

One strange thing still happens. When master1 (ma01) and master2 (sa02) run in HA everything is fine. Usually master1 writes to the IDO. But when master1 is turned off, master2 takes over the checks, but deletes all host objects from the database.

Debug.log of master2 (sa02)

[2022-02-18 16:51:41 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:26
[2022-02-18 16:51:41 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:51:41 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:51:41 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:51:41 +0100] notice/ApiListener: Current zone master: otn-ac-monq-ma01.localdomain
[2022-02-18 16:51:41 +0100] notice/ApiListener: Connected endpoints: otn-ac-monq-ma01.localdomain (1)
[2022-02-18 16:51:46 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:51:46 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:26
[2022-02-18 16:51:49 +0100] notice/JsonRpcConnection: Received 'event::SetLastCheckStarted' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-18 16:51:49 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-18 16:51:51 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:51:51 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:51:51 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:49
[2022-02-18 16:51:51 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:51:51 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:51:51 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:51:51 +0100] notice/ApiListener: Current zone master: otn-ac-monq-ma01.localdomain
[2022-02-18 16:51:51 +0100] notice/ApiListener: Connected endpoints: otn-ac-monq-ma01.localdomain (1)
[2022-02-18 16:51:53 +0100] notice/JsonRpcConnection: Received 'event::SetNextCheck' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-18 16:51:53 +0100] notice/JsonRpcConnection: Received 'event::CheckResult' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-18 16:51:56 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:51:56 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:53
[2022-02-18 16:51:58 +0100] notice/JsonRpcConnection: Received 'event::Heartbeat' message from identity 'otn-ac-monq-ma01.localdomain'.
[2022-02-18 16:52:01 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:01 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:01 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:53
[2022-02-18 16:52:01 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:52:01 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:52:01 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:52:01 +0100] notice/ApiListener: Current zone master: otn-ac-monq-ma01.localdomain
[2022-02-18 16:52:01 +0100] notice/ApiListener: Connected endpoints: otn-ac-monq-ma01.localdomain (1)
[2022-02-18 16:52:06 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:06 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:53
[2022-02-18 16:52:11 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:11 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:11 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:53
[2022-02-18 16:52:11 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:52:11 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:52:11 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:52:11 +0100] notice/ApiListener: Current zone master: otn-ac-monq-ma01.localdomain
[2022-02-18 16:52:11 +0100] notice/ApiListener: Connected endpoints: otn-ac-monq-ma01.localdomain (1)
[2022-02-18 16:52:16 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:16 +0100] notice/ApiListener: Setting log position for identity 'otn-ac-monq-ma01.localdomain': 2022/02/18 16:51:53
[2022-02-18 16:52:17 +0100] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'otn-ac-monq-ma01.localdomain': Error: short read

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# icinga::NetString::ReadStringFromStream(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x0000000000AFA8B7 in /usr/lib64/icinga2/sbin/icinga2
 5# 0x0000000000B06A89 in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/libboost_context.so.1.69.0
[2022-02-18 16:52:17 +0100] warning/JsonRpcConnection: API client disconnected for identity 'otn-ac-monq-ma01.localdomain'
[2022-02-18 16:52:17 +0100] warning/ApiListener: Removing API client for endpoint 'otn-ac-monq-ma01.localdomain'. 0 API clients left.
[2022-02-18 16:52:17 +0100] debug/EndpointDbObject: update is_connected=0 for endpoint 'otn-ac-monq-ma01.localdomain'
[2022-02-18 16:52:21 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:21 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:21 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:52:21 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:52:21 +0100] notice/ApiListener: Current zone master: otn-ac-monq-sa02.localdomain
[2022-02-18 16:52:21 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-18 16:52:21 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:52:21 +0100] information/IdoPgsqlConnection: 'ido-pgsql' resumed.
[2022-02-18 16:52:21 +0100] information/DbConnection: Resuming IDO connection: ido-pgsql
[2022-02-18 16:52:21 +0100] debug/IdoPgsqlConnection: Query: SELECT version FROM icinga_dbversion WHERE name='idoutils'
[2022-02-18 16:52:21 +0100] debug/IdoPgsqlConnection: Query: SELECT instance_id FROM icinga_instances WHERE instance_name = 'default'
[2022-02-18 16:52:21 +0100] debug/IdoPgsqlConnection: Query: SELECT UNIX_TIMESTAMP(status_update_time) AS status_update_time, endpoint_name FROM icinga_programstatus WHERE instance_id = 1
[2022-02-18 16:52:21 +0100] information/IdoPgsqlConnection: Last update by endpoint 'otn-ac-monq-ma01.localdomain' was 13.1428s ago (< failover timeout of 30s). Retrying.
[2022-02-18 16:52:26 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:31 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:31 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:31 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:52:31 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:52:31 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:52:31 +0100] notice/ApiListener: Current zone master: otn-ac-monq-sa02.localdomain
[2022-02-18 16:52:31 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-18 16:52:31 +0100] debug/IdoPgsqlConnection: Query: SELECT version FROM icinga_dbversion WHERE name='idoutils'
[2022-02-18 16:52:31 +0100] debug/IdoPgsqlConnection: Query: SELECT instance_id FROM icinga_instances WHERE instance_name = 'default'
[2022-02-18 16:52:31 +0100] debug/IdoPgsqlConnection: Query: SELECT UNIX_TIMESTAMP(status_update_time) AS status_update_time, endpoint_name FROM icinga_programstatus WHERE instance_id = 1
[2022-02-18 16:52:31 +0100] information/IdoPgsqlConnection: Last update by endpoint 'otn-ac-monq-ma01.localdomain' was 23.1433s ago (< failover timeout of 30s). Retrying.
[2022-02-18 16:52:36 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:41 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:41 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:41 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:52:41 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:52:41 +0100] notice/ApiListener: Current zone master: otn-ac-monq-sa02.localdomain
[2022-02-18 16:52:41 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-18 16:52:41 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT version FROM icinga_dbversion WHERE name='idoutils'
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT instance_id FROM icinga_instances WHERE instance_name = 'default'
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT UNIX_TIMESTAMP(status_update_time) AS status_update_time, endpoint_name FROM icinga_programstatus WHERE instance_id = 1
[2022-02-18 16:52:41 +0100] information/IdoPgsqlConnection: Last update by endpoint 'otn-ac-monq-ma01.localdomain' was 33.1426s ago. Taking over 'ido-pgsql' in HA zone 'master'.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Enabling IDO connection.
[2022-02-18 16:52:41 +0100] information/IdoPgsqlConnection: PGSQL IDO instance id: 1 (schema version: '1.14.3')
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:41 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: DELETE FROM icinga_programstatus WHERE instance_id = 1
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES ('1', '1', '1', 'otn-ac-monq-sa02.localdomain', '1', '1', 1, '1', TO_TIMESTAMP(1645199561) AT TIME ZONE 'UTC', '1', '1', '1', '2794', '1', TO_TIMESTAMP(1645199361) AT TIME ZONE 'UTC', '2.13.2-1', TO_TIMESTAMP(1645199561) AT TIME ZONE 'UTC')
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '0')
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '0')
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '0')
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '0')
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_conninfo (instance_id, connect_time, last_checkin_time, agent_name, agent_version, connect_type, data_start_time) VALUES (1, NOW(), NOW(), 'icinga2 db_ido_pgsql', '2.13.2-1', 'INITIAL', NOW())
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT zone_object_id AS object_id, zone_id, config_hash FROM icinga_zones
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT host_object_id AS object_id, host_id, config_hash FROM icinga_hosts
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT contactgroup_object_id AS object_id, contactgroup_id, config_hash FROM icinga_contactgroups
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT endpoint_object_id AS object_id, endpoint_id, config_hash FROM icinga_endpoints
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT contact_object_id AS object_id, contact_id, config_hash FROM icinga_contacts
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT hostgroup_object_id AS object_id, hostgroup_id, config_hash FROM icinga_hostgroups
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT service_object_id AS object_id, service_id, config_hash FROM icinga_services
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT object_id AS object_id, command_id, config_hash FROM icinga_commands
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT timeperiod_object_id AS object_id, timeperiod_id, config_hash FROM icinga_timeperiods
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT servicegroup_object_id AS object_id, servicegroup_id, config_hash FROM icinga_servicegroups
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: SELECT object_id, objecttype_id, name1, name2, is_active FROM icinga_objects WHERE instance_id = 1
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'otnCloudPetW45 - Bruno L [AC.OPS]' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'otnCloudPetW45 - Bruno L [AC.OPS]' name2: 'ping4'.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'otnCloudPetW45 - Bruno L [AC.OPS]' name2: 'ping'.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'otnCloudPetW45 - Bruno L [AC.OPS]2' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'linux-servers' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'windows-servers' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'notification_mail-host-notification' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'notification_mail-service-notification' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'otnCloudPetW45 - Bruno L [AC.OPS]2' name2: 'ping'.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'otnCloudPetW45 - Bruno L [AC.OPS]2' name2: 'ping4'.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'ping' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'http' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'disk' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'never' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: '9to5' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: '24x7' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'icingaadmin' name2: ''.
[2022-02-18 16:52:41 +0100] notice/IdoPgsqlConnection: Deactivate deleted object name1: 'icingaadmins' name2: ''.
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_endpoints SET config_hash = '61e197131b567c088a10a0811c4db7b9b59e4f6bd5bb2d9c5911cb9031c03cff',  config_type = '1',  endpoint_object_id = 289,  identity = 'otn-ac-monq-sa02.localdomain',  instance_id = 1,  node = 'otn-ac-monq-sa02.localdomain',  zone_object_id = 288 WHERE endpoint_object_id = 289
[2022-02-18 16:52:41 +0100] debug/EndpointDbObject: update status for endpoint 'otn-ac-monq-sa02.localdomain'
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_endpointstatus SET endpoint_object_id = 289,  identity = 'otn-ac-monq-sa02.localdomain',  instance_id = 1,  is_connected = '1',  node = 'otn-ac-monq-sa02.localdomain',  status_update_time = TO_TIMESTAMP(1645199561) AT TIME ZONE 'UTC',  zone_object_id = 288 WHERE endpoint_object_id = 289
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_endpoints SET config_hash = '648b2db26cd94e4cc3e90ded4151165d2fa4391159672f2337cf0a033efe79e7',  config_type = '1',  endpoint_object_id = 246,  identity = 'otn-ac-monq-ma01.localdomain',  instance_id = 1,  node = 'otn-ac-monq-sa02.localdomain',  zone_object_id = 288 WHERE endpoint_object_id = 246
[2022-02-18 16:52:41 +0100] debug/EndpointDbObject: update status for endpoint 'otn-ac-monq-ma01.localdomain'
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_endpointstatus SET endpoint_object_id = 246,  identity = 'otn-ac-monq-ma01.localdomain',  instance_id = 1,  is_connected = '0',  node = 'otn-ac-monq-sa02.localdomain',  status_update_time = TO_TIMESTAMP(1645199561) AT TIME ZONE 'UTC',  zone_object_id = 288 WHERE endpoint_object_id = 246
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 291
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 292
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 293
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 294
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 252
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 251
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 254
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 253
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 296
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 295
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 266
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 267
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 268
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 269
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 270
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 271
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 249
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: UPDATE icinga_objects SET is_active = 0 WHERE object_id = 250
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: DELETE FROM icinga_comments WHERE instance_id = 1 AND session_token <> 1645199361
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: DELETE FROM icinga_scheduleddowntime WHERE instance_id = 1 AND session_token <> 1645199361
[2022-02-18 16:52:41 +0100] information/IdoPgsqlConnection: Finished reconnecting to 'ido-pgsql' database 'icinga' in 0.0298412 second(s).
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:41 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:42 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:42 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:43 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:43 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:44 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:44 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:45 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:45 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:46 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:46 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:46 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:47 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:47 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:48 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:48 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:49 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:49 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:50 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:50 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:51 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: DELETE FROM icinga_programstatus WHERE instance_id = 1
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES ('1', '1', '1', 'otn-ac-monq-sa02.localdomain', '1', '1', 1, '1', TO_TIMESTAMP(1645199571) AT TIME ZONE 'UTC', '1', '1', '1', '2794', '1', TO_TIMESTAMP(1645199361) AT TIME ZONE 'UTC', '2.13.2-1', TO_TIMESTAMP(1645199571) AT TIME ZONE 'UTC')
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '0')
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '0')
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '0')
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '0')
[2022-02-18 16:52:51 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-18 16:52:51 +0100] information/IdoPgsqlConnection: Pending queries: 0 (Input: 3/s; Output: 3/s)
[2022-02-18 16:52:51 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-sa02.localdomain' because that's us.
[2022-02-18 16:52:51 +0100] debug/ApiListener: Not connecting to Endpoint 'otn-ac-monq-ma01.localdomain' because the host/port attributes are missing.
[2022-02-18 16:52:51 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'otn-ac-monq-sa02.localdomain'.
[2022-02-18 16:52:51 +0100] notice/ApiListener: Current zone master: otn-ac-monq-sa02.localdomain
[2022-02-18 16:52:51 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: SELECT 1
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:51 +0100] debug/IdoPgsqlConnection: Query: BEGIN
[2022-02-18 16:52:52 +0100] debug/IdoPgsqlConnection: Query: COMMIT
[2022-02-18 16:52:52 +0100] debug/IdoPgsqlConnection: Query: BEGIN

zones.conf of master1 (ma01)

object Endpoint "otn-ac-monq-ma01.localdomain" {
  // That's us
}

object Endpoint "otn-ac-monq-sa02.localdomain" {
  host = "otn-ac-monq-sa02.localdomain" // Actively connect to the secondary master
}

object Zone "master" {
  endpoints = [ "otn-ac-monq-ma01.localdomain", "otn-ac-monq-sa02.localdomain" ]
}

zones.conf of master2 (sa02)

object Endpoint "otn-ac-monq-ma01.localdomain" {
        #host = "otn-ac-monq-ma01.localdomain"
        #port = "5665"
}

object Zone "master" {
        endpoints = [ "otn-ac-monq-ma01.localdomain", "otn-ac-monq-sa02.localdomain" ]
}

object Endpoint "otn-ac-monq-sa02.localdomain" {
  //That's us
}

api.conf on master1 (ma01)

object ApiListener "api" {
  accept_config = true
  accept_commands = true

  ticket_salt = TicketSalt
}

api.conf on master2 (sa02)

object ApiListener "api" {
  accept_config = true
  accept_commands = true
}

ido-pgsql.conf on both nodes

object IdoPgsqlConnection "ido-pgsql" {
  user = "icinga"
  password = "12345"
  host = "otn-ac-monq-dvip.localdomain"
  database = "icinga"
  enable_ha=true
}
10

Have you tried naming the endpoints equally on both nodes?

11

@Al2Klimov the are named equally, it was just a copy and paste error, sorry.

12

Hello, I just quickly set up a cluster just like yours and works perfectly.

master-01 Logs:

[2022-02-25 09:44:09 +0100] warning/ApiListener: Ignoring config update. 'api' does not accept config.
[2022-02-25 09:44:09 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'satellite' in zone 'master'.
[2022-02-25 09:44:09 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'master-01'.
[2022-02-25 09:44:09 +0100] information/IdoPgsqlConnection: 'ido-pgsql' resumed.
[2022-02-25 09:44:09 +0100] information/DbConnection: Resuming IDO connection: ido-pgsql
[2022-02-25 09:44:09 +0100] information/ApiListener: Finished sending replay log for endpoint 'satellite' in zone 'master'.
[2022-02-25 09:44:09 +0100] information/ApiListener: Finished syncing endpoint 'satellite' in zone 'master'.
[2022-02-25 09:44:09 +0100] notice/IdoPgsqlConnection: Enabling IDO connection.
[2022-02-25 09:44:09 +0100] information/IdoPgsqlConnection: PGSQL IDO instance id: 1 (schema version: '1.14.3')
[2022-02-25 09:44:09 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-25 09:44:11 +0100] information/IdoPgsqlConnection: Finished reconnecting to 'ido-pgsql' database 'icinga2' in 1.63365 second(s).
[2022-02-25 09:44:19 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-25 09:44:19 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'master-01'.
[2022-02-25 09:44:19 +0100] notice/ApiListener: Current zone master: master-01
[2022-02-25 09:44:19 +0100] notice/ApiListener: Connected endpoints: satellite (1)
[2022-02-25 09:44:19 +0100] information/IdoPgsqlConnection: Pending queries: 0 (Input: 235/s; Output: 236/s)

When I stopped master-01, the second master was taking over after a few failover retrying.

[2022-02-25 09:45:05 +0100] warning/JsonRpcConnection: API client disconnected for identity 'master-01'
[2022-02-25 09:45:05 +0100] warning/ApiListener: Removing API client for endpoint 'master-01'. 0 API clients left.
[2022-02-25 09:45:12 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-25 09:45:12 +0100] notice/ApiListener: Current zone master: satellite
[2022-02-25 09:45:12 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-25 09:45:12 +0100] information/IdoPgsqlConnection: 'ido-pgsql' resumed.
[2022-02-25 09:45:12 +0100] information/DbConnection: Resuming IDO connection: ido-pgsql
[2022-02-25 09:45:12 +0100] information/IdoPgsqlConnection: Last update by endpoint 'master-01' was 13.9798s ago (< failover timeout of 30s). Retrying.
[2022-02-25 09:45:22 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-25 09:45:22 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'satellite'.
[2022-02-25 09:45:22 +0100] notice/ApiListener: Current zone master: satellite
[2022-02-25 09:45:22 +0100] notice/ApiListener: Connected endpoints: 
[2022-02-25 09:45:22 +0100] information/IdoPgsqlConnection: Last update by endpoint 'master-01' was 23.958s ago (< failover timeout of 30s). Retrying.
[2022-02-25 09:45:27 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 0; Checks/s: 0
[2022-02-25 09:45:32 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-25 09:45:32 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'satellite'.
[2022-02-25 09:45:32 +0100] notice/ApiListener: Current zone master: satellite
[2022-02-25 09:45:32 +0100] information/IdoPgsqlConnection: Last update by endpoint 'master-01' was 33.9587s ago. Taking over 'ido-pgsql' in HA zone 'master'.
[2022-02-25 09:45:32 +0100] notice/IdoPgsqlConnection: Enabling IDO connection.
[2022-02-25 09:45:32 +0100] information/IdoPgsqlConnection: PGSQL IDO instance id: 1 (schema version: '1.14.3')
[2022-02-25 09:45:32 +0100] notice/DbConnection: Updating programstatus table.
[2022-02-25 09:45:35 +0100] information/IdoPgsqlConnection: Finished reconnecting to 'ido-pgsql' database 'icinga2' in 2.67237 second(s).

Though you have told us that the masters both have the same feature enabled, but I can’t see any log of DBConnection in the second master of yours at all. So, can you please share the outputs of icinga2 feature list of both masters here.

13

@jonas

[root@otn-ac-monq-ma01 icinga2]# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-pgsql influxdb mainlog notification


[root@otn-ac-monq-sa02 ~]# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker mainlog

It seems that there is no ido-pgsql feature if it is setup as satellite instead of a master?

14

If you have master ↔ master replication, i.e. when one fails, the other can take its place and continue with the checks, notifications, etc… But if you don’t have the same features enabled in the second master as in master1, then the replication will not work either. You have to do this yourself! this does not happen automatically, i.e. if you for example enable ido-pgsql in master1 then you have to do the same in master2 as well. I just noticed that in master2 you are not only missing ido-pgsql but also influxdb and notification features too.