Master-Master Cluster Sync Problem

PinkFrog · November 15, 2019, 2:47pm

Hi everbody,
I have some problems with my Master-Master-Config-Sync. I can see the following Warning in Icingaweb 2:

After a Icinga2 restart I can see the following in the icinga2.log:
Master1:

[2019-11-15 14:57:17 +0100] warning/JsonRpcConnection: API client disconnected for identity 'master2'
[2019-11-15 14:57:17 +0100] warning/ApiListener: Removing API client for endpoint 'master2'. 0 API clients left.
[2019-11-15 14:57:18 +0100] information/ApiListener: New client connection for identity 'master2' from [xxx.xxx.xxx.xxx]:34518
[2019-11-15 14:57:18 +0100] information/ApiListener: Sending config updates for endpoint 'master2' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'master2'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'master2'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished sending config file updates for endpoint 'master2' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Syncing runtime objects to endpoint 'master2'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'master2'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'master2' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Sending replay log for endpoint 'master2' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished sending replay log for endpoint 'master2' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished syncing endpoint 'master2' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Applying config update from endpoint 'master2' of zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Ignoring config update from endpoint 'master2' for zone 'director-global' because we have an authoritative version of the zone's config.
[2019-11-15 14:57:18 +0100] information/ApiListener: Ignoring config update from endpoint 'master2' for zone 'master' because we have an authoritative version of the zone's config.
[2019-11-15 14:57:18 +0100] information/ApiListener: Received configuration updates (0) from endpoint 'master2' do not qualify for production, not triggering reload.

Master2:

[2019-11-15 14:57:17 +0100] information/ApiListener: Reconnecting to endpoint 'master1' via host 'master1' and port '5665'
[2019-11-15 14:57:18 +0100] information/ApiListener: New client connection for identity 'master1' to [xxx.xxx.xxx.xxx]:5665
[2019-11-15 14:57:18 +0100] information/ApiListener: Sending config updates for endpoint 'master1' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'master1'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'master1'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished sending config file updates for endpoint 'master1' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Syncing runtime objects to endpoint 'master1'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'master1'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'master1' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Sending replay log for endpoint 'master1' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished sending replay log for endpoint 'master1' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished syncing endpoint 'master1' in zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Finished reconnecting to endpoint 'master1' via host 'master1' and port '5665'
[2019-11-15 14:57:18 +0100] information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'.
[2019-11-15 14:57:18 +0100] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'master1'. Comparing the timestamp and checksums.
[2019-11-15 14:57:18 +0100] information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the timestamp and checksums.
[2019-11-15 14:57:18 +0100] information/ApiListener: Received configuration updates (2) from endpoint 'master1' do not qualify for production, not triggering reload.

And in the debug.log on Master2 I can find something like:

 [2019-11-15 14:51:39 +0100] information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/.checksums'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/.timestamp'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/_etc/commands.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/_etc/notification_commands.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/001-director-basics.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/commands.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/host_templates.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/hostgroups.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/service_apply.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/service_templates.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/servicesets.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/timeperiods.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/director-global/director/user_templates.conf'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'master1'. Comparing the timestamp and checksums.
[2019-11-15 14:51:39 +0100] debug/ApiListener: Checking for config change between stage and production. Old (13): '{"/.checksums":"e593991b918e92812a46ac18b6c5b8f82784afae84a5c373a32e764da266cf8c","/.timestamp":"6c55fb5fe204cd63d5b9992acadd92eda791e4a8dde36d70c1bae22df3b4042e","/_etc/commands.conf":"bdd3fc2da8d38feb4863b5409f0f162abd35802705266637afef82b8978ce97d","/_etc/notification_commands.conf":"628cb26f9c5f93f000a906b0e62dc649ae0fd3a4662f9ee0f6e4ac2107686ce2","/director/001-director-basics.conf":"f86583eec81c9bf3a1823a761991fb53d640bd0dc6cd12bf8c5e6a275359970f","/director/commands.conf":"213c51d3b2c5dbc27e3b612419a770053a7ac1ab7e56065d4159a513edb33ac1","/director/host_templates.conf":"5df70e653f0e4fd2f2b77ba2b9057da0e3f71845cdae19aee22c9bd20bc54a78","/director/hostgroups.conf":"57a5acafe45b5f2835490fb2ab9cdcb0d4f634481b163addad1f5b5f2264ff0b","/director/service_apply.conf":"12dba3428320ed44cbf1e20fcd54f0a9b12caba111934498d193ff3acf5c3763","/director/service_templates.conf":"cfbc1dab28283c64a2d4e2cebf65cd0e191317c2cc35e92902cdcad039d6b927","/director/servicesets.conf":"fe6e65ce8646acf4a17c02d634c5d9e0c75e9d3e8219f11e0c6f3f6c9e1fd592","/director/timeperiods.conf":"6d6fe409267a4f446fc9254e472b350b0199b503d7d902e5620438330a4b8b89","/director/user_templates.conf":"bc51198e7132eff30ef199b80c1290671e53308d20546d91222075ef71e9f0d9"}' vs. new (13): '{"/.checksums":"e593991b918e92812a46ac18b6c5b8f82784afae84a5c373a32e764da266cf8c","/.timestamp":"96904e185b76c7c49ebf3f38ee00b047a1deb11d58f6ecf4d100df1abb1fbe4a","/_etc/commands.conf":"bdd3fc2da8d38feb4863b5409f0f162abd35802705266637afef82b8978ce97d","/_etc/notification_commands.conf":"628cb26f9c5f93f000a906b0e62dc649ae0fd3a4662f9ee0f6e4ac2107686ce2","/director/001-director-basics.conf":"f86583eec81c9bf3a1823a761991fb53d640bd0dc6cd12bf8c5e6a275359970f","/director/commands.conf":"213c51d3b2c5dbc27e3b612419a770053a7ac1ab7e56065d4159a513edb33ac1","/director/host_templates.conf":"5df70e653f0e4fd2f2b77ba2b9057da0e3f71845cdae19aee22c9bd20bc54a78","/director/hostgroups.conf":"57a5acafe45b5f2835490fb2ab9cdcb0d4f634481b163addad1f5b5f2264ff0b","/director/service_apply.conf":"12dba3428320ed44cbf1e20fcd54f0a9b12caba111934498d193ff3acf5c3763","/director/service_templates.conf":"cfbc1dab28283c64a2d4e2cebf65cd0e191317c2cc35e92902cdcad039d6b927","/director/servicesets.conf":"fe6e65ce8646acf4a17c02d634c5d9e0c75e9d3e8219f11e0c6f3f6c9e1fd592","/director/timeperiods.conf":"6d6fe409267a4f446fc9254e472b350b0199b503d7d902e5620438330a4b8b89","/director/user_templates.conf":"bc51198e7132eff30ef199b80c1290671e53308d20546d91222075ef71e9f0d9"}'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//_etc/commands.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//_etc/notification_commands.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/commands.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/hostgroups.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_apply.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/timeperiods.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/user_templates.conf' for zone 'director-global'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (44270 Bytes).
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/.checksums'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/.timestamp'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/agent_endpoints.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/agent_zones.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/hosts.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/notification_apply.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/notification_templates.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/scheduled_downtime_apply.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/service_templates.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/services.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/usergroups.conf'.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Creating config update for file '/var/lib/icinga2/api/zones/master/director/users.conf'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the timestamp and checksums.
[2019-11-15 14:51:39 +0100] debug/ApiListener: Checking for config change between stage and production. Old (12): '{"/.checksums":"de4ef116c9e1d701602a9b3b102e658805fd4dcd978cc570fa786ee4a7d6f005","/.timestamp":"d593af8803543a38f7bf58ca6b5454a8ee8766707960a0c6dff1c92891456c0b","/director/agent_endpoints.conf":"53a0c25b8a7acf522e530105e21fc52c6635e20490a06e5bf873c7104ae7c0d1","/director/agent_zones.conf":"278252222522875b9ac7c4f4e04c3f1c950f87d220f65de008f79e1105bddd91","/director/hosts.conf":"d89a54cdc8c89d25234fd15a3eb6688be05e1e1d0fb4784ede7aa563de52fab6","/director/notification_apply.conf":"120f92c9537d6a1a7bf1f965ae9acdbed160488625c4082aff2f82894657c579","/director/notification_templates.conf":"8597394cce331a4625c8244a2de7d8cd2af62f0e1c1e71a7e11a77e158ea10cc","/director/scheduled_downtime_apply.conf":"831a5e922fcf863b4f3d01b3af6322ba868af416212ab6b807d58d5aa743c008","/director/service_templates.conf":"98f4eaed162d544ad94700eeefbd40c006edfc44669e7bd0cadddea77daabad7","/director/services.conf":"11b4482833a919e37455688c836bfa1e6bee08892e3f06b48a724b501487e343","/director/usergroups.conf":"4712a29dcc8d3b48048f2b0b27191ad21619644e3bcd98b9878932ab3c13ff4e","/director/users.conf":"213162303db429efadecc16f9706a7065d5e8b8432d26d528029a338bc03c6e4"}' vs. new (12): '{"/.checksums":"de4ef116c9e1d701602a9b3b102e658805fd4dcd978cc570fa786ee4a7d6f005","/.timestamp":"ecdc6e0bc14bc9bd35df9bb9b3dc4dcd0ba5ce13b189ba65f11be22c1435ac27","/director/agent_endpoints.conf":"53a0c25b8a7acf522e530105e21fc52c6635e20490a06e5bf873c7104ae7c0d1","/director/agent_zones.conf":"278252222522875b9ac7c4f4e04c3f1c950f87d220f65de008f79e1105bddd91","/director/hosts.conf":"d89a54cdc8c89d25234fd15a3eb6688be05e1e1d0fb4784ede7aa563de52fab6","/director/notification_apply.conf":"120f92c9537d6a1a7bf1f965ae9acdbed160488625c4082aff2f82894657c579","/director/notification_templates.conf":"8597394cce331a4625c8244a2de7d8cd2af62f0e1c1e71a7e11a77e158ea10cc","/director/scheduled_downtime_apply.conf":"831a5e922fcf863b4f3d01b3af6322ba868af416212ab6b807d58d5aa743c008","/director/service_templates.conf":"98f4eaed162d544ad94700eeefbd40c006edfc44669e7bd0cadddea77daabad7","/director/services.conf":"11b4482833a919e37455688c836bfa1e6bee08892e3f06b48a724b501487e343","/director/usergroups.conf":"4712a29dcc8d3b48048f2b0b27191ad21619644e3bcd98b9878932ab3c13ff4e","/director/users.conf":"213162303db429efadecc16f9706a7065d5e8b8432d26d528029a338bc03c6e4"}'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/agent_endpoints.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/agent_zones.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/hosts.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/notification_apply.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/notification_templates.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/scheduled_downtime_apply.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/service_templates.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/services.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/usergroups.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/master//director/users.conf' for zone 'master'.
[2019-11-15 14:51:39 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/master' (324225 Bytes).
[2019-11-15 14:51:39 +0100] information/ApiListener: Received configuration updates (2) from endpoint 'master1' do not qualify for production, not triggering reload.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Received config update for object: {"config":"object Downtime \"7ba6db01-9ace-4cf7-8166-8c0cdd8d9a63\" ignore_on_error {\n\tauthor = \"F. Hoffmann\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"Auslastung im Zeitraum 15:00-6:00 muss nicht \u00fcberwacht werden.\"\n\tconfig_owner = \"server.domain!load!Curalit-Load\"\n\tduration = 0.000000\n\tend_time = 1573858800.000000\n\tentry_time = 1573798231.584726\n\tfixed = true\n\thost_name = \"server.domain\"\n\tscheduled_by = \"server.domain!load!Curalit-Load\"\n\tservice_name = \"load\"\n\tstart_time = 1573826400.000000\n\ttriggered_by = \"\"\n\tversion = 1573798231.584782\n\tzone = \"master\"\n}\n","modified_attributes":{},"name":"server.domain!load!7ba6db01-9ace-4cf7-8166-8c0cdd8d9a63","original_attributes":[],"type":"Downtime","version":1573798231.584782}
[2019-11-15 14:51:39 +0100] notice/ApiListener: Discarding config update from 'master1' (endpoint: 'master1', zone: 'master') for object 'server.domain!load!7ba6db01-9ace-4cf7-8166-8c0cdd8d9a63': Object version 1573798231.584782 is more recent than the received version 1573798231.584782.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Received config update for object: {"config":"object Downtime \"e137e431-519e-46a7-a365-d66732c60ef1\" ignore_on_error {\n\tauthor = \"F. Hoffmann\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"Auslastung im Zeitraum 15:00-6:00 muss nicht \u00fcberwacht werden.\"\n\tconfig_owner = \"server.domain!load!Curalit-Load\"\n\tduration = 0.000000\n\tend_time = 1573858800.000000\n\tentry_time = 1573798231.612617\n\tfixed = true\n\thost_name = \"server.domain\"\n\tscheduled_by = \"server.domain!load!Curalit-Load\"\n\tservice_name = \"load\"\n\tstart_time = 1573826400.000000\n\ttriggered_by = \"\"\n\tversion = 1573798231.612648\n\tzone = \"master\"\n}\n","modified_attributes":{},"name":"server.domain!load!e137e431-519e-46a7-a365-d66732c60ef1","original_attributes":[],"type":"Downtime","version":1573798231.612648}
[2019-11-15 14:51:39 +0100] notice/ApiListener: Discarding config update from 'master1' (endpoint: 'master1', zone: 'master') for object 'server.domain!load!e137e431-519e-46a7-a365-d66732c60ef1': Object version 1573798231.612648 is more recent than the received version 1573798231.612648.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Received config update for object: {"config":"object Downtime \"f1410748-032f-4f6b-8a00-62043c1a39ff\" ignore_on_error {\n\tauthor = \"F. Hoffmann\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"Auslastung im Zeitraum 15:00-6:00 muss nicht \u00fcberwacht werden.\"\n\tconfig_owner = \"server.domain!load!Curalit-Load\"\n\tduration = 0.000000\n\tend_time = 1573858800.000000\n\tentry_time = 1573798231.603082\n\tfixed = true\n\thost_name = \"server.domain\"\n\tscheduled_by = \"server.domain!load!Curalit-Load\"\n\tservice_name = \"load\"\n\tstart_time = 1573826400.000000\n\ttriggered_by = \"\"\n\tversion = 1573798231.603100\n\tzone = \"master\"\n}\n","modified_attributes":{},"name":"server.domain!load!f1410748-032f-4f6b-8a00-62043c1a39ff","original_attributes":[],"type":"Downtime","version":1573798231.6031}
[2019-11-15 14:51:39 +0100] notice/ApiListener: Discarding config update from 'master1' (endpoint: 'master1', zone: 'master') for object 'server.domain!load!f1410748-032f-4f6b-8a00-62043c1a39ff': Object version 1573798231.603100 is more recent than the received version 1573798231.603100.
[2019-11-15 14:51:39 +0100] notice/ApiListener: Received config update for object: {"config":"object Downtime \"e53fb33e-5d3f-4c82-9c70-293d08ceb3d8\" ignore_on_error {\n\tauthor = \"M. Lux\"\n\tauthoritative_zone = \"master\"\n\tcomment = \"Auslastung im Zeitraum 02:00-04:15 Uhr muss nicht \u00fcberwacht werden.\"\n\tconfig_owner = \"server.domain!load!papfts-Load\"\n\tduration = 0.000000\n\tend_time = 1573874100.000000\n\tentry_time = 1573798231.595975\n\tfixed = true\n\thost_name = \"server.domain\"\n\tscheduled_by = \"server.domain!load!papfts-Load\"\n\tservice_name = \"load\"\n\tstart_time = 1573866000.000000\n\ttriggered_by = \"\"\n\tversion = 1573798231.595993\n\tzone = \"master\"\n}\n","modified_attributes":{},"name":"server.domain!load!e53fb33e-5d3f-4c82-9c70-293d08ceb3d8","original_attributes":[],"type":"Downtime","version":1573798231.595993}
[2019-11-15 14:51:39 +0100] notice/ApiListener: Discarding config update from 'master1' (endpoint: 'master1', zone: 'master') for object 'server.domain!load!e53fb33e-5d3f-4c82-9c70-293d08ceb3d8': Object version 1573798231.595993 is more recent than the received version 1573798231.595993.

I don’t see some issues because of this, but I’m confused about it.

I found the function which causes the

notice/ApiListener: Discarding config update from '' (endpoint: '', zone: 'master') for object '': Object version is more recent than the received version .

notice, and it seems like working like planned to discard config updates if the .timestamp is equal… so I have no more ideas how to go further.

Any suggestions to get the warning in the monitoring solved?

Best regards
Alicia

anon66228339 · November 18, 2019, 7:14am

Do a icinga2 daemon -C on both Masters, one of them has a problem with the new configuration.
Solve the problems that the command will report and you will be fine again.

PinkFrog · November 18, 2019, 8:28am

Hi Carster,
I don’t get any error doing icinga2 daemon -C
Master1:

information/cli: Icinga application loader (version: r2.11.2-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: psmici101.ad.ekom21.de
information/ConfigItem: Instantiated 4 ScheduledDowntimes.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 4 NotificationCommands.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 3 HostGroups.
information/ConfigItem: Instantiated 183 Hosts.
information/ConfigItem: Instantiated 4 Downtimes.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 179 Zones.
information/ConfigItem: Instantiated 178 Endpoints.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 2 ApiUsers.
information/ConfigItem: Instantiated 310 UserGroups.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 245 CheckCommands.
information/ConfigItem: Instantiated 2 TimePeriods.
information/ConfigItem: Instantiated 549 Users.
information/ConfigItem: Instantiated 897 Services.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

Master2:

information/cli: Icinga application loader (version: r2.11.2-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: psmici102.ad.ekom21.de
information/ConfigItem: Instantiated 4 ScheduledDowntimes.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 4 NotificationCommands.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 3 HostGroups.
information/ConfigItem: Instantiated 183 Hosts.
information/ConfigItem: Instantiated 4 Downtimes.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 179 Zones.
information/ConfigItem: Instantiated 178 Endpoints.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 310 UserGroups.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 245 CheckCommands.
information/ConfigItem: Instantiated 2 TimePeriods.
information/ConfigItem: Instantiated 549 Users.
information/ConfigItem: Instantiated 897 Services.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

So I guess this didn’t help.
If i run a icinga2 daemon -C -x debug I get only some debug/HostGroup: Assigning membership for group extra.

Sorry.

anon66228339 · November 18, 2019, 12:02pm

looks like master2 doesnt like the changes. Maybe a cleanup of the api directores helps.

rm -rf /var/lib/icinga2/{packages,zones,zones-stage}/*
systemctl restart icinga2

btw if you are on icinga2 2.11 you can set the apilistener from master 1 also to receive commands/config

  accept_commands = true
  accept_config = true

PinkFrog · November 18, 2019, 12:33pm

Thanks again for your reply,

rm -rf /var/lib/icinga2/api/{zones,zones-stage}/* I had tried before, but the packages/* was new for me, so I tried id with rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/* but it doesn’t seems to make a big difference:

# systemctl restart icinga2.service && tail -f /var/log/icinga2/icinga2.log | grep master
information/ApiListener: Reconnecting to endpoint 'master1' via host 'master1' and port '5665'
information/ApiListener: New client connection for identity 'master1' to [xxx.xxx.xxx.xxx]:5665
information/ApiListener: Sending config updates for endpoint 'master1' in zone 'master'.
information/ApiListener: Finished sending config file updates for endpoint 'master1' in zone 'master'.
information/ApiListener: Syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished sending runtime config updates for endpoint 'master1' in zone 'master'.
information/ApiListener: Sending replay log for endpoint 'master1' in zone 'master'.
information/ApiListener: Finished sending replay log for endpoint 'master1' in zone 'master'.
information/ApiListener: Finished syncing endpoint 'master1' in zone 'master'.
information/ApiListener: Finished reconnecting to endpoint 'master1' via host 'master1' and port '5665'
information/IdoMysqlConnection: Last update by endpoint 'master1' was 7.35944s ago (< failover timeout of 30s). Retrying.
information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'.
information/ApiListener: Received configuration for zone 'director-global' from endpoint 'master1'. Comparing the timestamp and checksums.
information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the timestamp and checksums.
information/ApiListener: Received configuration updates (2) from endpoint 'master1' are different to production, triggering validation and reload.
information/ApiListener: Reconnecting to endpoint 'master1' via host 'master1' and port '5665'
information/ApiListener: New client connection for identity 'master1' to [xxx.xxx.xxx.xxx]:5665
information/ApiListener: Sending config updates for endpoint 'master1' in zone 'master'.
information/ApiListener: Syncing configuration files for zone 'master' to endpoint 'master1'.
information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'master1'.
information/ApiListener: Finished sending config file updates for endpoint 'master1' in zone 'master'.
information/ApiListener: Syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished syncing runtime objects to endpoint 'master1'.
information/ApiListener: Finished sending runtime config updates for endpoint 'master1' in zone 'master'.
information/ApiListener: Sending replay log for endpoint 'master1' in zone 'master'.
information/IdoMysqlConnection: Last update by endpoint 'master1' was 10.8944s ago (< failover timeout of 30s). Retrying.
information/ApiListener: Finished sending replay log for endpoint 'master1' in zone 'master'.
information/ApiListener: Finished syncing endpoint 'master1' in zone 'master'.
information/ApiListener: Finished reconnecting to endpoint 'master1' via host 'master1' and port '5665'
information/ApiListener: Applying config update from endpoint 'master1' of zone 'master'.
information/ApiListener: Received configuration for zone 'director-global' from endpoint 'master1'. Comparing the timestamp and checksums.
information/ApiListener: Received configuration for zone 'master' from endpoint 'master1'. Comparing the timestamp and checksums.
information/ApiListener: Received configuration updates (2) from endpoint 'master1' do not qualify for production, not triggering reload.
information/IdoMysqlConnection: Last update by endpoint 'master1' was 20.8893s ago (< failover timeout of 30s). Retrying.
information/IdoMysqlConnection: Last update by endpoint 'master1' was 30.8857s ago. Taking over 'ido-mysql' in HA zone 'master'.
information/ExternalCommandListener: Executing external command: [1574079592] SCHEDULE_FORCED_SVC_CHECK;master1;icinga;1574079592

Yes, I’m accepting configs and commands also on master1

master1:

# cat /etc/icinga2/features-enabled/api.conf
/**
 * The API listener is used for distributed monitoring setups.
 */
object ApiListener "api" {

  ticket_salt = TicketSalt

  accept_config = true
  accept_commands = true
}

master2:

# cat /etc/icinga2/features-enabled/api.conf
/**
 * The API listener is used for distributed monitoring setups.
 */
object ApiListener "api" {
  accept_config = true
  accept_commands = true
}

Thanks for your help…

Best regards,
Alicia

anon66228339 · November 18, 2019, 12:39pm

Hmm there is something on master2 that differs from master1.
Did you check your zone config on master2 and its exactly the same as on master1 for master zone & global zones?
Or different versions of icinga2 or did you manualy manipulate the itl commands on master 1?

PinkFrog · November 18, 2019, 1:03pm

The only difference between both zones.conf is the order of the master-Endpoints, I started with the server itself on every server. Could that be a problem? The rest of the file is exactly the same.

Version is r2.11.2-1 on both systems.

What do you mean with that? I only have own check commands in the directory /etc/icinga2/zones.d/director-global/ on master1. I wouldn’t know how to manipulate something in the itl commands.

PinkFrog · November 18, 2019, 1:57pm

But there is something, I’m recognizing now. This starts while I was deleting a extra zone from the setup. Maybe there is another point where Icinga2 is saving such “old” Zone-Settings? I have removed the Zone and the endpoint of this zone from both zones.conf and made a new Director Kickstart, maybe it is saved in any other file?

And I compared the output of icinga2 object list from both masters, there is no difference.

anon66228339 · November 18, 2019, 2:02pm

Grep through the api directory for that deleted zone. Maybe the director renders it still.

PinkFrog · November 18, 2019, 2:10pm

No, there is nothing with the name of the old zone.

But I found another difference between both configurations, only on master1 is the file conf.d/api-users.conf included, so there is no configuration for api-users on master2. This is the line more in the output of icinga2 daemon -C on master1. Could this be a problem?

anon66228339 · November 18, 2019, 2:11pm

differrent api-user configuration shoulld not be a problem (beside for the api users)

PinkFrog · November 18, 2019, 2:22pm

Is there a possibility to make manually a sync stage validation like the icinga check?

It confuses me, that the date is the same since it come up first on 2019-11-13 18:05:41.

Icinga 2 has been running for 8 hours, 11 minutes and 30 seconds. Version: r2.11.2-1; Last zone sync stage validation failed at 2019-11-13 18:05:41 +0100

And also confusing that there is no startup.log anymore.

dnsmichi · November 18, 2019, 2:41pm

What happens if you change a custom variable for a host, and deploy the changes via the Director on master1?

PinkFrog · November 18, 2019, 2:49pm

There is no problem to change a custom variable or anything else. The deployment is successful and the changing takes effect.

I actually checked again the output of icinga2 daemon -C, this time with the notices and searching for “ignor” and found this on master1:

# icinga2 daemon -C -x notice | grep -i ignor
notice/config: Ignoring explicit load request for library "db_ido_mysql".
notice/config: Ignoring non local config include for zone 'director-global': We already have an authoritative copy included.
notice/config: Ignoring non local config include for zone 'master': We already have an authoritative copy included.

on master2 only:

# icinga2 daemon -C -x notice | grep -i ignor
notice/config: Ignoring explicit load request for library "db_ido_mysql".

dnsmichi · November 18, 2019, 2:50pm

And this change doesn’t change the output of the icinga check when you force a recheck of that service?

PinkFrog · November 18, 2019, 2:54pm

No, that’s weird:

Icinga 2 has been running for 8 hours, 42 minutes and 28 seconds. Version: r2.11.2-1; Last zone sync stage validation failed at 2019-11-13 18:05:41 +0100

Seems like there wasn’t a reload of icinga2.service, if i restart the service manually, the uptime for the service is changing, but the date for the stage validation failed does not change.

dnsmichi · November 18, 2019, 3:23pm

master2 should log something like this:

Copying file ... from config sync staging to production zones directory

and then clear the last failed zone sync stage validation entry.

PinkFrog · November 18, 2019, 3:26pm

Yes, there are such lines at this time in the log.

Where can I clear that?

dnsmichi · November 18, 2019, 4:18pm

That sentence referred to icinga2 should ... so the core itself clears that. It is weird that it does not do that. The icinga2.state file should have the entry removed.

PinkFrog · November 18, 2019, 4:32pm

But it doesn’t. Maybe a bug?

The check “icinga” is a check of the file /var/lib/icinga2/icinga2.state?