Last zone sync stage validation failed

Upgrade whole system to r2.11.0-1, everything worked fine, except one host. I receive this notification:

Icinga 2 has been running for 5 hours, 18 minutes and 23 seconds. Version: r2.11.0-1; Last zone sync stage validation failed at 2019-09-23 08:02:42 +0200

Can’t find any hint, what to do. I receive notification for this machine, but how could i solve this failed sync?

Hi,

2.11 comes with a new cluster config sync with staging, as noted here: https://icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/#config-sync

In order to troubleshooting further, please follow this new entry in the docs: https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#cluster-troubleshooting-config-sync

Please post your findings from it, I’m here for additional questions.

Cheers,
Michael

Hi @lindesbs,

I have had the same 'problem" and found out that you can find a startup.log in case there are startup problems. (no problems means no file;) (described as zone sync stage validation that failed)
In my case those came from a service change on the master zone but on that side you wont see it will gives errors on the endpoint/satellite.

Check the stage zone for errors and look for the startup.log file. ( icinga2 daemon -C )

/var/lib/icinga2/api/zones-stage

Dont know if it’s necessary but I reloaded the service on the endpoint/satellite and master.

Hi,

i see the exact same behaviour. Except that it does not happens on every reload/config change but remains until i delete /var/lib/icinga2/api/zones and /var/lib/icinga2/api/zones-stage.

Config validations via “icinga2 daemon -C” and the debug log looks completly fine to me.

It occurs (almost) always on the same satellite in a HA zone.

1 Like

Jepp, still no solution yet. All other hosts in the zone works fine, except this one.
Checked config multiple times, evrything is green

Is “icinga2 daemon -C” all green?
If everything is good the startup.log file shouldnt be visible in /var/lib/icinga2/api/zones-stage/

Yes, all green and no startup.log in /var/lib/icinga2/api/zones-stage

Maybe you already tried this, but my few cents:

icinga2 feature enable debuglog
tail /var/log/icinga2/debug.log -f

Maybe you’ll find something there.

Debug Log looks also completly fine from my point of view.

Still existing after update to 2.11.1-1 an Debian Buster stable

Icinga 2 has been running for 10 minutes and 14 seconds. Version: r2.11.1-1; Last zone sync stage validation failed at 2019-09-26 14:22:31 +0200

Is there a possibility to redistribute config to all nodes?

Hi Stefan,

Could you, just for testing sake, stop icinga on that agent, delete /var/lib/icinga2/api/zones and /var/lib/icinga2/api/zones-stage , start icinga service ?
Btw, you master should have at least the same version as you agent otherwise you’ll get such message to. If that’s not possible, downgrade your agent to match the master.

Cool. That works with version: r2.11.2-1. Checked this in the past with version 2.11.1, which does not solve this issue.

But now it works.

Thanks for this tip.

1 Like

Good morning,

after upgrading my enviroment I have the same issue on 3/34 nodes:

I already try to delete /var/lib/icinga2/ icinga2.state modified-attributes.conf and /var/lib/icinga2/api/ packeges zones zones-stages and restart the service.

[2020-02-07 07:53:29 +0100] information/ApiListener: Received configuration updates (2) from endpoint ‘98lipmoni1’ do not qualify for production, not triggering reload.

icinga2 daemon -C on both masters are clear.

I hope there is a final solution.

zones.conf Master1:

/etc/icinga2# cat zones.conf
/*
 * Endpoint and Zone configuration for a cluster setup
 * This local example requires `NodeName` defined in
 * constants.conf.
 */

object Endpoint "01livmoni1" {
}

object Endpoint "03livmoni1" {
}

object Endpoint "04livmoni1" {
}

object Endpoint "07livmoni1" {
}

object Endpoint "08livmoni1" {
}

object Endpoint "09livmoni1" {
}

object Endpoint "10livmoni1" {
}

object Endpoint "11livmoni1" {
}

object Endpoint "12livmoni1" {
}

object Endpoint "13livmoni1" {
}

object Endpoint "14livmoni1" {
}

object Endpoint "15livmoni1" {
}

object Endpoint "16livmoni1" {
}

object Endpoint "19livmoni1" {
}

object Endpoint "20livmoni1" {
}

object Endpoint "20livmoni1-dmz" {
}

object Endpoint "21livmoni1" {
}

object Endpoint "22livmoni1" {
}

object Endpoint "26livmoni1" {
}

object Endpoint "27livmoni1" {
}

object Endpoint "29livmoni1" {
}

object Endpoint "32livmoni1" {
}

object Endpoint "33livmoni1" {
}

object Endpoint "37livmoni1" {
}

object Endpoint "38livmoni1" {
}

object Endpoint "43livmoni1" {
}

object Endpoint "50livmoni1" {
}

object Endpoint "51livmoni1" {
}

object Endpoint "74livmoni1" {
}

object Endpoint "77livmoni1" {
}

object Endpoint "98livmoni1" {
}

object Endpoint "98livmoni2" {
}

object Endpoint "98livmoni1-dmz" {
}

object Endpoint "98lipmoni1" {
  host = "98lipmoni1"
}

object Endpoint "98lipmoni2" {
  host = "98lipmoni2"
}

object Zone "01rost" {
  parent = "master"
  endpoints = [ "01livmoni1" ]
}

object Zone "03lueb" {
  parent = "master"
  endpoints = [ "03livmoni1" ]
}

object Zone "04gbgh" {
  parent = "master"
  endpoints = [ "04livmoni1" ]
}

object Zone "07flen" {
  parent = "master"
  endpoints = [ "07livmoni1" ]
}

object Zone "08hhba" {
  parent = "master"
  endpoints = [ "08livmoni1" ]
}

object Zone "09kiel" {
  parent = "master"
  endpoints = [ "09livmoni1" ]
}

object Zone "10nbgh" {
  parent = "master"
  endpoints = [ "10livmoni1" ]
}

object Zone "11kigh" {
  parent = "master"
  endpoints = [ "11livmoni1" ]
}

object Zone "12lugh" {
  parent = "master"
  endpoints = [ "12livmoni1" ]
}

object Zone "13zogh" {
  parent = "master"
  endpoints = [ "13livmoni1" ]
}

object Zone "14zmgh" {
  parent = "master"
  endpoints = [ "14livmoni1" ]
}

object Zone "15rnur" {
  parent = "master"
  endpoints = [ "15livmoni1" ]
}

object Zone "16haat" {
  parent = "master"
  endpoints = [ "16livmoni1" ]
}

object Zone "19szgh" {
  parent = "master"
  endpoints = [ "19livmoni1" ]
}

object Zone "20sodk" {
  parent = "master"
  endpoints = [ "20livmoni1" ]
}

object Zone "20dmz" {
  parent = "master"
  endpoints = [ "20livmoni1-dmz" ]
}

object Zone "21kodk" {
  parent = "master"
  endpoints = [ "21livmoni1" ]
}

object Zone "22mase" {
  parent = "master"
  endpoints = [ "22livmoni1" ]
}

object Zone "26tapl" {
  parent = "master"
  endpoints = [ "26livmoni1" ]
}

object Zone "27wapl" {
  parent = "master"
  endpoints = [ "27livmoni1" ]
}

object Zone "29pzpl" {
  parent = "master"
  endpoints = [ "29livmoni1" ]
}

object Zone "32hbgt" {
  parent = "master"
  endpoints = [ "32livmoni1" ]
}

object Zone "33hegh" {
  parent = "master"
  endpoints = [ "33livmoni1" ]
}

object Zone "37magh" {
  parent = "master"
  endpoints = [ "37livmoni1" ]
}

object Zone "38hagt" {
  parent = "master"
  endpoints = [ "38livmoni1" ]
}

object Zone "43wagh" {
  parent = "master"
  endpoints = [ "43livmoni1" ]
}

object Zone "50jowe" {
  parent = "master"
  endpoints = [ "50livmoni1" ]
}

object Zone "51jobi" {
  parent = "master"
  endpoints = [ "51livmoni1" ]
}

object Zone "74kize" {
  parent = "master"
  endpoints = [ "74livmoni1" ]
}

object Zone "77tegh" {
  parent = "master"
  endpoints = [ "77livmoni1" ]
}

object Zone "98dmz" {
  parent = "master"
  endpoints = [ "98livmoni1-dmz" ]
}

object Zone "98zent" {
  parent = "master"
  endpoints = [ "98livmoni1", "98livmoni2" ]
}

object Zone "master" {
  endpoints = [ "98lipmoni1", "98lipmoni2" ]
}

object Zone "global" {
  global = true
}

zones.conf Master2:

/etc/icinga2# cat zones.conf
/*
 * Endpoint and Zone configuration for a cluster setup
 * This local example requires `NodeName` defined in
 * constants.conf.
 */

object Endpoint "01livmoni1" {
}

object Endpoint "03livmoni1" {
}

object Endpoint "04livmoni1" {
}

object Endpoint "07livmoni1" {
}

object Endpoint "08livmoni1" {
}

object Endpoint "09livmoni1" {
}

object Endpoint "10livmoni1" {
}

object Endpoint "11livmoni1" {
}

object Endpoint "12livmoni1" {
}

object Endpoint "13livmoni1" {
}

object Endpoint "14livmoni1" {
}

object Endpoint "15livmoni1" {
}

object Endpoint "16livmoni1" {
}

object Endpoint "19livmoni1" {
}

object Endpoint "20livmoni1" {
}

object Endpoint "20livmoni1-dmz" {
}

object Endpoint "21livmoni1" {
}

object Endpoint "22livmoni1" {
}

object Endpoint "26livmoni1" {
}

object Endpoint "27livmoni1" {
}

object Endpoint "29livmoni1" {
}

object Endpoint "32livmoni1" {
}

object Endpoint "33livmoni1" {
}

object Endpoint "37livmoni1" {
}

object Endpoint "38livmoni1" {
}

object Endpoint "43livmoni1" {
}

object Endpoint "50livmoni1" {
}

object Endpoint "51livmoni1" {
}

object Endpoint "74livmoni1" {
}

object Endpoint "77livmoni1" {
}

object Endpoint "98livmoni1" {
}

object Endpoint "98livmoni2" {
}

object Endpoint "98livmoni1-dmz" {
}

object Endpoint "98lipmoni1" {
  host = "98lipmoni1"
}

object Endpoint "98lipmoni2" {
  host = "98lipmoni2"
}

object Zone "01rost" {
  parent = "master"
  endpoints = [ "01livmoni1" ]
}

object Zone "03lueb" {
  parent = "master"
  endpoints = [ "03livmoni1" ]
}

object Zone "04gbgh" {
  parent = "master"
  endpoints = [ "04livmoni1" ]
}

object Zone "07flen" {
  parent = "master"
  endpoints = [ "07livmoni1" ]
}

object Zone "08hhba" {
  parent = "master"
  endpoints = [ "08livmoni1" ]
}

object Zone "09kiel" {
  parent = "master"
  endpoints = [ "09livmoni1" ]
}

object Zone "10nbgh" {
  parent = "master"
  endpoints = [ "10livmoni1" ]
}

object Zone "11kigh" {
  parent = "master"
  endpoints = [ "11livmoni1" ]
}

object Zone "12lugh" {
  parent = "master"
  endpoints = [ "12livmoni1" ]
}

object Zone "13zogh" {
  parent = "master"
  endpoints = [ "13livmoni1" ]
}

object Zone "14zmgh" {
  parent = "master"
  endpoints = [ "14livmoni1" ]
}

object Zone "15rnur" {
  parent = "master"
  endpoints = [ "15livmoni1" ]
}

object Zone "16haat" {
  parent = "master"
  endpoints = [ "16livmoni1" ]
}

object Zone "19szgh" {
  parent = "master"
  endpoints = [ "19livmoni1" ]
}

object Zone "20sodk" {
  parent = "master"
  endpoints = [ "20livmoni1" ]
}

object Zone "20dmz" {
  parent = "master"
  endpoints = [ "20livmoni1-dmz" ]
}

object Zone "21kodk" {
  parent = "master"
  endpoints = [ "21livmoni1" ]
}

object Zone "22mase" {
  parent = "master"
  endpoints = [ "22livmoni1" ]
}

object Zone "26tapl" {
  parent = "master"
  endpoints = [ "26livmoni1" ]
}

object Zone "27wapl" {
  parent = "master"
  endpoints = [ "27livmoni1" ]
}

object Zone "29pzpl" {
  parent = "master"
  endpoints = [ "29livmoni1" ]
}

object Zone "32hbgt" {
  parent = "master"
  endpoints = [ "32livmoni1" ]
}

object Zone "33hegh" {
  parent = "master"
  endpoints = [ "33livmoni1" ]
}

object Zone "37magh" {
  parent = "master"
  endpoints = [ "37livmoni1" ]
}

object Zone "38hagt" {
  parent = "master"
  endpoints = [ "38livmoni1" ]
}

object Zone "43wagh" {
  parent = "master"
  endpoints = [ "43livmoni1" ]
}

object Zone "50jowe" {
  parent = "master"
  endpoints = [ "50livmoni1" ]
}

object Zone "51jobi" {
  parent = "master"
  endpoints = [ "51livmoni1" ]
}

object Zone "74kize" {
  parent = "master"
  endpoints = [ "74livmoni1" ]
}

object Zone "77tegh" {
  parent = "master"
  endpoints = [ "77livmoni1" ]
}

object Zone "98dmz" {
  parent = "master"
  endpoints = [ "98livmoni1-dmz" ]
}

object Zone "98zent" {
  parent = "master"
  endpoints = [ "98livmoni1", "98livmoni2" ]
}

object Zone "master" {
  endpoints = [ "98lipmoni1", "98lipmoni2" ]
}

object Zone "global" {
  global = true
}

zones.conf satellite:

/etc/icinga2# cat zones.conf 
object Endpoint "98livmoni1-dmz" {
}

object Endpoint "98lipmoni1" {
  host = "98lipmoni1"
}

object Endpoint "98lipmoni2" {
  host = "98lipmoni2"
}

object Zone "98dmz" {
  parent = "master"
  endpoints = [ "98livmoni1-dmz" ]
}

object Zone "master" {
  endpoints = [ "98lipmoni1", "98lipmoni2" ]
}

object Zone "global" {
  global = true
}

Update:

after changing a service and reload on the config master everthing is ok. Very strange :grinning: