Last zone sync stage validation failed

Upgrade whole system to r2.11.0-1, everything worked fine, except one host. I receive this notification:

Icinga 2 has been running for 5 hours, 18 minutes and 23 seconds. Version: r2.11.0-1; Last zone sync stage validation failed at 2019-09-23 08:02:42 +0200

Can’t find any hint, what to do. I receive notification for this machine, but how could i solve this failed sync?

Hi,

2.11 comes with a new cluster config sync with staging, as noted here: https://icinga.com/docs/icinga2/latest/doc/16-upgrading-icinga-2/#config-sync

In order to troubleshooting further, please follow this new entry in the docs: https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#cluster-troubleshooting-config-sync

Please post your findings from it, I’m here for additional questions.

Cheers,
Michael

Hi @lindesbs,

I have had the same 'problem" and found out that you can find a startup.log in case there are startup problems. (no problems means no file;) (described as zone sync stage validation that failed)
In my case those came from a service change on the master zone but on that side you wont see it will gives errors on the endpoint/satellite.

Check the stage zone for errors and look for the startup.log file. ( icinga2 daemon -C )

/var/lib/icinga2/api/zones-stage

Dont know if it’s necessary but I reloaded the service on the endpoint/satellite and master.

Hi,

i see the exact same behaviour. Except that it does not happens on every reload/config change but remains until i delete /var/lib/icinga2/api/zones and /var/lib/icinga2/api/zones-stage.

Config validations via “icinga2 daemon -C” and the debug log looks completly fine to me.

It occurs (almost) always on the same satellite in a HA zone.

Jepp, still no solution yet. All other hosts in the zone works fine, except this one.
Checked config multiple times, evrything is green

Is “icinga2 daemon -C” all green?
If everything is good the startup.log file shouldnt be visible in /var/lib/icinga2/api/zones-stage/

Yes, all green and no startup.log in /var/lib/icinga2/api/zones-stage

Maybe you already tried this, but my few cents:

icinga2 feature enable debuglog
tail /var/log/icinga2/debug.log -f

Maybe you’ll find something there.

Debug Log looks also completly fine from my point of view.

Still existing after update to 2.11.1-1 an Debian Buster stable

Icinga 2 has been running for 10 minutes and 14 seconds. Version: r2.11.1-1; Last zone sync stage validation failed at 2019-09-26 14:22:31 +0200

Is there a possibility to redistribute config to all nodes?

Hi Stefan,

Could you, just for testing sake, stop icinga on that agent, delete /var/lib/icinga2/api/zones and /var/lib/icinga2/api/zones-stage , start icinga service ?
Btw, you master should have at least the same version as you agent otherwise you’ll get such message to. If that’s not possible, downgrade your agent to match the master.

Cool. That works with version: r2.11.2-1. Checked this in the past with version 2.11.1, which does not solve this issue.

But now it works.

Thanks for this tip.

1 Like

Good morning,

after upgrading my enviroment I have the same issue on 3/34 nodes:

I already try to delete /var/lib/icinga2/ icinga2.state modified-attributes.conf and /var/lib/icinga2/api/ packeges zones zones-stages and restart the service.

[2020-02-07 07:53:29 +0100] information/ApiListener: Received configuration updates (2) from endpoint ‘98lipmoni1’ do not qualify for production, not triggering reload.

icinga2 daemon -C on both masters are clear.

I hope there is a final solution.

zones.conf Master1:

/etc/icinga2# cat zones.conf
/*
 * Endpoint and Zone configuration for a cluster setup
 * This local example requires `NodeName` defined in
 * constants.conf.
 */

object Endpoint "01livmoni1" {
}

object Endpoint "03livmoni1" {
}

object Endpoint "04livmoni1" {
}

object Endpoint "07livmoni1" {
}

object Endpoint "08livmoni1" {
}

object Endpoint "09livmoni1" {
}

object Endpoint "10livmoni1" {
}

object Endpoint "11livmoni1" {
}

object Endpoint "12livmoni1" {
}

object Endpoint "13livmoni1" {
}

object Endpoint "14livmoni1" {
}

object Endpoint "15livmoni1" {
}

object Endpoint "16livmoni1" {
}

object Endpoint "19livmoni1" {
}

object Endpoint "20livmoni1" {
}

object Endpoint "20livmoni1-dmz" {
}

object Endpoint "21livmoni1" {
}

object Endpoint "22livmoni1" {
}

object Endpoint "26livmoni1" {
}

object Endpoint "27livmoni1" {
}

object Endpoint "29livmoni1" {
}

object Endpoint "32livmoni1" {
}

object Endpoint "33livmoni1" {
}

object Endpoint "37livmoni1" {
}

object Endpoint "38livmoni1" {
}

object Endpoint "43livmoni1" {
}

object Endpoint "50livmoni1" {
}

object Endpoint "51livmoni1" {
}

object Endpoint "74livmoni1" {
}

object Endpoint "77livmoni1" {
}

object Endpoint "98livmoni1" {
}

object Endpoint "98livmoni2" {
}

object Endpoint "98livmoni1-dmz" {
}

object Endpoint "98lipmoni1" {
  host = "98lipmoni1"
}

object Endpoint "98lipmoni2" {
  host = "98lipmoni2"
}

object Zone "01rost" {
  parent = "master"
  endpoints = [ "01livmoni1" ]
}

object Zone "03lueb" {
  parent = "master"
  endpoints = [ "03livmoni1" ]
}

object Zone "04gbgh" {
  parent = "master"
  endpoints = [ "04livmoni1" ]
}

object Zone "07flen" {
  parent = "master"
  endpoints = [ "07livmoni1" ]
}

object Zone "08hhba" {
  parent = "master"
  endpoints = [ "08livmoni1" ]
}

object Zone "09kiel" {
  parent = "master"
  endpoints = [ "09livmoni1" ]
}

object Zone "10nbgh" {
  parent = "master"
  endpoints = [ "10livmoni1" ]
}

object Zone "11kigh" {
  parent = "master"
  endpoints = [ "11livmoni1" ]
}

object Zone "12lugh" {
  parent = "master"
  endpoints = [ "12livmoni1" ]
}

object Zone "13zogh" {
  parent = "master"
  endpoints = [ "13livmoni1" ]
}

object Zone "14zmgh" {
  parent = "master"
  endpoints = [ "14livmoni1" ]
}

object Zone "15rnur" {
  parent = "master"
  endpoints = [ "15livmoni1" ]
}

object Zone "16haat" {
  parent = "master"
  endpoints = [ "16livmoni1" ]
}

object Zone "19szgh" {
  parent = "master"
  endpoints = [ "19livmoni1" ]
}

object Zone "20sodk" {
  parent = "master"
  endpoints = [ "20livmoni1" ]
}

object Zone "20dmz" {
  parent = "master"
  endpoints = [ "20livmoni1-dmz" ]
}

object Zone "21kodk" {
  parent = "master"
  endpoints = [ "21livmoni1" ]
}

object Zone "22mase" {
  parent = "master"
  endpoints = [ "22livmoni1" ]
}

object Zone "26tapl" {
  parent = "master"
  endpoints = [ "26livmoni1" ]
}

object Zone "27wapl" {
  parent = "master"
  endpoints = [ "27livmoni1" ]
}

object Zone "29pzpl" {
  parent = "master"
  endpoints = [ "29livmoni1" ]
}

object Zone "32hbgt" {
  parent = "master"
  endpoints = [ "32livmoni1" ]
}

object Zone "33hegh" {
  parent = "master"
  endpoints = [ "33livmoni1" ]
}

object Zone "37magh" {
  parent = "master"
  endpoints = [ "37livmoni1" ]
}

object Zone "38hagt" {
  parent = "master"
  endpoints = [ "38livmoni1" ]
}

object Zone "43wagh" {
  parent = "master"
  endpoints = [ "43livmoni1" ]
}

object Zone "50jowe" {
  parent = "master"
  endpoints = [ "50livmoni1" ]
}

object Zone "51jobi" {
  parent = "master"
  endpoints = [ "51livmoni1" ]
}

object Zone "74kize" {
  parent = "master"
  endpoints = [ "74livmoni1" ]
}

object Zone "77tegh" {
  parent = "master"
  endpoints = [ "77livmoni1" ]
}

object Zone "98dmz" {
  parent = "master"
  endpoints = [ "98livmoni1-dmz" ]
}

object Zone "98zent" {
  parent = "master"
  endpoints = [ "98livmoni1", "98livmoni2" ]
}

object Zone "master" {
  endpoints = [ "98lipmoni1", "98lipmoni2" ]
}

object Zone "global" {
  global = true
}

zones.conf Master2:

/etc/icinga2# cat zones.conf
/*
 * Endpoint and Zone configuration for a cluster setup
 * This local example requires `NodeName` defined in
 * constants.conf.
 */

object Endpoint "01livmoni1" {
}

object Endpoint "03livmoni1" {
}

object Endpoint "04livmoni1" {
}

object Endpoint "07livmoni1" {
}

object Endpoint "08livmoni1" {
}

object Endpoint "09livmoni1" {
}

object Endpoint "10livmoni1" {
}

object Endpoint "11livmoni1" {
}

object Endpoint "12livmoni1" {
}

object Endpoint "13livmoni1" {
}

object Endpoint "14livmoni1" {
}

object Endpoint "15livmoni1" {
}

object Endpoint "16livmoni1" {
}

object Endpoint "19livmoni1" {
}

object Endpoint "20livmoni1" {
}

object Endpoint "20livmoni1-dmz" {
}

object Endpoint "21livmoni1" {
}

object Endpoint "22livmoni1" {
}

object Endpoint "26livmoni1" {
}

object Endpoint "27livmoni1" {
}

object Endpoint "29livmoni1" {
}

object Endpoint "32livmoni1" {
}

object Endpoint "33livmoni1" {
}

object Endpoint "37livmoni1" {
}

object Endpoint "38livmoni1" {
}

object Endpoint "43livmoni1" {
}

object Endpoint "50livmoni1" {
}

object Endpoint "51livmoni1" {
}

object Endpoint "74livmoni1" {
}

object Endpoint "77livmoni1" {
}

object Endpoint "98livmoni1" {
}

object Endpoint "98livmoni2" {
}

object Endpoint "98livmoni1-dmz" {
}

object Endpoint "98lipmoni1" {
  host = "98lipmoni1"
}

object Endpoint "98lipmoni2" {
  host = "98lipmoni2"
}

object Zone "01rost" {
  parent = "master"
  endpoints = [ "01livmoni1" ]
}

object Zone "03lueb" {
  parent = "master"
  endpoints = [ "03livmoni1" ]
}

object Zone "04gbgh" {
  parent = "master"
  endpoints = [ "04livmoni1" ]
}

object Zone "07flen" {
  parent = "master"
  endpoints = [ "07livmoni1" ]
}

object Zone "08hhba" {
  parent = "master"
  endpoints = [ "08livmoni1" ]
}

object Zone "09kiel" {
  parent = "master"
  endpoints = [ "09livmoni1" ]
}

object Zone "10nbgh" {
  parent = "master"
  endpoints = [ "10livmoni1" ]
}

object Zone "11kigh" {
  parent = "master"
  endpoints = [ "11livmoni1" ]
}

object Zone "12lugh" {
  parent = "master"
  endpoints = [ "12livmoni1" ]
}

object Zone "13zogh" {
  parent = "master"
  endpoints = [ "13livmoni1" ]
}

object Zone "14zmgh" {
  parent = "master"
  endpoints = [ "14livmoni1" ]
}

object Zone "15rnur" {
  parent = "master"
  endpoints = [ "15livmoni1" ]
}

object Zone "16haat" {
  parent = "master"
  endpoints = [ "16livmoni1" ]
}

object Zone "19szgh" {
  parent = "master"
  endpoints = [ "19livmoni1" ]
}

object Zone "20sodk" {
  parent = "master"
  endpoints = [ "20livmoni1" ]
}

object Zone "20dmz" {
  parent = "master"
  endpoints = [ "20livmoni1-dmz" ]
}

object Zone "21kodk" {
  parent = "master"
  endpoints = [ "21livmoni1" ]
}

object Zone "22mase" {
  parent = "master"
  endpoints = [ "22livmoni1" ]
}

object Zone "26tapl" {
  parent = "master"
  endpoints = [ "26livmoni1" ]
}

object Zone "27wapl" {
  parent = "master"
  endpoints = [ "27livmoni1" ]
}

object Zone "29pzpl" {
  parent = "master"
  endpoints = [ "29livmoni1" ]
}

object Zone "32hbgt" {
  parent = "master"
  endpoints = [ "32livmoni1" ]
}

object Zone "33hegh" {
  parent = "master"
  endpoints = [ "33livmoni1" ]
}

object Zone "37magh" {
  parent = "master"
  endpoints = [ "37livmoni1" ]
}

object Zone "38hagt" {
  parent = "master"
  endpoints = [ "38livmoni1" ]
}

object Zone "43wagh" {
  parent = "master"
  endpoints = [ "43livmoni1" ]
}

object Zone "50jowe" {
  parent = "master"
  endpoints = [ "50livmoni1" ]
}

object Zone "51jobi" {
  parent = "master"
  endpoints = [ "51livmoni1" ]
}

object Zone "74kize" {
  parent = "master"
  endpoints = [ "74livmoni1" ]
}

object Zone "77tegh" {
  parent = "master"
  endpoints = [ "77livmoni1" ]
}

object Zone "98dmz" {
  parent = "master"
  endpoints = [ "98livmoni1-dmz" ]
}

object Zone "98zent" {
  parent = "master"
  endpoints = [ "98livmoni1", "98livmoni2" ]
}

object Zone "master" {
  endpoints = [ "98lipmoni1", "98lipmoni2" ]
}

object Zone "global" {
  global = true
}

zones.conf satellite:

/etc/icinga2# cat zones.conf 
object Endpoint "98livmoni1-dmz" {
}

object Endpoint "98lipmoni1" {
  host = "98lipmoni1"
}

object Endpoint "98lipmoni2" {
  host = "98lipmoni2"
}

object Zone "98dmz" {
  parent = "master"
  endpoints = [ "98livmoni1-dmz" ]
}

object Zone "master" {
  endpoints = [ "98lipmoni1", "98lipmoni2" ]
}

object Zone "global" {
  global = true
}

Update:

after changing a service and reload on the config master everthing is ok. Very strange :grinning:

Hi all,
The Icinga2 version on the master is r2.14.0-1, and the version on the satellite is r2.11.2-1. I am experiencing a problem with my Icinga2 server; the configurations from the master won’t synchronize to the satellite anymore due to the error below. The system worked fine for a few months, but when I attempted to add another satellite to the system, I noticed that the configurations wouldn’t synchronize to the satellite.

Can you provide me with a hint on how to fix this error, please?
Note the hosts was run on the master use the same template and are fine!

Error.
[2023-11-10 16:51:29 -0600] critical/config: Error: Argument is not a callable object.
Location: in /var/lib/icinga2/api/zones-stage//wi-madison-imms/director/host_templates.conf: 1:6-1:45
/var/lib/icinga2/api/zones-stage//wi-madison-imms/director/host_templates.conf(1): if (! get_template(Host, “ga-camera-template”)) { template Host “ga-camera-template” {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//wi-madison-imms/director/host_templates.conf(2): import “camera_template”
/var/lib/icinga2/api/zones-stage//wi-madison-imms/director/host_templates.conf(3): import “dummy_template”

[2023-11-10 16:51:29 -0600] critical/cli: Config validation failed. Re-run with ‘icinga2 daemon -C’ after fixing the config.

root@imms2-ird-relay:/var/lib/icinga2# tail -f /var/log/icinga2/icinga2.log
[2023-11-10 16:51:28 -0600] information/ApiListener: Received configuration updates (11) from endpoint ‘imms2’ are different to production, triggering validation and reload.
[2023-11-10 16:51:29 -0600] critical/ApiListener: Config validation failed for staged cluster config sync in ‘/var/lib/icinga2/api/zones-stage/’. Aborting. Logs: ’

I upgraded the satellite to R2.14.0-1 to the same version with master. it works.