I’ll add some thoughts and findings here. I will not repeat what’s written in the Changelog nor upgrading docs, read them prior to upgrading.
Cluster Config Sync
This has been made more robust with 2.11. Some changes may now unveil wrong or broken configuration which “somehow” worked before, but not all the time.
Read about the technical design here: https://github.com/Icinga/icinga2/issues/6716
Stage before Prod
As you have read in the upgrading docs, there’s a stage before the production directory, called zones-stage. The received cluster message JSON is extracted and the config files are put in there first.
The old buggy timestamp and file content comparison for detecting config changes has been moved into a checksum calculated version. This for example ensures that NTP time sync issues do not cause nodes to always restart.
For upgrading purposes, we recommend to do the masters first, then satellites, then agents. The other way around has been tested, but may unveil edge cases where it doesn’t work.
In order to troubleshoot this, follow this new docs entry: https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#new-configuration-does-not-trigger-a-reload
Binary Scripts in the Config Sync
Never supported, we know that some users kept doing it. With 2.11, we strictly forbid this on the master as the config sync with checksums would always result in a loop.
Details: https://github.com/Icinga/icinga2/issues/7382
Troubleshooting: https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#syncing-binary-files-is-denied
Syncing Zones in Zones
If you are using the Director, you may have attempted to manage zones in there, and not in your local zones.conf file during master/satellite setup.
One thing you must not configure inside the Director itself, are master/satellite and global zones. They need to exist on the agent before any config sync happens. Otherwise you have a chicken egg problem.
The config compiler is now only including directories where a zone has been configured. Otherwise it would include renamed old zones, broken zones, etc. which is not wanted.
Solve this with putting the required parts into zones.conf on the agent: Icinga Update 2.11; Director Commands not available
Similar chicken egg problem with satellites syncing the agent zones, and the agents syncing their own local config (not command_endpoint): https://github.com/Icinga/icinga2/issues/7519
Example
A more concrete example: Masters and Satellites still need to know the Zone hierarchy outside of zones.d synced configuration.
Doesn’t work
vim /etc/icinga2/zones.conf
object Zone "master" {
endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
}
vim /etc/icinga2/zones.d/master/satellite-zones.conf
object Zone "satellite" {
endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite1.localdomain" ]
}
vim /etc/icinga2/zones.d/satellite/satellite-hosts.conf
object Host "agent" { ... }
The agent host object will never reach the satellite, since the master does not have the satellite zone configured outside of zones.d.
Works
Each instance needs to know this, and know about the endpoints first:
vim /etc/icinga2/zones.conf
object Endpoint "icinga2-master1.localdomain" { ... }
object Endpoint "icinga2-master2.localdomain" { ... }
object Endpoint "icinga2-satellite1.localdomain" { ... }
object Endpoint "icinga2-satellite2.localdomain" { ... }
Then the zone hierarchy as trust and also config sync inclusion is required.
vim /etc/icinga2/zones.conf
object Zone "master" {
endpoints = [ "icinga2-master1.localdomain", "icinga2-master2.localdomain" ]
}
object Zone "satellite" {
endpoints = [ "icinga2-satellite1.localdomain", "icinga2-satellite1.localdomain" ]
}
Once done, you can start deploying actual monitoring objects into the satellite zone.
vim /etc/icinga2/zones.d/satellite/satellite-hosts.conf
object Host "agent" { ... }
That’s also explained and described in the documentation: https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#three-levels-with-masters-satellites-and-agents
The thing you can do: For command_endpoint agents like inside the Director: Host -> Agent -> yes, there is no config sync for this zone in place. Therefore it is valid to just sync their zones via the config sync.
TL;DR - with using the Director, its cluster zones and agent hosts, you are safe. Manage the master/satellite instances outside in zones.conf and import them via kickstart wizard.
Config Compiler
If you put a host/service object into conf.d or anywhere else, and use command_endpoint, Icinga sometimes refused to check and update its object authority. In order to ensure that these checks run, the config compiler now breaks on objects which do not have the zone attribute specified. Apparently the log message is just "command endpoint must not be set` which needs to be improved.
In order to fix this, ensure that remote command endpoint checks are only defined in zones.d/<zonename>. If you only have a single master, use its zone directory.
Discussion: https://github.com/Icinga/icinga2/issues/7514