How to organize configuration files in a HA distributed system?

I am currently trying to set up a test environment for learning how to manage a highly available distributed system, but am struggling a bit how to manage the configuration files (zones.conf and zones.d)

The test environment will look something like this:

  • 2 Masters (HA)
  • 2 Satellites
  • 2 Agents

The 2 masters are supposed to be in a Master zone with HA enabled.
Then we have 2 satellites beneath that which is parent to 1 agent each.

I will also be using Director to manage configuration files where this is possible.

What i am struggling most with i think is the configuration files, and which to use and how to manage it best.

For example, first of all you are supposed to disable the use of the conf.d folder and all the (example) configuration files inside that folder?

Do i use the zones.conf file, the zones.d folder or should i use them both?

I read somewhere that you use the folder zones.d to make subfolders for each of your zones.
Does this mean i have to make 3 folders inside zones.d in a flat structure (master, satellite1, satellite2), or should i have one folder master that contains two folders (satellite1 and satellite2) inside.

Also have some configuration related questions:

  • Should i use Director only for modifying config files, or can i combine it with manual editing?
  • If i modify config files manually, do i then have to run Kickstart inside Director afterwards?
  • If i mess up something, what is the best way to reset and try again? (which files should i delete)

Hope this is understandable and that someone can help me in the right direction to sort this out :slight_smile:

There is a config master the first one you set up normally.
The config file zones.conf is used to manage the connections so you define endpoints and zones there on every node. Yes flat folders for every zone in zones.d but most probably only the master is needed as the director will manage most if not all of the rest for you.

  • I mix Director and config but only if needed - notifications, API Users.
  • yes if you want the director to know about your change
  • If you do plan on messing with configuration and nightly backups aren’t enough I would use Git to manage the local config outside of the Director

Thank you for your answer :slight_smile:

Got some followup questions to clarify:

  • So the zones.conf file should be present on every node, and this should include endpoints and zones for the entire environment?

  • Will zones.conf be managed by Director after initial Kickstart, so that every node gets an updated zones.conf file?

  • After initially and manually creating the subfolders for zones in zones.d (on Master node only) this will be managed by director after initial kickstart?

…and lastly, in case i should manage to mess things up (configuration files):

  • I can just redefine zones.conf and zones.d on the config master, and run Kickstart in Director afterwards to start from scratch? (No need to delete anything defined in the GUI Icinga Web?)
  • yes and no - it needs to be present on every node but only needs the masters, the master zone and the global zones on the masters (not sure about satellite zones). I only define the local endpoint and zone, the parent endpoints and zone and again the global zones an a normal node.
  • I don’t think so but it builds a zones.conf in the agent tab of the host for you to deploy
  • Yes, have a look at /var/lib/icinga2/api/zones/master/director/agent_endpoints.conf and /var/lib/icinga2/api/zones/master/director/agent_zones.conf
  • well certain stuff needs to match at certain points like zones, api users and the like
    if you want to start form scratch, I can imagine if a kickstart isn’t enough emptying the Director DB could help