We’re looking to move to a distributed setup, where we would have a satellite per environment, all reporting up to one master. Today, we have a master per environment and it’s getting a bit unwieldily.
What I’m having trouble wrapping my head around is when checks needs credentials, and those vary by environment. Currently, we pass in these credentials using variables that are set in either the host object, or the service apply object. Since we have a master per environment, this works fine today.
It’s not clear to me how you would deal with this in a distributed setup, especially in top down config sync where all config lives on the master.
How are people dealing with this? Have checks source their credentials from something outside icinga? Is it possible to have some config on the satellites that’s then unique per satellite/environment?
Makes sense! Do these templates live on the master? Or do you drop these on the satellites only? IE: Does the master itself still have all the configuration on it, which includes the credentials?
I think what we’ll end up doing is defining the credentials as environment variables on our satellites or agents. We’ll make these accessible to icinga using systemd’s EnvironmentFile/PassEnvironment, and then reference them using the env runtime macro which should be evaluated wherever the check is actually executed.
The reason for this is we don’t want credentials for all environments to live on disk on a single box, if the master gets compromised we don’t want all our environments to be at risk due to credential theft.
Lots of different solutions here I guess, this one will suit our needs.