We have ~800 hosts and about half of them need some checks over a REST API running locally on the checked host, that are done by self-written plugins (called by ssh). These are using credetials to access the API, which are different on each host and stored in an external database. The icinga2 configuration is saved in our internal gitlab repo, so I would prefer not to save all these passwords in clear text.
These could either be written by cron in a separate file that is outside of the git checkout (in variables containing the host name), or maybe even directly queried from there by a script…
Does anyone have a similar situation? How would you go about getting these credentials into the icinga2 config? (I know these will land in all kind of files and in the postgres db on the monitoring hosts, but then at least they are somehow protected by the system rights and not stored in the gitlab repo)
As Icinga does not need the credentials but the plugins, I would first look for an option to get the credentials directly in the plugin.
If the risk is too high that you can get all the passwords then from all hosts, I would create a wrapper plugin on the Icinga system that gets the credentials from the database and then performs the original plugin via ssh.
If this is too much load on the database, slows down the plugin too much or something similar with negative performance impact, I would introduce a cache for the credentials in some way.
contrary to the suggestion of @dgoetz we decided to do this with hidden files and keep the the scripts clean from passwords. Every script has to catch the credentials first and after run the check. The hidden file can be protected with file permission and SELinux.
With this way we can use diffrent passwords and also we can transfer to other servers and send it to other colleagues