I’ve been using Icinga2 for clustered setups for quit a a few years. Normally I handle configuration using config files in zones.d/ and deploy these using Ansible or similar. Works fine. I have also worked with the API to dynamically add and remove hosts.
Now I have a setup where many services are configured from host variables. Problem is that these variables need to changes dynamically and it is not feasible to handle those changes by deploying new static configuration files.
Enter the API. There is no problem to update the the relevant host variables using the API. Problem is that I cannot in any way get the the services to actually re-evaluate the new host variable values. There are some (not that clear) disclaimers about this in the API documentation (in the Modifying Objects section). Also there are some issues indicating this on Github.
That’s fine but one would expect these updated variables to be re-evaluated at some point! Even a restart of the daemon doesn’t trigger any changes!
Just to be specific, say I have a host variable “a=1”, I then have a service using this variable as parameter to a check command, say “vars.http_uri = “/” + host.vars.a”. My idea is that changing “a=2” should, at some point" change the check URI to “/2”. That doesn’t happen.
I also looked into configuration packages but I didn’t really see how these would solve anything. By the way, what is the reason to expose different package stages when you cannot manually roll them back or anything?
As you’ve found out already, that is a missing feature not yet implemented. If you’re interested in more technical details, this is a tricky one:
You’ll need to store all affected changes by such an attribute. This can influence how apply rules generate other objects, but also group memberships. If things go wrong even further, there may variable assignments in other objects too from inside the DSL.
Once you modify such an attribute, you need to rollback everything created/assigned before.
Then modify the variable, and re-run everything which depends on this exact variable.
Since you may not know whether the user has created additional dependencies meanwhile, you’ll like just re-run the entire configuration validation for just a single object modification. Large scale environments take some minutes to do this, and it is not desirable for a system where you update many attributes at once and often on deployments.
You can of course try to combine such updates into a large transaction, and hoping that everything turns out good. The problem with that is that changes are not immediate, and users might wonder why the deployment update doesn’t show up in the web interface, or even the API itself.
A colleague of mine had a look at a possible implementation, turns out that there is no suitable one at this time. Since no-one ever since wanted to sponsor resources for solving this together, the priorities have shifted to more immanent things like the network stack, config syncs and additional features in and around the Icinga stack.
If you have possible idea and implementation proposal for this feature, please join the conversation in the linked GitHub issue and discuss.
Config packages help in a way that the configuration is fully managed as static DSL deployed via REST API, and then evaluated all at once. They won’t help with runtime moditications though, a restart/reload is still required.
Still, shouldn’t a change for a variable be applied if i stop and restart the Icinga daemon? In my case the host variable would still be changed (a=2) but the service var (http_uri=/1) isn’t updated. At least I expect that if the variable looks changed (value changed in IcingaWeb2) and daemon is restarted, the new value should propagate.
I then see the companyname.ec2_instance_id variable value change in Icingaweb2. Also, if I fetch the host using the API the value is correct.
I then restart icinga2 (service icinga2 restart on Debian).
After restart, host variable is set to the value updated using the API. At this point I also expect one service on the host to have updated check command parameters. The parameter is configured as:
Unfortunately Icingaweb2 still shows the parameter value to be InstanceId=i-1122334455 instead of the expected InstanceId=i-aabbccddee.
I might have gotten something wrong but in this case I really expect the command parameter on the service to have its value updated due to host variable value having changed AND Icinga2 has been restarted.
Understood, thanks. I think it may be the case that modified attributes are applied in the wrong order here, as seen in this issue. The POST request actually results in a modified attribute in the background.
Read that issue before my first post but obviously didn’t understand the implications. Soooo, the answer is that currently there is no way to do what I try to achieve if I don’t spend time with the code base?
Sounds like that. I am not sure yet how to fix this problem within the code and multiple compilation stages plus loading modified attributes in the proper way.
