we try to automate our environment regarding tresholds.
We have an agent/master based setup and only use servicesets and assign rules to assing checks to our hosts with the help of the director.
We would like to periodically run scripts on the Icinga Agent machines, to set specific tresholds for checks like check_mysql. Every mysql instance can have different settings (i.e. max_connections) and they can change from time to time.
We tried to set the tresholds via director API from the script.
First approach was to just set the warn/crit treshold variable from the check_plugin like:
vars.mysql_health_warning =100 for the specific host, but it seems not to work for servicesets but for single servicees (which we dont want to use). I think thats the reason the director handles overrides in servicesets like this:
So the second approach was to set the override in the same way, you can see in the director with the _override_servicevars variable. But we had no luck to make it work.
3rd approch was to send the same override request, we saw in the networkwork tab of the browser, during the override in the browser. This leads to:
curl “https://icinga2.DOMAIN.int/director/host/servicesetservice?name=op-icn2db-p101.DOMAIN.int&service=MySQL"%“20threads-connected&set=MySQL”%"20Cluster” -H “User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0” -H “Accept: /” -H “Accept-Language: en-US,en;q=0.5” --compressed -H “Content-Type: application/x-www-form-urlencoded; charset=UTF-8” -H “X-Icinga-Accept: text/html” -H “X-Icinga-WindowId: ubqcxtjhsgnv_cpikbz” -H “X-Requested-With: XMLHttpRequest” -H “DNT: 1” -H “Connection: keep-alive” -H “Referer: https://icinga2.DOMAIN.int/director/hosts?q=icn2” -H “Cookie:********” --data “__FORM_NAME=IcingaModuleDirectorFormsIcingaServiceForm&__FORM_CSRF=1431321837”%“7C0dcf81a6464038a7e6fe390f38132032cd1aec5af8ed9bd424d6b2cbfd63281a&var_mysql_health_critical=&var_mysql_health_warning=777&var_mysql_health_hostname=&var_mysql_health_mode=&var_mysql_health_password=&var_mysql_health_port=&var_mysql_health_statefilesdir=&var_mysql_health_username=&var_mysql_health_database=&var_mysql_health_name=&var_mysql_health_name2=&Overridevars=Override+vars”
This worked fine becasue of te Cookie, but as soon as I used our API User for the director, we just got HTTP 302 and a redireict to the login page. Even after the API User got full administrative access, we can not change anything with the above URL.
We are lost. Dynamic tresholds would be really helpful with about 2000 hosts and 30k servicechecks.
Usually the inheritance of threshold variables for services starts with command definition and will be overwritten by settings in service templates, apply rules, service sets, individual host values takes highest precedence.
Let’s say you set a mysql_max_conn variable per host and set individual thresholds, then the service check command should use this value, regards of the way they are assigned (service, service apply rule, service set). Did you verify that you use the same variable name for the host var and the var which is used in command definition?
Do you use the commands provided by Icinga2 ITL or your own command definitions?
The ITL command definitions are imported by Director kickstart assistant and are show as
external commands. These commands import default variables/settings which are not editable
within Director until you create these variables in Director.
I’m not using this way of inheritance until now, I do it by applying checks with different settings/thresholds based on host variables. Maybe you need to access this variable as host.vars.XXXX, there was a similar post on the old forum. Then you can set the threshold by setting the service variable to the hosts variable - presupposes that each host has set the threshold.
I copied the serviceset and changed the variable to $host.vars.mysql_health_critical$. And holy cow, it worked! This is a huge step forward to our goal.
Now we need to set the serv ice variable to the host variable as you mentioned. Is it possible to do it in the director? The solution @dnsmichi mentioned seems not to be possible in the director. What whould be a clean solution here?