Vars override in serviceset

Hi,

we try to automate our environment regarding tresholds.
We have an agent/master based setup and only use servicesets and assign rules to assing checks to our hosts with the help of the director.

We would like to periodically run scripts on the Icinga Agent machines, to set specific tresholds for checks like check_mysql. Every mysql instance can have different settings (i.e. max_connections) and they can change from time to time.
We tried to set the tresholds via director API from the script.

First approach was to just set the warn/crit treshold variable from the check_plugin like:
vars.mysql_health_warning =100 for the specific host, but it seems not to work for servicesets but for single servicees (which we dont want to use). I think thats the reason the director handles overrides in servicesets like this:

vars["_override_servicevars"] = {

"MySQL threads-connected" = { 	 	 
    mysql_health_warning = "666" 	 	 
} 	 	 

}

So the second approach was to set the override in the same way, you can see in the director with the _override_servicevars variable. But we had no luck to make it work.

3rd approch was to send the same override request, we saw in the networkwork tab of the browser, during the override in the browser. This leads to:
curl “https://icinga2.DOMAIN.int/director/host/servicesetservice?name=op-icn2db-p101.DOMAIN.int&service=MySQL"%“20threads-connected&set=MySQL”%"20Cluster” -H “User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0” -H “Accept: /” -H “Accept-Language: en-US,en;q=0.5” --compressed -H “Content-Type: application/x-www-form-urlencoded; charset=UTF-8” -H “X-Icinga-Accept: text/html” -H “X-Icinga-WindowId: ubqcxtjhsgnv_cpikbz” -H “X-Requested-With: XMLHttpRequest” -H “DNT: 1” -H “Connection: keep-alive” -H “Referer: https://icinga2.DOMAIN.int/director/hosts?q=icn2” -H “Cookie:********” --data “__FORM_NAME=IcingaModuleDirectorFormsIcingaServiceForm&__FORM_CSRF=1431321837”%“7C0dcf81a6464038a7e6fe390f38132032cd1aec5af8ed9bd424d6b2cbfd63281a&var_mysql_health_critical=&var_mysql_health_warning=777&var_mysql_health_hostname=&var_mysql_health_mode=&var_mysql_health_password=&var_mysql_health_port=&var_mysql_health_statefilesdir=&var_mysql_health_username=&var_mysql_health_database=&var_mysql_health_name=&var_mysql_health_name2=&Overridevars=Override+vars”

This worked fine becasue of te Cookie, but as soon as I used our API User for the director, we just got HTTP 302 and a redireict to the login page. Even after the API User got full administrative access, we can not change anything with the above URL.

We are lost. Dynamic tresholds would be really helpful with about 2000 hosts and 30k servicechecks.

ENV:
Ubuntu 18.04
icinga r2.10.5-1
director 1.6.2

Thanks for your help.

Usually the inheritance of threshold variables for services starts with command definition and will be overwritten by settings in service templates, apply rules, service sets, individual host values takes highest precedence.

Let’s say you set a mysql_max_conn variable per host and set individual thresholds, then the service check command should use this value, regards of the way they are assigned (service, service apply rule, service set). Did you verify that you use the same variable name for the host var and the var which is used in command definition?

2 Likes

Hi @a1mw ,

thanks for your response.
That was exactly our understanding, too. But it seems not to work :frowning:

This is our simple Test:
zones.d/master-zone/hosts.conf
object Host “db-master” {
import “Linux Standard by ssh”

    display_name = "db-master"
    address = "1.2.3.4"
    check_command = "hostalive"
    vars.bereitschaft = "mag"
    vars.codename = "squeeze"
    vars.environment = "prod"
    vars.mysql_health_critical = "999"
    vars.plattform = "vm"
    vars.services = [ "mysql_cluster" ]
}

Custom Host Vars from IcingaWeb Overview
image

Custom Service Vars from IcingaWeb (after clicken the connected_threads service)

Executed command (after clicking Inspect):
‘/usr/lib/nagios/plugins/check_by_ssh’ ‘-C’ ‘’’’/usr/lib/nagios/plugins/check_mysql_health’’’ ‘’’–critical’’’ ‘’‘900’’’ ‘’’–mode’’’ ‘’‘threads-connected’’’ ‘’’–password’’’ ‘’‘XXXXX’’’ ‘’’–port’’’ ‘’‘3306’’’ ‘’’–statefilesdir’’’ ‘’’/tmp/’’’ ‘’’–username’’’ ‘’‘XXXXX’’’ ‘’’–warning’’’ ‘’‘750’’’’ ‘-H’ ‘1.2.3.4’ ‘-o’ ‘StrictHostKeyChecking=no’ ‘-t’ ‘15’

The Host got the right 999. But as soon as I look into the service, I can see 900, what is the value from the standard in the service template.

Any ideas?

Thanks and regards

Do you use the commands provided by Icinga2 ITL or your own command definitions?

The ITL command definitions are imported by Director kickstart assistant and are show as
external commands. These commands import default variables/settings which are not editable
within Director until you create these variables in Director.

I’m not using this way of inheritance until now, I do it by applying checks with different settings/thresholds based on host variables. Maybe you need to access this variable as host.vars.XXXX, there was a similar post on the old forum. Then you can set the threshold by setting the service variable to the hosts variable - presupposes that each host has set the threshold.

2 Likes

Indeed we are using the variables provided by Icinga ITL.
This is the custom variable output for myql_health_critial from director:

I copied the serviceset and changed the variable to $host.vars.mysql_health_critical$. And holy cow, it worked! This is a huge step forward to our goal.

Now we need to set the serv ice variable to the host variable as you mentioned. Is it possible to do it in the director? The solution @dnsmichi mentioned seems not to be possible in the director. What whould be a clean solution here?

Thanks you so far for your great help!