Empty variable for services depending on host managed package

sysres-dev · July 24, 2019, 11:33am

Hello,
I’m in a testing environment with few hosts, with 2 icinga servers, one zone each, one master and one child, both are running r2.10.5-1

I’m not sure what behaviour to expect from following operations :

1. setup services/commands configuration in /etc/icinga2/zones.d/global-templates
1. setup host throught stage in a newly created package
1. change throught API at runtime one of the generated service variable depending on a host created in previous package
1. apply a change to the host in a new stage of the same package

The idea of those tests is to check if is it possible to change some tresholds at runtime for configuration managed with packages.

When i check for the service variables on icinga console, i still have the values which was previously set at step 3) (which is what i’m needing)
but when i’m looking at debug.log or last_check_result.command the variable disk_args is empty, like if it was not resolved. I made sure to restart and reload both master and satellites.

I’m not sure if this is an expected behaviour, but in a general way, i’d like to know more about runtime operations on configurations issued from packages, i’m probably missing something.
The general idea is to find a clean way to manage services variables at runtime depending on hosts managed in a package.

Here are all the related configurations :

etc configurations (step 1) :

/etc/icinga2/zones.d/global-templates/DASHBOARD/commands.conf

object CheckCommand "custom_check_disk" {
    command = [ SysconfDir + "/icinga2/scripts/custom_check_disk.py" ]
    arguments = {
        "-H" = {
            required = true
            value = "$address$"
        }
        "-D" = {
            required = true
            value = {{ Json.encode(host.vars.data["disk"]["global"] + macro("$disk_args$")) }}
        }
    }
}

/etc/icinga2/zones.d/global-templates/DASHBOARD/services.conf

apply Service "custom_check_disk" for (disk_args in host.vars.data["disk"].fordisks) {
    name = "custom_check_disk_" + disk_args.disk_name
    retry_interval = 5m
    max_check_attempts = 3
    check_interval = 5m
    retry_interval = 2m
    vars.disk_args = disk_args
    check_command = "custom_check_disk"
    assign where host.vars.data["disk"].fordisks != null
}

/etc/icinga2/zones.d/global-templates/CUSTOM/templates.conf

template Host "custom-host" {
  max_check_attempts = 3
  check_interval = 5m
  retry_interval = 2m
  check_command = "hostalive"
}

/etc/icinga2/zones.conf

object Zone "global-templates" {
        global = true
}

object Endpoint "icinga2-api.my.domain" {
  host = "icinga2-api.my.domain"
}

object Endpoint "icinga2-intranet.my.domain" {
  host = "icinga2-intranet.my.domain"
}

# master zone
object Zone "ressource" {
  endpoints = [ "icinga2-api.my.domain" ]
}

# child zone
object Zone "intranet" {
  endpoints = [ "icinga2-intranet.my.domain" ]
  parent = "ressource"
}

Package creation (step 2):

curl -s -k -u admin:password -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/config/packages/test-cmdb?pretty=1'

Stage creation (step 2):

curl -s -k -u admin:password -H 'Accept: application/json' -X POST -d '{ "files": { "zones.d/intranet/hosts.conf": "object Host \"package_test_1\" { import \"custom-host\" , address = \"1.2.3.4\" , vars = { data = { cpu = { global = { description = \"CPU\" , equipement = \"test-T303\" , name_path = \"CUST\" , param = \"5,15,30\" , script = \"custom_check_cpu\" , sub_dashboard = \"CUST_Cpu_test\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } }, disk = { fordisks = [ { description = \"DiskC\" , disk_name = \"C\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" }, { description = \"DiskD\" , disk_name = \"D\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } ], global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_disk\" , sub_dashboard = \"CUST_Disk_test\" } }, ram = { global = { description = \"RAM\" , equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_ram\" , sub_dashboard = \"CUST_Ram_test\" , threshold_critical = \"> 90\" , threshold_ok = \"< 50\" } }, service = { forservices = [ { description = \"Dfs,DFS Namespace\" , display_name = \"DFS Namespace\" , service_name = \"Dfs\" , type_service = \"auto_running\" }, { description = \"MSSQLSERVER,SQL Server (MSSQLSERVER)\" , display_name = \"SQL Server (MSSQLSERVER)\" , service_name = \"MSSQLSERVER\" , type_service = \"auto_running\" }, { description = \"wuauserv,Windows Update\" , display_name = \"Windows Update\" , service_name = \"wuauserv\" , type_service = \"disabled_stopped\" } ], global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_service\" , sub_dashboard = \"CUST_Service_test\" } } } }, zone = \"intranet\"},object Host \"package_test_DPH_2\" { import \"custom-host\" , address = \"1.2.3.4\" , vars = { data = { cpu = { global = { description = \"CPU\" , equipement = \"test-T303\" , name_path = \"CUST\" , param = \"5,15,30\" , script = \"custom_check_cpu\" , sub_dashboard = \"CUST_Cpu_test\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } } , disk = { fordisks = [ { description = \"DiskC\" , disk_name = \"C\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" }, { description = \"DiskD\" , disk_name = \"D\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } ] , global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_disk\" , sub_dashboard = \"CUST_Disk_test\" } } , ram = { global = { description = \"RAM\" , equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_ram\" , sub_dashboard = \"CUST_Ram_test\" , threshold_critical = \"> 90\" , threshold_ok = \"< 50\" } } , service = { forservices = [ { description = \"Dfs,DFS Namespace\" , display_name = \"DFS Namespace\" , service_name = \"Dfs\" , type_service = \"auto_running\" }, { description = \"MSSQLSERVER,SQL Server (MSSQLSERVER)\" , display_name = \"SQL Server (MSSQLSERVER)\" , service_name = \"MSSQLSERVER\" , type_service = \"auto_running\" }, { description = \"wuauserv,Windows Update\" , display_name = \"Windows Update\" , service_name = \"wuauserv\" , type_service = \"disabled_stopped\" } ] , global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_service\" , sub_dashboard = \"CUST_Service_test\" } } } } , zone = \"intranet\"}" }, "pretty": true }' 'https://localhost:5665/v1/config/stages/test-cmdb'

Runtime service variable change (step 3):

curl -s -k -u admin:password -H 'Accept: application/json' -X POST 'https://localhost:5665/v1/objects/services/package_test_1!custom_check_disk_C' -d '{ "attrs": { "vars": {"service_args":{"description":"DiskC","disk_name ":"C","threshold_critical":"> 90","threshold_ok":"< 90"}}}, "pretty": true }'

2nd Stage creation (step 4):

curl -s -k -u admin:password -H 'Accept: application/json' -X POST -d '{ "files": { "zones.d/intranet/hosts.conf": "object Host \"package_test_1\" { import \"custom-host\" , address = \"1.2.3.4\" , vars = { data = { cpu = { global = { description = \"CPU\" , equipement = \"test-T303\" , name_path = \"CUST\" , param = \"5,15,30\" , script = \"custom_check_cpu\" , sub_dashboard = \"CUST_Cpu_test\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } }, disk = { fordisks = [ { description = \"DiskC\" , disk_name = \"C\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" }, { description = \"DiskD\" , disk_name = \"D\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } ], global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_disk\" , sub_dashboard = \"CUST_Disk_test\" } }, ram = { global = { description = \"RAM\" , equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_ram\" , sub_dashboard = \"CUST_Ram_test\" , threshold_critical = \"> 90\" , threshold_ok = \"< 50\" } }, service = { forservices = [ { description = \"Ffs,FFS Namespace\" , display_name = \"FFS Namespace\" , service_name = \"Ffs\" , type_service = \"auto_Funning\" }, { description = \"MSSQLSERVER,SQL Server (MSSQLSERVER)\" , display_name = \"SQL Server (MSSQLSERVER)\" , service_name = \"MSSQLSERVER\" , type_service = \"auto_running\" }, { description = \"wuauserv,Windows Update\" , display_name = \"Windows Update\" , service_name = \"wuauserv\" , type_service = \"disabled_stopped\" } ], global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_service\" , sub_dashboard = \"CUST_Service_test\" } } } }, zone = \"intranet\"},object Host \"package_test_DPH_2\" { import \"custom-host\" , address = \"1.2.3.4\" , vars = { data = { cpu = { global = { description = \"CPU\" , equipement = \"test-T303\" , name_path = \"CUST\" , param = \"5,15,30\" , script = \"custom_check_cpu\" , sub_dashboard = \"CUST_Cpu_test\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } } , disk = { fordisks = [ { description = \"DiskC\" , disk_name = \"C\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" }, { description = \"DiskD\" , disk_name = \"D\" , threshold_critical = \"> 80\" , threshold_ok = \"< 80\" } ] , global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_disk\" , sub_dashboard = \"CUST_Disk_test\" } } , ram = { global = { description = \"RAM\" , equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_ram\" , sub_dashboard = \"CUST_Ram_test\" , threshold_critical = \"> 90\" , threshold_ok = \"< 50\" } } , service = { forservices = [ { description = \"Dfs,DFS Namespace\" , display_name = \"DFS Namespace\" , service_name = \"Dfs\" , type_service = \"auto_running\" }, { description = \"MSSQLSERVER,SQL Server (MSSQLSERVER)\" , display_name = \"SQL Server (MSSQLSERVER)\" , service_name = \"MSSQLSERVER\" , type_service = \"auto_running\" }, { description = \"wuauserv,Windows Update\" , display_name = \"Windows Update\" , service_name = \"wuauserv\" , type_service = \"disabled_stopped\" } ] , global = { equipement = \"test-T303\" , name_path = \"CUST\" , script = \"custom_check_service\" , sub_dashboard = \"CUST_Service_test\" } } } } , zone = \"intranet\"}" }, "pretty": true }' 'https://localhost:5665/v1/config/stages/test-cmdb'

Verifications :
on Node icinga2-intranet.my.domain :

<1> => get_service("package_test_1","custom_check_disk_C").vars
{
        service_args = {
                description = "DiskC"
                "disk_name " = "C"
                threshold_critical = "> 90"
                threshold_ok = "< 90"
        }
}
<2> => get_service("package_test_1","custom_check_disk_C").last_check_result.command
[ "/etc/icinga2/scripts/custom_check_disk.py", "-D", "{\"equipement\":\"test-T303\",\"name_path\":\"CUST\",\"script\":\"custom_check_disk\",\"sub_dashboard\":\"CUST_Disk_test\"}", "-H", "1.2.3.4" ]
<3> => get_host("package_test_1").vars["data"]["disk"]
{
        fordisks = [ {
                description = "DiskC"
                disk_name = "C"
                threshold_critical = "> 80"
                threshold_ok = "< 80"
        }, {
                description = "DiskD"
                disk_name = "D"
                threshold_critical = "> 80"
                threshold_ok = "< 80"
        } ]
        global = {
                equipement = "test-T303"
                name_path = "CUST"
                script = "custom_check_disk"
                sub_dashboard = "CUST_Disk_test"
        }
}

Debug.log confirmation :

[2019-07-23 19:18:54 +0200] debug/CheckerComponent: Scheduling info for checkable 'package_test_1!custom_check_disk_C' (2019-07-23 19:18:54 +0200): Object 'package_test_1!custom_check_disk_C', Next Check: 2019-07-23 19:18:54 +0200(1.5639e+09).
[2019-07-23 19:18:54 +0200] debug/CheckerComponent: Executing check for 'package_test_1!custom_check_disk_C'
[2019-07-23 19:18:54 +0200] debug/Checkable: Update checkable 'package_test_1!custom_check_disk_C' with check interval '300' from last check time at 2019-07-23 19:13:56 +0200 (1.5639e+09) to next check time at 2019-07-23 19:23:52 +0200(1.5639e+09).
[2019-07-23 19:18:54 +0200] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2019-07-23 19:18:54 +0200] notice/ApiListener: Sending message 'event::SetNextCheck' to 'icinga2-api.my.domain'
[2019-07-23 19:18:54 +0200] notice/Process: Running command '/etc/icinga2/scripts/custom_check_disk.py' '-D' '{"equipement":"test-T303","name_path":"CUST","script":"custom_check_disk","sub_dashboard":"CUST_Disk_test"}' '-H' '1.2.3.4': PID 17639

dnsmichi · July 26, 2019, 12:11pm

I’ve been reading the text and output 3 times now and I couldn’t figure out which parameter is causing trouble here. Can you pinpoint the examples and explain their intent plus expected behaviour? The overall example seems to be really complex.

(might also be the heat over here in Germany )

sysres-dev · July 26, 2019, 5:21pm

Thank you for your answer and your time.

After a lot more testing, i found the error, it was a typo in runtime service variable change at step 3 (service_args key instead of disk_args in dictionnary in curl), i’m sorry for this mistake, shame on me

Now i can confirm that service vars runtime changes take precedence over vars defined in package and is persistent after a restart. On overall, it works fine to change package based configuration at runtime.

( it is also hot here )