Creating an Apply For rule across multiple variables

Icinga 2
1

Our team is just getting started with icinga2. We have an existing homegrown tool that checks a lot of different things depending on an argument so we have a series of CheckCommand objects that vary by only one argument. These checks need to be applied to multiple database instances across multiple hosts. Creating separate CheckCommands is working fine, like this:

object Host "some.oracle.host" {
  import "oracle-host"
  address = "172.1.1.1"

  vars.db_instances = {
    "EnvName" = {
      "username" = "username"
      "password" =  "EQo1OF95G8Gs"
      "dbname" =  "FOO"
      "connstr" = "jdbc:oracle:thin:@(FOO)"
    }
}

apply Service for (env => instance in host.vars.db_instances) {
  import "oracle-db-svc"
  check_command = "check_oracle_foocheck"
  display_name = env + "-foocheck"
  vars.svc_dbname = instance["dbname"]
}

apply Service for (env => instance in host.vars.db_instances) {
  import "oracle-db-svc"
  check_command = "check_oracle_barcheck"
  display_name = env + "-barcheck"
  vars.svc_dbname = instance["dbname"]
}

Here we are creating new “check_oracle_foocheck” and “check_oracle_barcheck” services for each instance of each host that has a db_instances variable. It works well but the problem is, we have a couple dozen of those checks to implement. I’m looking for something simpler, something like:

const OracleChecks = ["foocheck", "barcheck"]

apply Service for (check in OracleChecks) {
  import "oracle-db-svc"  
  for(env => instance in host.vars.db_instances){
    check_command = "$check$"
    display_name = env + "-$check$"
  }
}

I assume this wouldn’t work because the apply for rule isn’t actually returning and maybe you can’t iterate on hosts within an apply for. Here I’m just trying to convey the objective.

What I’m after is an approach to make sort of a matrix assignment of a set of services to sets of instances which live on multiple hosts. Any tips?

Andy

p.s. - Cross-posted to SOF

2

I’ve not got any bites here or or on SOF so I believe this isn’t really possible. I thought if a function could return an object then perhaps an apply could iterate over a loop something like this:

// disclaimer: this is sort of a mock example, it may not make sense in detail but hopefully
// conveys the general idea
apply Service for (env => instance in host.vars.db_instances) {
  import "oracle-db-svc"
  var.check_cmd = "check_oracle_barcheck"
  for(svc_check in const_list_checks) {
    var.display_name = env + "-" + svc_check + "-barcheck"
    vars.svc_dbname = instance["dbname"]
    fn_create_service(check_cmd,display_name)
  }
}

But something tells me that wouldn’t work. So at this point I’m proceeding to an API solution but if anyone ever runs across this and has an idea, please let me know!

Andy

3

Maybe this could be a starting point to achieve what you need. In my sample I’ve created two hosts with a simplified dictionary from your initial post. The const SomeChecks is an array with two elements (as your const OracleChecks).

Using for loops can be real fun :slight_smile:

image

Summary 
object Host "test-01" {
  import "generic-host"

  address = "127.0.0.1"

  vars.db_instances = {
    "inst01" = {
      "username" = "username"
    }
  }
}

object Host "test-02" {
  import "generic-host"

  address = "127.0.0.1"

  vars.db_instances = {
    "inst01" = {
      "username" = "username"
    },
    "inst02" = {
      "username" = "username"
    }
  }
}

const SomeChecks = [ "check1", "check2" ]

for ( checkname in SomeChecks ) {

  apply Service checkname + "-" for (env => instance in host.vars.db_instances) use(checkname) {
    import "generic-service"

    check_command = "ping4"
    display_name = env + "-" + checkname

    assign where host.address && match("test-*", host.name)
  }

}

Please note, the service’s object name starts with the check name from the const array. The display name is overwritten inside the apply rule, so you get your desired result in Icingaweb2.

For example, inst01-check1 is the object name of the service and (flipped parts) it’s check1-inst01 for the display name.
image

2 Likes
4

Very nice example :+1: . Just a few notes for better understanding:

  • const and for need to exist in the same scope and namespace
  • loops inside apply for loops don’t have any effect on object creation - each iteration of apply for generates a new object.
  • learn about scopes and binding them (“use”) with closures
  • host.vars.db_instances needs to be a “flat” dictionary whose key is the future service name. If that is not the case, you can create a function which does exactly that, mentioned in this blog post.
  • The above example doesn’t use instance which would assign the username key in there to something else, this will be the case for some database related checks then.

Cheers,
Michael

2 Likes
5

Ah, ok! Yes, there are numerous learnings in there for me that will help with this challenge. Wow, thank you @bernd! I’ll give this is a shot in my environment and see things go.

Andy

6

Thanks, @dnsmichi, I appreciate the help with the key elements of @bernd’s code. I’ll look into use, that seems like an important key to this one. I’m not sure about your comment about the “flat” dictionary but will likely understand it later this morning as I work through it… :wink:

Also, thanks for articulating the workings of “loops inside apply for loops”. That was my instinct but wasn’t sure how to describe it.

Andy

1 Like
7

This worked! :slight_smile: Thank you both for your tutelage!

One question: I don’t “get” why we need use. It seems that the apply would be within the scope of the definition of checkname. I didn’t try it without (coz it ain’t broke!) but is the use required? The documentation on use doesn’t have a lot to say about it though if I’m understanding it apply doesn’t inherit its scope as you would see in other coding constructs.

Andy

1 Like
8

Scopes

checkname is a scoped variable. It exists inside the for loop, but not inside other inner scopes, like an object or an apply rule.

for (c in [ "c1", "c2" ]) {
  object Host c { //this works, the declaration of the object is in the parent scope
    check_command = "dummy"
    vars.cv = c //this doesn't work, c is not available inside the object scope
  }
}

The DSL is very strict with scopes, this is to avoid accidental overrides from upper layers. One would not expect that a variable defined somewhere above would actually be global. Or - which source wins with many wrapped scopes?

Note

These loops might not be handy for production though. For developers and tests, they are pretty awesome, since we can generate many objects just with the DSL. The famous “many.conf” also exists inside the Vagrant boxes for that purpose :slight_smile:

I have an advanced example in progress for IcingaDB tests which also renders zone/endpoint memberships, but you know … time.

Global constants vs local variables

In contrast to that, a global variable or constant can be used, since globals is implicitly bound into an object/apply rule. this and locals isn’t.

Try that:

var c = "c1"

object Host c {
  check_command = "dummy"
  vars.cv = c
}

[2019-05-29 19:18:44 +0200] critical/config: Error: Error while evaluating expression: Tried to access undefined script variable 'c'
Location: in /etc/icinga2/tests/scope.conf: 5:13-5:13
/etc/icinga2/tests/scope.conf(3): object Host c {
/etc/icinga2/tests/scope.conf(4):   check_command = "dummy"
/etc/icinga2/tests/scope.conf(5):   vars.cv = c
                                              ^
/etc/icinga2/tests/scope.conf(6): }
/etc/icinga2/tests/scope.conf(7): 

[2019-05-29 19:18:44 +0200] critical/config: 1 error

vs

const c = "c1"

object Host c {
  check_command = "dummy"
  vars.cv = c
}

<works>

Agreed, the docs are a bit sparse on this … but you know, I cannot do everything :kissing_heart:

Imho one can only learn these advances techniques by discussing them here on the forums. I think it really is very hard to explain in the docs - if you look at the loop unrolling which tries to explain the apply for loop iterations … well, not happy with it, but at least something.

Cheers,
Michael

2 Likes
9

Thank you both for all the feedback and also the scope explanation :+1:

2 Likes
10

Very cool… I’m at the point in this where I can see a lot more of the learning curve ahead but at least now I can turn around and see something behind me besides dirt! :slight_smile: Thanks so much for the explanation!

Andy