Different kinds of service/http checks in multiple servers

Hi Team,

In my environment, we have almost 200-300 servers are there. I have been struggling to monitor service & http checks in multiple servers… Currently for every host i have been adding one separate host config file… For example i have got the requirement to monitor A & B services that are exists in all servers. A & B both services exists in around 50 servers. A service exists in 70 servers & B service exists in 80 Servers.
This is so much of work if i do update custom attribute in every host conf file… Here the configuration files looks like.

Host.config file

object Endpoint "<hostname>" {
}

object Zone "<hostname>" {
     endpoints = [ "<hostname>" ]
     parent = "master"
}


// Host Objects
object Host "<hostname>" {
    check_command = "hostalive"
    address = "*********"
    vars.os="Windows"
     vars.service1 ="A"
    vars.service2 ="B"
    vars.client_endpoint = name //follows the convention that host name == endpoint name
// Custom Optional check - START
    vars.local_disks["/app Filesystem"] = {
       disk_partitions = "/app"
  }

    vars.local_http_vhosts["http"] = {
       http_uri = "/"
  }
  vars.nscp_boot = false
    vars.notification["mail"] = {
        groups = [ "icingausers" ]
}
    vars.local_tcp_port["tcp"] ={
       tcp_port = "23"
       service_name = "Telnet Check"
       port_number = "Port 23"

Server config file

apply Service "A-service" {
  check_command = "service-windows"
  vars.service_win_service = "A"
  notes = "Service check for A"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint && host.vars.service1 == "A"
}
apply Service "B-service" {
  check_command = "service-windows"
  vars.service_win_service = "B"
  notes = "Service check for B"
  command_endpoint = host.vars.client_endpoint
  assign where host.vars.client_endpoint && host.vars.service2 == "B"
}

If i do this way… This is taking so much of work & this is the only way we have?

Can someone please advise me the best way of monitoring services in multiples servers? thanks in advance

Hi,

AFAIK we had this discussion in another topic already, where I had suggested to hide these things in host templates. With importing the host templates into specific hosts, you can create generic apply rules.

Templates and Apply

template Host "basic-host" {
  check_command = "hostalive"
}

template Host "windows-host-service-A" {
  vars.service = "A"
}

template Host "windows-host-service-B" {
   vars.service = "B"
}

template Host "windows-host-service-all" {
  import "windows-host-service-A"
  import "windows-host-service-B"
}

object Host "host-A" {
  import "base-host"
  import "windows-host-service-A" 
}

object Host "host-B" {
  import "base-host"
  import "windows-host-service-B" 
}

object Host "host-all" {
  import "base-host"
  import "windows-host-service-all"
}

Base on the template tree, you’ll automatically get certain attributes and the imported templates stored in the templates attribute.

This detail can be used to create generic service apply rules.

apply Service "service-A" {
  check_command = "service-windows"
  vars.service_win_service = host.vars.service //access the host custom vars

  command_endpoint = host.vars.client_endpoint

  assign where host.vars.service == "A" && host.vars.client_endpoint
}

apply Service "service-B" {
  check_command = "service-windows"
  vars.service_win_service = host.vars.service //access the host custom vars

  command_endpoint = host.vars.client_endpoint

  assign where host.vars.service == "B" && host.vars.client_endpoint
}

Advanced: Apply for

Based on the example above, you can use an array with service names hidden in the host template tree.
The idea is to only have one apply for rule, and the hosts as information provider with host.vars.services. Bonus: Arrays can be extended from template inheritance.

template Host "basic-host" {
  check_command = "hostalive"
}

template Host "windows-host-service-A" {
  vars.services += [ "A" ] //note the additive assignment operator, and also adding a new array element
}

template Host "windows-host-service-B" {
  vars.services += [ "B" ] //note the additive assignment operator, and also adding a new array element
}

template Host "windows-host-service-all" {
  import "windows-host-service-A"
  import "windows-host-service-B"
}

object Host "host-A" {
  import "base-host"
  import "windows-host-service-A" 
}

object Host "host-B" {
  import "base-host"
  import "windows-host-service-B" 
}

object Host "host-all" {
  import "base-host"
  import "windows-host-service-all"
}

This technique uses a little DSL trick: Whether services is initialized as array already, it adds a new array element with the additive assignment operator +=. If the value was previously undefined, the added array element initializes this variable as array, with one element.

The resulting host objects now either have one array element with A or B, and the host-all object has [ "A", "B" ].

You can test such things in the debug console:

$ icinga2 console
Icinga 2 (version: v2.10.5-854-gf72ef9360)
Type $help to view available commands.
<1> => var services
null
<2> => services
null
<3> => services += [ "A" ]
null
<4> => services
[ "A" ]
<5> => services += [ "B" ]
null
<6> => services
[ "A", "B" ]

This detail can be used to build an apply for loop over this host scoped custom variable array.

apply for "service-" (s in host.vars.services) {
  check_command = "service-windows"

  vars.service_win_service = s //take the loop iteration variable value

  command_endpoint = host.vars.client_endpoint

  assign where host.vars.client_endpoint //In addition to host.vars.services being not empty, this condition must be met as well.
}

Specific notes, excerpt from the docs:

  • service- + the loop iteration variable value for s results in the final name
  • host.vars.services contains the array elements as service name strings
  • s can be used inside the apply for context, e.g. to create the display_name or, more importantly, to assign it to a custom variable required as parameter.
  • the apply for loop will only be evaluated if host.vars.services contains 1+ elements. In addition to creating service objects, the client_endpoint custom variable must also be set. This is checked again with assign where.

Note: I did not test the above, may contain minor typos. You’ll get the idea though.

Cheers,
Michael

yes… Now i understood… thanks a lot… One question as you mentioned object definitions like

object Host “host-A” {
import “base-host”
import “windows-host-service-A”
}

object Host “host-B” {
import “base-host”
import “windows-host-service-B”
}

object Host “host-all” {
import “base-host”
import “windows-host-service-all”
}

is this object defintions are just like host confi files that made? I have to do the template import in the below host config file for all servers? I have made 200 config files for 200 servers…

object Endpoint “” {
}

object Zone “” {
endpoints = [ “” ]
parent = “master”
}

// Host Objects
object Host “” {
check_command = “hostalive”
address = “*********”
vars.os=“Windows”
vars.service1 =“A”
vars.service2 =“B”
vars.client_endpoint = name //follows the convention that host name == endpoint name
// Custom Optional check - START
vars.local_disks["/app Filesystem"] = {
disk_partitions = “/app”
}

vars.local_http_vhosts["http"] = {
   http_uri = "/"

}
vars.nscp_boot = false
vars.notification[“mail”] = {
groups = [ “icingausers” ]
}
vars.local_tcp_port[“tcp”] ={
tcp_port = “23”
service_name = “Telnet Check”
port_number = “Port 23”

Put the templates into a file called templates.conf, not necessarily filled with Host templates but others as well. If you need that template on a satellite endpoint where the checks are executed, you might put them into a global zone as well.

Depending on your file structure, either put all hosts into a single file (decreases config compiler load with 1 vs many files), or have them separated with FQDN.conf to find them easier.

Typically, I would also advise to put everything into a template which is used multiple times in your hosts. local_tcp_port for instance looks like as if this would be used commonly, hide that in an imported template. Same goes for notification with mail, groups, icingausers thing. Or nscp_boot.

My host objects typically look like:

  • import template tree
  • check_command, if not already imported
  • address
  • special custom variables which override the template imports

Short and readable. The detailed attribute view is what you get with object list and the REST API anyways.

Cheers,
Michael