Master, Satellite, Client configuration issue

Hello,

A couple months ago I installed icinga2 on a CentOS 7 VM and connectedsome agents/clients. All was working well. The current configuration is:

Master --> Client

Screen Shot 2020-06-19 at 11.40.17 AM

I was recently asked to provide an icinga2 monitoring satellite for a group of systems.

Master —> Satellite —> Client

Screen Shot 2020-06-19 at 11.37.25 AM

I’ve been following the document:

https://icinga.com/docs/icinga2/latest/doc/06-distributed-monitoring/#three-levels-with-masters-satellites-and-agents

This scenario appears to be the one needed to add the Satellite andclients to the existing Master/client configuration. I’ve been workingon this all week and there are just some pieces that aren’t working as expected.

After installing icinga2 on my CentOS VM, I ran the “icinga2 node wizard” command setting the “parent” to the icinga master and changing the “master” zone to be “satellite”.

I then followed the documentation to customize my zones files on the satellite/master to work in a Top Down Sync (see below)

Master: /etc/icinga2/zones.conf

object Endpoint “icinga2.domain.edu” {
// That’s us the master icinga2 server
}

object Zone “master” {
endpoints = [ “icinga2.domain.edu” ]
}

object Endpoint “satellite.domain.edu” {
host = “10.10.194.180” // Actively connect to the satellites.
}

object Zone “satellite-zone” {
endpoints = [ “satellite.domain.edu” ]
parent = “master”
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}


Satellite: /etc/icinga2/zones.conf

object Endpoint “icinga2.domain.edu” {
// This icinga2 master endpoint will connect to us
}

object Zone “master” {
endpoints = [ “icinga2.domain.edu” ]
}

object Endpoint “satellite.domain.edu” {
// That’s us
}

object Zone “satellite-zone” {
endpoints = [ “satellite.domain.edu” ]
parent = “master”
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}


Before adding the client to the satellite, I wanted to make sure I could monitor the satellite within the icingaweb2 software.

Since I am now going to have two zones (master, satellite), I created the following directory structure:

Master: /etc/icinga2/zones.d

          |
          |--->   master (already existed, and is populated with
                          hosts.conf, services.conf, etc)
          |--->   global-templates (which contains some global
                         services I want to monitor on the 
                         satellite and all clients.)
          |--->   satellite

Per the documentation I created the following files within the satellite directory for testing:

Master: /etc/icinga2/zones.conf/satellite

          |
          |--> satellite.conf
          |--> services.conf

These files are configured with the following test contents per the documentation:

Master:/etc/icinga2/zones.conf/satellite/satelite.conf:

object Host “satellite.domain.edu” {
display_name= “TEST Satellite”
check_command = “hostalive”
address = “10.10.194.180”
zone = “master” //optional trick: sync the required host object to the satellite, but enforce the “master” zone to execute the check
}

Master:/etc/icinga2/zones.conf/satellite/services.conf:

object Service “Satellite Server Disk” {
host_name = “satellite.domain.edu”
check_command = “disk”
}

The command “icinga2 daemon -C” on the master/satellite shows no errors so I restarted the services.

I see the satellite system now under hosts and the field for “Satellite Server disk”. I am able to click on the disk and see the correct mounted partitions so I know I’m not looking at the icinga2 master disks.

From my icinga2 master, I can also issue a “Check now” and it appears to contact the satellite and update the information correctly.

Here are my questions:

1 - Why aren’t any of the global services defined under Master:/etc/icinga2/global-templates/services populating?

For example, I have an entry in the above file:

    apply Service "Disk" {
      check_command = "disk"
      vars.disk_wfree = "15%"
      vars.disk_cfree = "10%"
      command_endpoint = host.vars.client_endpoint
      assign where host.vars.client_endpoint
    }

But icingaweb2 doesnt populate those field areas for the satellite server.

Im assuming that because the satellites endpoint and zone are defined in the zones.conf file and not in the satellite.conf file under the zone “satellite”.

Note:

I did try to connect clientA.domain.edu to the icinga2 server (icinga2.domain.com) through the satellite and it looks like it picked up the global definitions.

The zone configuration file on clientA is:

clientA:/etc/icinga2/zone.conf:

object Endpoint “satellite.domain.edu” {
host = “10.10.194.180”
port = “5665”
}

object Zone “satellite-zone” {
endpoints = [ “satellite.domain.edu” ]
}

object Endpoint “clientA.domain.edu” {
// That’s me
}

object Zone “clientA.domain.edu” {
endpoints = [ “clientA.domain.edu” ]
parent = “satellite-zone”
}

object Zone “global-templates” {
global = true
}

object Zone “director-global” {
global = true
}

The clientA.conf (master:/etc/icinga2/zones.d/satellite/clientA.conf) file contains:

// Endpoints & Zones
object Endpoint “clientA.domain.edu” {
}

object Zone “clientA.domain.edu” {
endpoints = [ “clientA.domain.edu” ]
parent = “satellite-zone”
}

// Host Objects
object Host “clientA.edu” {
check_command = “hostalive”
address = “10.10.194.235”
vars.client_endpoint = name //follows the convention that host name == endpoint name


So the clients (clientA, clientB) appear working through the satellite, but the behavior of the satellite itself is not behaving as I’d expect.

2 - When I turn off icinga2 on clientA, all the globally defined services go to “Unknown”/purple. I would expect this since this is the same behavior I see for the clients that hang off my icinga2 master.

However, turning off the icinga2 service on the satellite doesn’t have the same behavior.

A “check now” no longer works, and the service doesn’t change state. The same thing happens for a “check now” on clientA.

So, if I lose a satellite system, I’m getting no indication from the icinga2 master that there’s any issues. Basically, the satellite and it’s clients are giving false positives.

Suggestions appreciated. Sorry for the long post.

Cheers,

Update: While I don’t have director installed, I did find this: https : // community . icinga .com/t/satelite-zone-offline-but-up-in-icinga/1050 which resembles my issue. Investigating now.