Check on satellite appears as PENDING - Baffled NEWBIE Questions

I’ve started this post several times, it’s hard to know where to begin. I’ve been all over the forum but can’t get a clear understand of how the MASTER->SATELLITE configuration works.

I’ll try to keep this basic. I had an existing Icinga2 installation, working fine. I set it up as a master and everything went smoothly except that my current configuration failed because all the services were then missing because it complained that no masterhostname.org zone exists (even though no zone attribute was configured in the service objects). I assumed it recognized there was a master and decided to look for a zone. I renamed the default zone to be my masterhostname.org and that problem went away.

Many iterations later, here is my situation:

  1. My satellite is talking to my master, I get messages like this on the satellite:
    [2020-02-13 13:30:06 -0800] information/ApiListener: Finished syncing endpoint 'masterhostname.org' in zone 'masterhostname.org'. 
    [2020-02-13 13:30:06 -0800] information/ApiListener: Applying config update from endpoint 'masterhostname.org' of zone 'masterhostname.org'.
    [2020-02-13 13:30:06 -0800] warning/ApiListener: Ignoring config update for unknown zone 'satelliteZone'.

You can see my master Zone is the same as my master Host name which is the same as my Endpoint name – does it matter or do the names need to be unique?

  1. My satellite zone is called satelliteZone and I have that in zones.conf. But if I have it there, should I also have it here:
  /etc/icinga2/zones.conf/satelliteZone/hosts.conf:

What’s the difference?? Are they both required, do they work together (either/and) or should I use either zones.conf OR zones.d?

  1. What is “synched” to the satellite? I can see files under /var/lib/icinga2/api/zones but the satellite doesn’t seem to realize it’s in the satelliteZone. Also, there’s an additional zone listed under that lib folder that is no longer configured but the master does not show it in an object list but the satellite DOES show it. This persists even if I restart. What I need to put manually on the satellite vs what is synched vs what is kept only on the master is the #1 most confusing thing.

  2. How can I troubleshoot the fact that my satellite host is PENDING on icingaweb?

Thanks for any help or even just ignoring all this and suggesting what the most basic setup is. I tried to keep it per the documentation but it didn’t work right out of the gate and as I dug around more then I started seeing all the different possible permutations.

Thanks -

Andy

Hi,

can you please share the /etc/icinga2/zones.conf file from both master and satellite node? This helps to spot visible errors.
Also, please add the version from icinga2 --version to get a better picture.

Next to that, the error message from 1) says that the satellite won’t be accepting configuration files sent from the master. This is a security barrier where it will only accept configs for a zone it knows about. This must be defined in the aforementioned zones.conf file, and loaded during startup (before the sync happens). If you add this to endpoint to the zone, it will work automatically.

With 2), whenever we refer to zones.conf, the standardized path is /etc/icinga2/zones.conf. This is because of the official packages use that path, and it also is the default structure you get after a fresh setup. There are more advanced ways to configure that, e.g. with Puppet or the Director, but let’s keep it simple here.

In terms of the cluster config sync, this is fairly simple.

Master

  • Reads config content from /etc/icinga2/zones.d/satellitezonename/* into memory
  • Wraps the config text into JSON to sent it over the cluster as JSON-RPC message

Satellite

  • Receives the JSON-RPC message
  • Evaluates whether the zone target satellitezonename exists, i.e. a Zone object of that name exists, and its parent is the master zone
  • Since 2.11, it puts the config files into a staging directory
  • Runs a separate process to validate the configuration
    • If the configuration is valid, copies that to production and reloads
    • If the configuration fails, i.e. with a missing template or so, it halts and the old config will stay running in production

The exact details are described inside the technical concepts chapter: https://icinga.com/docs/icinga2/latest/doc/19-technical-concepts/#config-sync

To conclude with troubleshooting, you already know that the satellite somehow doesn’t get the configuration synced. With that knowledge, try following the steps explained in the troubleshooting docs: https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#cluster-troubleshooting-config-sync

Cheers,
Michael

2 Likes

Thanks, so much, Michael, this really helps.

masterhostname:/etc/icinga2/zones.conf:

    object Endpoint "masterhostname.org" {
            host = "172.16.242.146"
    }

    object Endpoint "satellitehost.org" {
            host = "172.16.237.252"
    }

    object Zone "masterhostname.org" {
            endpoints = [ "masterhostname.org" ]
    }

    object Zone "satelliteZone" {
            parent = "masterhostname.org"
            endpoints = [ "satellitehost.org" ]
    }

    object Zone "global-templates" {
            global = true
    }

    object Zone "director-global" {
            global = true
    }
satellitehost.org:/etc/icinga2/zones.conf:

object Endpoint "masterhostname.org" {
       host = "172.16.242.146"
       port = "5665"
}

object Zone "masterhostname.org" {
       endpoints = [ "masterhostname.org" ]
}

object Endpoint "satellitehost.org" {
}

object Zone "satellitehost.org" {
       endpoints = [ "satellitehost.org" ]
       parent = "masterhostname.org"
}

object Zone "global-templates" {
       global = true
}

object Zone "director-global" {
       global = true
}

I didn’t realize I should do something with the satellite zones.conf. Please advise if you see any issues. I’ll read through the links you sent. Thank you!

Andy

P.s. Version is r2.10.4-1

1 Like

Making progress! I tweaked the zones.conf file on the satellite host (accordingly). As you probably could see, it reflected the initial setput params I used while running the node wizard, I guess I subsequently changed it during troubleshooting. Now of my two configured satellite services (ping and disk), disk works but ping shows PENDING. I guess I have to add that Service to the zones.d structure?

[root@satellitehost icinga2]# icinga2 object list --type Service |grep name
  * __name = "satellitehost.org!disk"
  * display_name = "disk"
  * host_name = "satellitehost.org"
  * name = "disk"

Trying that now.

1 Like

Thanks, @dnsmichi, I think the only problem was that the zone file wasn’t right on the satellite. I forgot that is setup during the initial satellite wizard so my subsequent fiddlings got it out of whack. I have one more question now that things are working, will start a new thread. Thanks, again!

1 Like

For later readers, I think this was a key point in the documentation:

The satellite zone will not only accept zone configuration for its own zone, but also all configured child zones. That is why it is important to configure the zone hierarchy on the satellite as well.

The zone configuration file on the satellite host did not reflect the zones.d structure on the master.

2 Likes

Hi,

I guessed that before, though I wanted to provide a gentle guidance on all your questions being asked. That’s why I didn’t throw in a short sentence but encouraged you to learn it by yourself :slight_smile:

See, now I am happy that you learned something new, and you are happy with solving the problem. If you like that, stay a little longer and help others in this community in the same way :kissing_heart:

Cheers,
Michael

1 Like