Director not pushing correct configuration to satellites

Good Morning All,

Topology

I have been working on setting up a Top Down Command Endpoint style of monitoring and am running into a unique issue. I am performing simple HTTP/SSH/Ping remote checks (no agent installed on the client endpoints only on the satellites and master) and it is working on some but not all. Take a look here

These two hosts have the exact same configuration (minus their IP and name obviously) and yet one of them is working as intended (Site-A-Host-01). For some reason Site-A-Host-02 is being checked by IcingaMaster and not its satellite. If I understand the documentation correctly then all the files I should be looking at are in /var/lib/icinga2/api on Site-A-Satellite.

root@Site-A-Satellite:/var/lib/icinga2/api# tree
.
β”œβ”€β”€ log
β”‚   └── current
β”œβ”€β”€ packages
β”‚   └── _api
β”‚       β”œβ”€β”€ 9935623f-701a-4a38-a515-1f2044b95778
β”‚       β”‚   β”œβ”€β”€ conf.d
β”‚       β”‚   β”‚   └── downtimes
β”‚       β”‚   β”œβ”€β”€ include.conf
β”‚       β”‚   └── zones.d
β”‚       β”œβ”€β”€ active.conf
β”‚       β”œβ”€β”€ active-stage
β”‚       └── include.conf
β”œβ”€β”€ repository
β”œβ”€β”€ zones
β”‚   β”œβ”€β”€ director-global
β”‚   β”‚   └── director
β”‚   β”‚       β”œβ”€β”€ 001-director-basics.conf
β”‚   β”‚       β”œβ”€β”€ host_templates.conf
β”‚   β”‚       β”œβ”€β”€ servicesets.conf
β”‚   β”‚       └── service_templates.conf
β”‚   └── Site-A
β”‚       └── director
β”‚           └── hosts.conf
└── zones-stage
    β”œβ”€β”€ director-global
    β”‚   └── director
    β”‚       β”œβ”€β”€ 001-director-basics.conf
    β”‚       β”œβ”€β”€ host_templates.conf
    β”‚       β”œβ”€β”€ servicesets.conf
    β”‚       └── service_templates.conf
    └── Site-A
        └── director
            └── hosts.conf

18 directories, 15 files

When I look in there I notice that Site-A-Host-01 is the only host in the hosts.conf file

root@Site-A-Satellite:/var/lib/icinga2/api# cat zones/Site-A/director/hosts.conf
object Host "Site-A-Host-01" {
    import "SiteA-Linux-Basic-Host"

    display_name = "Site-A-Host-01"
    address = "10.40.0.20"
}
root@Site-A-Satellite:/var/lib/icinga2/api# grep -rin "Site-A-Host-02" .
root@Site-A-Satellite:/var/lib/icinga2/api# cd /etc/icinga2/
root@Site-A-Satellite:/etc/icinga2# grep -rin "Site-A-Host-02" .
root@IcingaMaster:/etc/icinga2# grep -rin "Site-A-Host-02" .
root@IcingaMaster:/etc/icinga2# cd /var/lib/icinga2/api
root@IcingaMaster:/var/lib/icinga2/api# grep -rin "Site-A-Host-02" .
root@IcingaMaster:/var/lib/icinga2/api# grep -rin "Site-A-Host-02" .
./zones/master/director/hosts.conf:1:object Host "Site-A-Host-02" {
./zones/master/director/hosts.conf:4:    display_name = "Site-A-Host-02"
./packages/director/c4bd36f5-7e0c-41d2-8e65-669966cde477/zones.d/master/hosts.conf:1:object Host "Site-A-Host-02" {
./packages/director/c4bd36f5-7e0c-41d2-8e65-669966cde477/zones.d/master/hosts.conf:4:    display_name = "Site-A-Host-02"
./packages/director/0101059d-0be6-4d2e-9f80-21da817070f1/zones.d/master/hosts.conf:1:object Host "Site-A-Host-02" {
./packages/director/0101059d-0be6-4d2e-9f80-21da817070f1/zones.d/master/hosts.conf:4:    display_name = "Site-A-Host-02"

Alright that’s weird.

root@IcingaMaster:/var/lib/icinga2/api# cat zones/master/director/hosts.conf
object Host "Site-A-Host-02" {
    import "SiteA-Linux-Basic-Host"

    display_name = "Site-A-Host-02"
    address = "10.40.0.30"
}

object Host "Site-B-Host-01" {
    import "SiteB-Linux-Basic-Host"

    display_name = "Site-B-Host-01"
    address = "10.50.0.30"
}

object Host "Site-C-Host-01" {
    import "SiteC-Linux-Basic-Host"

    display_name = "Site-C-Host-01"
    address = "10.60.0.20"
}

object Host "Site-C-Host-02" {
    import "SiteC-Linux-Basic-Host"

    display_name = "Site-C-Host-02"
    address = "10.60.0.30"
}

So this confirms what the Web UI is showing. The host configuration for these hosts are not being placed on their satellites but rather on IcingaMaster.

Their host template appears to be correct I think.

Anyone have any clues or ideas of where I can look and or what I can change?

Thanks!

Hi,

did you check the icinga logs as well? Normally you should find a hint what happens during deploying a new config to a satellite. Like wrong certificate, any connection issue.
Also what happens during receving a new config from the master on your satellite.

If we had such errors, it was often a problem with the (local) firewall or incorrect certificates

Based on the documentation I should be looking in /var/log/icinga2/ for the log entries. I made a fake host for Site A and then deployed the configuration. Here are the 150 or so log entries created on IcingaMaster the were generated during that.

[2020-12-09 13:18:51 +0000] information/ApiListener: Copying 1 zone configuration files for zone 'Site-A' to '/var/lib/icinga2/api/zones/Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/Site-A//director/hosts.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Copying 3 zone configuration files for zone 'master' to '/var/lib/icinga2/api/zones/master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/hosts.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//director/services.conf
[2020-12-09 13:18:51 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-12-09 13:18:51 +0000] information/ExternalCommandListener: 'command' started.
[2020-12-09 13:18:51 +0000] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'

[2020-12-09 13:18:51 +0000] information/DbConnection: 'ido-mysql' started.
[2020-12-09 13:18:51 +0000] information/NotificationComponent: 'notification' started.
[2020-12-09 13:18:51 +0000] information/CheckerComponent: 'checker' started.
[2020-12-09 13:18:51 +0000] information/ApiListener: Reconnecting to endpoint 'Site-B-Satellite.picnicsecurity.com' via host '10.50.0.20' and port '5665'
[2020-12-09 13:18:51 +0000] information/ConfigItem: Activated all objects.
[2020-12-09 13:18:51 +0000] information/ApiListener: Reconnecting to endpoint 'Site-A-Satellite.picnicsecurity.com' via host '10.40.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/ApiListener: Reconnecting to endpoint 'Site-C-Satellite.picnicsecurity.com' via host '10.60.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2020-12-09 13:18:51 +0000] information/DbConnection: Resuming IDO connection: ido-mysql
[2020-12-09 13:18:51 +0000] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'Site-C-Satellite.picnicsecurity.com' to [10.60.0.10]:5665
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for zone 'Site-C' to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'Site-C-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'Site-C-Satellite.picnicsecurity.com' in zone 'Site-C'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished reconnecting to endpoint 'Site-C-Satellite.picnicsecurity.com' via host '10.60.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-C-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-C-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'Site-B-Satellite.picnicsecurity.com' to [10.50.0.20]:5665
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished reconnecting to endpoint 'Site-B-Satellite.picnicsecurity.com' via host '10.50.0.20' and port '5665'
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for zone 'Site-B' to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'Site-B-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'Site-B-Satellite.picnicsecurity.com' in zone 'Site-B'.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-B-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-B-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'Site-A-Satellite.picnicsecurity.com' to [10.40.0.10]:5665
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished reconnecting to endpoint 'Site-A-Satellite.picnicsecurity.com' via host '10.40.0.10' and port '5665'
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for global zone 'director-global' to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing configuration files for zone 'Site-A' to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'Site-A-Satellite.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-A-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-A-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'Site-A-Satellite.picnicsecurity.com' in zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga2' in 0.154311 second(s).
[2020-12-09 13:18:53 +0000] information/ApiListener: New client connection from [10.50.0.10]:41670 (no client certificate)
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/stages/director/bcd05b3c-8f05-4076-9617-bd8a6f1b5b4c (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/files/director/bcd05b3c-8f05-4076-9617-bd8a6f1b5b4c/status (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/files/director/bcd05b3c-8f05-4076-9617-bd8a6f1b5b4c/startup.log (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41670), user: director, agent: , status: OK).
[2020-12-09 13:18:53 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41670)
[2020-12-09 13:18:57 +0000] information/ApiListener: New client connection from [10.50.0.10]:41672 (no client certificate)
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41672), user: director, agent: , status: OK).
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41672), user: director, agent: , status: OK).
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41672)
[2020-12-09 13:19:00 +0000] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.116667/s (7/min 7/5min 7/15min);
[2020-12-09 13:19:00 +0000] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-12-09 13:19:01 +0000] information/IdoMysqlConnection: Pending queries: 9 (Input: 3/s; Output: 4/s)
[2020-12-09 13:19:11 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:19:17 +0000] information/ApiListener: New client connection from [10.50.0.10]:41674 (no client certificate)
[2020-12-09 13:19:17 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41674), user: director, agent: , status: OK).
[2020-12-09 13:19:17 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41674), user: director, agent: , status: OK).
[2020-12-09 13:19:17 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41674)
[2020-12-09 13:19:21 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:19:31 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:19:38 +0000] information/ApiListener: New client connection from [10.50.0.10]:41676 (no client certificate)
[2020-12-09 13:19:38 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41676), user: director, agent: , status: OK).
[2020-12-09 13:19:38 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41676), user: director, agent: , status: OK).
[2020-12-09 13:19:38 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41676)
[2020-12-09 13:19:58 +0000] information/ApiListener: New client connection from [10.50.0.10]:41680 (no client certificate)
[2020-12-09 13:19:58 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41680), user: director, agent: , status: OK).
[2020-12-09 13:19:58 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41680), user: director, agent: , status: OK).
[2020-12-09 13:19:58 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41680)
[2020-12-09 13:20:01 +0000] information/IdoMysqlConnection: Pending queries: 10 (Input: 3/s; Output: 2/s)
[2020-12-09 13:20:11 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2020-12-09 13:20:19 +0000] information/ApiListener: New client connection from [10.50.0.10]:41682 (no client certificate)
[2020-12-09 13:20:19 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41682), user: director, agent: , status: OK).
[2020-12-09 13:20:19 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41682), user: director, agent: , status: OK).
[2020-12-09 13:20:19 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41682)
[2020-12-09 13:20:39 +0000] information/ApiListener: New client connection from [10.50.0.10]:41684 (no client certificate)
[2020-12-09 13:20:39 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41684), user: director, agent: , status: OK).
[2020-12-09 13:20:39 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41684), user: director, agent: , status: OK).
[2020-12-09 13:20:39 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41684)
[2020-12-09 13:20:41 +0000] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)

The things that are standing out to me are

[2020-12-09 13:18:51 +0000] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-C-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-C-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-B-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-B-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: Received certificate request for CN 'Site-A-Satellite.picnicsecurity.com' signed by our CA.
[2020-12-09 13:18:51 +0000] information/JsonRpcConnection: The certificate for CN 'Site-A-Satellite.picnicsecurity.com' is valid and uptodate. Skipping automated renewal.
2020-12-09 13:18:53 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:41670)
[2020-12-09 13:18:57 +0000] information/ApiListener: New client connection from [10.50.0.10]:41672 (no client certificate)
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:41672), user: director, agent: , status: OK).
[2020-12-09 13:18:57 +0000] information/HttpServerConnection: Request: GET /v1/config/packages (from [10.50.0.10]:41672), user: director, agent: , status: OK).

This log entry seems to repeat a lot.

So as far as the master is concerned it is able to connect to the satellites and push configuration to them.

Looking at Site-A-Satellite most recent log entries I see this

[2020-12-09 13:09:55 +0000] information/ApiListener: New client connection for identity 'IcingaMaster.picnicsecurity.com' from [10.50.0.10]:34390
[2020-12-09 13:09:55 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Sending config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished sending config file updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Finished syncing endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Applying config update from endpoint 'IcingaMaster.picnicsecurity.com' of zone 'master'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Received configuration for zone 'Site-A' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Site-A//director/hosts.conf' for zone 'Site-A'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A' (242 Bytes).
[2020-12-09 13:09:55 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2020-12-09 13:09:55 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (4632 Bytes).
[2020-12-09 13:09:55 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'IcingaMaster.picnicsecurity.com' are equal to production, skipping validation and reload.
[2020-12-09 13:10:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:11:16 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 12/5min 13/15min);
[2020-12-09 13:11:16 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-12-09 13:12:14 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:13:12 +0000] information/ApiListener: New client connection from [10.50.0.10]:34438 (no client certificate)
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/types (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:34438)
[2020-12-09 13:15:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:16:26 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-12-09 13:16:26 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 16/5min 29/15min);
[2020-12-09 13:17:04 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:18:51 +0000] warning/JsonRpcConnection: API client disconnected for identity 'IcingaMaster.picnicsecurity.com'
[2020-12-09 13:18:51 +0000] warning/ApiListener: Removing API client for endpoint 'IcingaMaster.picnicsecurity.com'. 0 API clients left.
[2020-12-09 13:18:51 +0000] information/ApiListener: New client connection for identity 'IcingaMaster.picnicsecurity.com' from [10.50.0.10]:34490
[2020-12-09 13:18:51 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending config file updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'IcingaMaster.picnicsecurity.com'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished sending replay log for endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Finished syncing endpoint 'IcingaMaster.picnicsecurity.com' in zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Applying config update from endpoint 'IcingaMaster.picnicsecurity.com' of zone 'master'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Received configuration for zone 'Site-A' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Site-A//director/hosts.conf' for zone 'Site-A'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A' (242 Bytes).
[2020-12-09 13:18:51 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2020-12-09 13:18:51 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (4632 Bytes).
[2020-12-09 13:18:51 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'IcingaMaster.picnicsecurity.com' are equal to production, skipping validation and reload.
[2020-12-09 13:19:44 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:20:44 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:20:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:21:36 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-12-09 13:21:36 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 12/5min 40/15min);
[2020-12-09 13:22:04 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:25:34 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:25:57 +0000] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2020-12-09 13:26:46 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0666667/s (4/min 12/5min 36/15min);
[2020-12-09 13:26:46 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-12-09 13:27:04 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);

The things that stand out to me here are

[2020-12-09 13:05:57 +0000] information/ApiListener: Applying config update from endpoint 'IcingaMaster.picnicsecurity.com' of zone 'master'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A'. Current timestamp '2020-12-09 13:05:53 +0000' (1607519153.953211) >= received timestamp '2020-12-09 13:05:53 +0000' (1607519153.953211).
[2020-12-09 13:05:57 +0000] information/ApiListener: Received configuration for zone 'Site-A' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/Site-A//director/hosts.conf' for zone 'Site-A'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/Site-A' (242 Bytes).
[2020-12-09 13:05:57 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global'. Current timestamp '2020-12-09 13:05:53 +0000' (1607519153.955098) >= received timestamp '2020-12-09 13:05:53 +0000' (1607519153.955098).
[2020-12-09 13:05:57 +0000] information/ApiListener: Received configuration for zone 'director-global' from endpoint 'IcingaMaster.picnicsecurity.com'. Comparing the timestamp and checksums.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/001-director-basics.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/host_templates.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/service_templates.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/director-global//director/servicesets.conf' for zone 'director-global'.
[2020-12-09 13:05:57 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/director-global' (4632 Bytes).
[2020-12-09 13:05:57 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'IcingaMaster.picnicsecurity.com' are equal to production, skipping validation and reload.
[2020-12-09 13:06:06 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.0166667/s (1/min 1/5min 1/15min);
[2020-12-09 13:06:06 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2020-12-09 13:07:24 +0000] information/RemoteCheckQueue: items: 0, rate: 0/s (6/min 30/5min 90/15min);
[2020-12-09 13:09:22 +0000] warning/JsonRpcConnection: API client disconnected for identity 'IcingaMaster.picnicsecurity.com'
[2020-12-09 13:09:22 +0000] warning/ApiListener: Removing API client for endpoint 'IcingaMaster.picnicsecurity.com'. 0 API clients left.

These logs seem to indicate that Site-A is not applying configuration updates because it thinks it is ahead? But then it seems to apply it anyways so I am not sure.

[2020-12-09 13:13:12 +0000] information/ApiListener: New client connection from [10.50.0.10]:34438 (no client certificate)
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/ (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: Request: GET /v1/types (from [10.50.0.10]:34438), user: director, agent: , status: OK).
[2020-12-09 13:13:12 +0000] information/HttpServerConnection: HTTP client disconnected (from [10.50.0.10]:34438)

These are the same entries as ones found on IcingaMaster. Not sure what they mean though

You need to setup Cluster Zone for Host-02 to Site-A-Satellite.

Assuming that you mean setting the cluster zone in the service template

I have fiddled around with that in my troubleshooting but not with confidence. Am I supposed to create a service check for each Zone that I have? How does that scale cleanly? Regardless I am still seeing the same results

Its weird that Service Templates dont have a Command endpoint like Host Templates

I am currently reading through https://icinga.com/docs/icinga-2/latest/doc/15-troubleshooting/ to see if I can find anything that could help figure this out

Wait a second. Do I just have the completely wrong setup?

I have been using the Top Down Command Endpoint style and I just stumbled on a post from 2017 that seems to indicate that I actually need to do To Down Config Sync?

I am not sure how Director would be able to tell the difference between these two but I am looking at my zones.conf file for satellite and master and they both have the director-global zones

On my IcingaMaster endpoint I have zones defined in zones.d

root@IcingaMaster:/etc/icinga2/zones.d# ls
api-users.conf  master  README  Site-A  Site-B  Site-C

I am assuming I do not need this setup on my Satellites as director will make the directories and configuration files needed in /var/lib/icinga2/api correct?

root@Site-A-Satellite:/etc/icinga2/zones.d# ls
api-users.conf  README

I am trying to determine if there is something about my file structure that is not setup correctly. Trying to determine if I needed to setup some sort of file directory structure manually prior to installing Director?

No, your host object needs to be in the correct zone.

Yes, that’s a recommendation. And it could be easily archieved with e.g.:

apply Service "icinga_zone" {
   display_name = "Icinga Zone"
   check_command = "cluster-zone"

   assign where get_object("Endpoint", host.name)
}

This belongs the host endpoint and host zone objects only. Satellites still needs to be configured in zones.conf only at the master and the satellites…

I did not know about this. What exactly is this doing? In Director I was able to get close to this configuration but I can not get the assign where get_object("Endpoint", host.name) correct.



Alright this is good news then because I have zones.conf setup on each satellite.

root@IcingaMaster:/etc/icinga2# cat zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-12-04 18:20:20 +0000
 */

object Endpoint "IcingaMaster.picnicsecurity.com" {
        host = "10.50.0.10"
        log_duration = 604800
}

object Zone "master" {
        endpoints = [ "IcingaMaster.picnicsecurity.com" ]
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

object Endpoint "Site-A-Satellite.picnicsecurity.com" {
        host = "10.40.0.10"
}

object Zone "Site-A" {
        endpoints = [ "Site-A-Satellite.picnicsecurity.com" ]
        parent = "master"
}

object Endpoint "Site-B-Satellite.picnicsecurity.com" {
        host = "10.50.0.20"
}

object Zone "Site-B" {
        endpoints = [ "Site-B-Satellite.picnicsecurity.com" ]
        parent = "master"
}

object Endpoint "Site-C-Satellite.picnicsecurity.com" {
        host = "10.60.0.10"
}

object Zone "Site-C" {
        endpoints = [ "Site-C-Satellite.picnicsecurity.com" ]
        parent = "master"
}
root@Site-A-Satellite:/etc/icinga2#  cat zones.conf
/*
 * Generated by Icinga 2 node setup commands
 * on 2020-12-04 18:52:21 +0000
 */

object Endpoint "IcingaMaster.picnicsecurity.com" {
        host = "10.50.0.10"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "IcingaMaster.picnicsecurity.com" ]
}

object Endpoint "Site-A-Satellite.picnicsecurity.com" {
        host = "10.40.0.10"
        log_duration = 0
}

object Zone "Site-A" {
        endpoints = [ "Site-A-Satellite.picnicsecurity.com" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

Alright I got this to work but not in a way I am necessarily happy with.

I am not sure if there is a supported way to achieve what I am looking to do here or not but I found this old pull request and it solved my issue (so far). I have not run into any issues with it yet but if I do I will let you all know.

I am leaving this unsolved for now because I still want to know if what I am doing is actually supported by Director or not. The modification I am using is obviously not supported and thus if things break I am on my own and this is not ideal in any way. I would like to know if Director supports these distributed setups like the one that I have or not.

Icinga’s documentation heavily suggests the usage of Director but never actually says which setups it supports. If I can figure out which type of setups Director supports then I can help update the documentation for those that come after me

I will mark this as the β€œsolution” in 2 weeks if no other progress is made. Note that this is not a solution, but a work around

The director only supports Top Down Command Endpoint. That means your Site-A-Hosts belong to your zone Site-A. The endpoint for Site-A is your Site-A-Satellite, hence, all β€œremote” Checks e.g. ping, ssh, http etc. scheduled by and executed at your Site-A-Satellite.

If you need checks to be executed on a host locally e.g. load or disk, you need to define command_endpoint for those services. Hence, the check is scheduled by your Site-A-Satellite but executed at the host locally. Using the director you simply need to enable Run on agent. Details are described here.

1 Like

Sorry for the delayed response been a busy week.

This is indeed true!

As it turns out all I need to do is the following.

Make a service template with the service I wish to check and not set the run on agent or cluster zone fields

Then make a host templates with the hostalive check command and the service template from above added to it. I am again not setting any of the cluster zone, icinga2 agent, or command endpoint values here either.

Lastly I make my host and here is where I set the cluster zone and icinga2 agent values. Since I am doing remote checks on site-a I set the cluster zone to site-a and icinga2 agent to no.

Repeat this for the other hosts and…

Voila!

It all works!!!

Thank you @rsx !