New endpoint CSR CA request ends up on satellite instead of a master

Icinga 2
, ,
1

Dear Icinga gurus,

Since few weeks ago, I have noticed an issue when adding new endpoints, they end up with certificates not yet been validated. After a certain time of troubleshooting, I have noticed that even though I am using the same method as before, ca sign requests now end up on a satellite node, not on a master.

I have master-satellite-endpoint setup with four zones - master | {hilversum,sanjose} | endpoint zone.

Master:

  • single node
  • icinga version -
icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.0-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1062.18.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Satellite:

  • single node
icinga2 - The Icinga 2 network monitoring daemon (version: 2.11.4-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1127.8.2.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-ltrjqz9n-project-322-concurrent-0

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Endpoint:
I am just adding one I am currently troubleshooting/testing against:

icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.4-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.25.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Zones (/etc/icinga2/zones.conf):
on master node:

object Endpoint "icinga2-master-1.avinity.tv" {
}

object Zone "master" {
    endpoints = [ "icinga2-master-1.avinity.tv" ]
}

object Endpoint "icinga2-hilversum-satellite-1.avinity.tv" {
        host = "172.16.2.91"
    }
object Endpoint "icinga2-hilversum-satellite-2.avinity.tv" {
        host = "172.17.29.113"
    }

object Zone "hilversum" {
    endpoints = [ "icinga2-hilversum-satellite-1.avinity.tv",  "icinga2-hilversum-satellite-2.avinity.tv" ]
    parent = "master"
}

object Endpoint "icinga2-sanjose-satellite-1.ictv.com" {
        host = "10.200.30.200"
    }

object Zone "sanjose" {
    endpoints = [ "icinga2-sanjose-satellite-1.ictv.com" ]
    parent = "master"
}

object Zone "global-templates" {
    global = true
}

object Zone "director-global" {
    global = true
}

on satellite nodes (same file for both zones):

object Endpoint "icinga2-master-1.avinity.tv" {
}

object Zone "master" {
    endpoints = [ "icinga2-master-1.avinity.tv" ]
}

object Endpoint "icinga2-hilversum-satellite-1.avinity.tv" {
}
object Endpoint "icinga2-hilversum-satellite-2.avinity.tv" {
}

object Zone "hilversum" {
    endpoints = [ "icinga2-hilversum-satellite-1.avinity.tv",  "icinga2-hilversum-satellite-2.avinity.tv" ]
    parent = "master"
}

object Endpoint "icinga2-sanjose-satellite-1.ictv.com" {
}

object Zone "sanjose" {
    endpoints = [ "icinga2-sanjose-satellite-1.ictv.com" ]
    parent = "master"
}

object Zone "global-templates" {
    global = true
}

object Zone "director-global" {
    global = true
}

on edpoint node:

object Endpoint "icinga2-sanjose-satellite-1.ictv.com" {
}

object Endpoint "ovirt-sj-06.ictv.com" {
}

object Zone "sanjose" {
        endpoints = [ "icinga2-sanjose-satellite-1.ictv.com" ]
}

object Zone "ovirt-sj-06.ictv.com" {
        endpoints = [ "ovirt-sj-06.ictv.com" ]
        parent = "sanjose"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true

Config validation:
on master:

[2021-06-03 10:18:44 +0200] information/cli: Icinga application loader (version: 2.12.0-1)
[2021-06-03 10:18:44 +0200] information/cli: Loading configuration file(s).
[2021-06-03 10:18:44 +0200] information/ConfigItem: Committing config item(s).
[2021-06-03 10:18:44 +0200] information/ApiListener: My API identity: icinga2-master-1.avinity.tv
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘ping6’ (in /etc/icinga2/zones.d/global-templates/services.conf: 11:1-11:21) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘’ (in /etc/icinga2/zones.d/global-templates/services.conf: 23:1-23:63) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘Webserver responsiveness check’ (in /etc/icinga2/zones.d/global-templates/services.conf: 34:1-34:46) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘SSL certificate check’ (in /etc/icinga2/zones.d/global-templates/services.conf: 42:1-42:37) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘swap’ (in /etc/icinga2/zones.d/global-templates/services.conf: 147:1-147:20) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘’ (in /etc/icinga2/zones.d/global-templates/services.conf: 172:1-172:61) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘vcenter-volumes’ (in /etc/icinga2/zones.d/global-templates/vmware-services.conf: 2:1-2:31) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘vcenter-listvms’ (in /etc/icinga2/zones.d/global-templates/vmware-services.conf: 10:1-10:31) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘vcenter-listhosts’ (in /etc/icinga2/zones.d/global-templates/vmware-services.conf: 18:1-18:33) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘vmware-listcluster’ (in /etc/icinga2/zones.d/global-templates/vmware-services.conf: 26:1-26:34) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘vcenter-tools’ (in /etc/icinga2/zones.d/global-templates/vmware-services.conf: 34:1-34:29) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] warning/ApplyRule: Apply rule ‘esx-soap-host-check’ (in /etc/icinga2/zones.d/global-templates/vmware-services.conf: 42:1-42:35) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 226 Hosts.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 96 Downtimes.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 4 NotificationCommands.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 13 Comments.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1969 Notifications.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 28 HostGroups.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 108 Zones.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 107 Endpoints.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 3 ApiUsers.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 261 CheckCommands.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 2 Users.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 1438 Services.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 5 ServiceGroups.
[2021-06-03 10:18:44 +0200] information/ConfigItem: Instantiated 96 ScheduledDowntimes.
[2021-06-03 10:18:44 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2021-06-03 10:18:44 +0200] information/cli: Finished validating the configuration file(s).

on satellite:

[2021-06-03 10:18:50 +0200] information/cli: Icinga application loader (version: 2.11.3-1)
[2021-06-03 10:18:50 +0200] information/cli: Loading configuration file(s).
[2021-06-03 10:18:50 +0200] information/ConfigItem: Committing config item(s).
[2021-06-03 10:18:50 +0200] information/ApiListener: My API identity: icinga2-hilversum-satellite-1.avinity.tv
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘netapp_volumes_ovirt_ca’ (in /var/lib/icinga2/api/zones/global-templates/_etc/netapp-services.conf: 91:1-91:39) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘netapp_volumes_netapp2019_svm’ (in /var/lib/icinga2/api/zones/global-templates/_etc/netapp-services.conf: 196:1-196:45) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘netapp_volumes_VMWare’ (in /var/lib/icinga2/api/zones/global-templates/_etc/netapp-services.conf: 211:1-211:37) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘netapp_volumes_final_backup’ (in /var/lib/icinga2/api/zones/global-templates/_etc/netapp-services.conf: 226:1-226:43) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘ping6’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 11:1-11:21) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 23:1-23:63) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘Webserver responsiveness check’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 34:1-34:46) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘SSL certificate check’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 42:1-42:37) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 51:1-51:65) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘swap’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 147:1-147:20) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘users’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 154:1-154:21) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 172:1-172:61) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘load-windows’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 207:1-207:28) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘memory-windows’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 213:1-213:30) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘uptime-windows’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 219:1-219:30) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘disk-windows’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 225:1-225:28) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘firewall-windows’ (in /var/lib/icinga2/api/zones/global-templates/_etc/services.conf: 243:1-243:32) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘vcenter-volumes’ (in /var/lib/icinga2/api/zones/global-templates/_etc/vmware-services.conf: 2:1-2:31) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘vcenter-listvms’ (in /var/lib/icinga2/api/zones/global-templates/_etc/vmware-services.conf: 10:1-10:31) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘vcenter-listhosts’ (in /var/lib/icinga2/api/zones/global-templates/_etc/vmware-services.conf: 18:1-18:33) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘vmware-listcluster’ (in /var/lib/icinga2/api/zones/global-templates/_etc/vmware-services.conf: 26:1-26:34) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘vcenter-tools’ (in /var/lib/icinga2/api/zones/global-templates/_etc/vmware-services.conf: 34:1-34:29) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] warning/ApplyRule: Apply rule ‘esx-soap-host-check’ (in /var/lib/icinga2/api/zones/global-templates/_etc/vmware-services.conf: 42:1-42:35) for type ‘Service’ does not match anywhere!
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 28 HostGroups.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 1 FileLogger.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 4 NotificationCommands.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 1030 Notifications.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 111 Hosts.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 53 Downtimes.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 12 Comments.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 58 Zones.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 57 Endpoints.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 261 CheckCommands.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 2 Users.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 773 Services.
[2021-06-03 10:18:51 +0200] information/ConfigItem: Instantiated 5 ServiceGroups.
[2021-06-03 10:18:51 +0200] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2021-06-03 10:18:51 +0200] information/cli: Finished validating the configuration file(s).

on endpoint:

[2021-06-03 01:17:55 -0700] information/cli: Icinga application loader (version: 2.12.4-1)
[2021-06-03 01:17:55 -0700] information/cli: Loading configuration file(s).
[2021-06-03 01:17:55 -0700] information/ConfigItem: Committing config item(s).
[2021-06-03 01:17:55 -0700] information/ApiListener: My API identity: ovirt-sj-06.ictv.com
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 4 Zones.
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 2 Endpoints.
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 1 FileLogger.
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 235 CheckCommands.
[2021-06-03 01:17:55 -0700] information/ConfigItem: Instantiated 1 ApiListener.
[2021-06-03 01:17:55 -0700] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2021-06-03 01:17:55 -0700] information/cli: Finished validating the configuration file(s).

I have tried both manual and via ansible, but the method is pretty much the same:
via ansible:
in this case:

icinga2-master-1.avinity.tv <== master
icinga2-sanjose-satellite-1.ictv.com <== satellite
ovirt-sj-06.ictv.com <== endpoint

        - name: "Create certificate directory"
          file:
            path: /var/lib/icinga2/certs
            group: "{{ 'icinga' if ansible_distribution == 'CentOS' else 'nagios' }}"
            state: directory
            mode: 0750
            owner: "{{ 'icinga' if ansible_distribution == 'CentOS' else 'nagios' }}"

        - name: "Create new client certificate"
          command: "icinga2 pki new-cert --cn {{ inventory_hostname }} --key /var/lib/icinga2/certs/{{ inventory_hostname }}.key --cert /var/lib/icinga2/certs/{{ inventory_hostname}}.crt"
          args:
              creates: /var/lib/icinga2/certs/{{ inventory_hostname }}.key

        - copy:
            src: "var/lib/icinga2/certs/{{ groups.masters[0] }}.crt"
            dest: "/var/lib/icinga2/certs/{{ groups.masters[0] }}.crt"
          register: trusted_master_crt

        - name: "Generate ticket for the client"
          command: "icinga2 pki ticket --cn {{ inventory_hostname }}"
          delegate_to: "{{ groups.masters[0] }}"
          register: node_ticket
          when: trusted_master_crt.changed

        - name: "Signing and setting up node"
          command: "icinga2 node setup --ticket {{ node_ticket.stdout }} \
                                        --cn {{ inventory_hostname }} \
                                        --endpoint {{ groups.hilversum[0] }} \
                                        --zone {{ inventory_hostname }} \
                                        --parent_host {{ groups.masters[0] }} \
                                        --parent_zone {{ icinga_endpoint_parentzone }}
                                        --trustedcert /var/lib/icinga2/certs/{{ groups.masters[0] }}.crt \
                                        --disable-confd \
                                        --accept-commands --accept-config"
          when: trusted_master_crt.changed and (inventory_hostname in groups['nlendpoints'])
          notify: Restart icinga2

        - name: "Signing and setting up node"
          command: "icinga2 node setup --ticket {{ node_ticket.stdout }} \
                                        --cn {{ inventory_hostname }} \
                                        --endpoint {{ groups.sanjose[0] }} \
                                        --zone {{ inventory_hostname }} \
                                        --parent_host {{ groups.masters[0] }} \
                                        --parent_zone {{ icinga_endpoint_parentzone }}
                                        --trustedcert /var/lib/icinga2/certs/{{ groups.masters[0] }}.crt \
                                        --disable-confd \
                                        --accept-commands --accept-config"
          when: trusted_master_crt.changed and inventory_hostname in groups['caendpoints']
          notify: Restart icinga2
      when: (ansible_distribution == 'CentOS' and ansible_distribution_version is version( '7', '>=')) or ansible_distribution == 'Ubuntu'

via command line:
icinga2 node setup --ticket 81f12b1e045f4707e15ec3c9ac4bccc80e8be451 --cn ovirt-sj-06.ictv.com --endpoint icinga2-sanjose-satellite-1.ictv.com --zone ovirt-sj-06.ictv.com --parent_host icinga2-master-1.avinity.tv --parent_zone sanjose --trustedcert /var/lib/icinga2/certs/icinga2-master-1.avinity.tv.crt --disable-confd --accept-commands --accept-config

In both cases - the sign request ends up on the satellite - why - maybe it’s normal? Should the satellite forward the request to the master? If so, why it does not? Is it due to version missmatch?

  • Enabled features (icinga2 feature list)

Disabled features: compatlog debuglog elasticsearch gelf graphite icingadb livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-pgsql influxdb mainlog notification

  • Icinga Web 2 version and modules (System - About)>

icingaweb2-selinux-2.7.3-1.el7.icinga.noarch
icingaweb2-vendor-JShrink-2.7.3-1.el7.icinga.noarch
icingaweb2-vendor-dompdf-2.7.3-1.el7.icinga.noarch
icingaweb2-vendor-Parsedown-2.7.3-1.el7.icinga.noarch
icingaweb2-common-2.7.3-1.el7.icinga.noarch
icingaweb2-vendor-lessphp-2.7.3-1.el7.icinga.noarch
icingaweb2-vendor-zf1-2.7.3-1.el7.icinga.noarch
icingaweb2-2.7.3-1.el7.icinga.noarch
icingaweb2-vendor-HTMLPurifier-2.7.3-1.el7.icinga.noarch

  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
    I am adding it here only for node I am currently testing/troubleshooting on:
[root@ovirt-sj-06 ~]# icinga2 object list --type Endpoint
Object 'ovirt-sj-06.ictv.com' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 4:1-4:38
  * __name = "ovirt-sj-06.ictv.com"
  * host = ""
  * log_duration = 86400
  * name = "ovirt-sj-06.ictv.com"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 4
    * last_column = 38
    * last_line = 4
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "ovirt-sj-06.ictv.com" ]
    % = modified in '/etc/icinga2/zones.conf', lines 4:1-4:38
  * type = "Endpoint"
  * zone = ""

Object 'icinga2-sanjose-satellite-1.ictv.com' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 1:0-1:53
  * __name = "icinga2-sanjose-satellite-1.ictv.com"
  * host = ""
  * log_duration = 86400
  * name = "icinga2-sanjose-satellite-1.ictv.com"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 0
    * first_line = 1
    * last_column = 53
    * last_line = 1
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "icinga2-sanjose-satellite-1.ictv.com" ]
    % = modified in '/etc/icinga2/zones.conf', lines 1:0-1:53
  * type = "Endpoint"
  * zone = ""

[root@ovirt-sj-06 ~]# icinga2 object list --type zone
Object 'sanjose' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 7:1-7:21
  * __name = "sanjose"
  * endpoints = [ "icinga2-sanjose-satellite-1.ictv.com" ]
    % = modified in '/etc/icinga2/zones.conf', lines 8:2-8:55
  * global = false
  * name = "sanjose"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 7
    * last_column = 21
    * last_line = 7
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "sanjose" ]
    % = modified in '/etc/icinga2/zones.conf', lines 7:1-7:21
  * type = "Zone"
  * zone = ""

Object 'ovirt-sj-06.ictv.com' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 11:1-11:34
  * __name = "ovirt-sj-06.ictv.com"
  * endpoints = [ "ovirt-sj-06.ictv.com" ]
    % = modified in '/etc/icinga2/zones.conf', lines 12:2-12:39
  * global = false
  * name = "ovirt-sj-06.ictv.com"
  * package = "_etc"
  * parent = "sanjose"
    % = modified in '/etc/icinga2/zones.conf', lines 13:2-13:19
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 34
    * last_line = 11
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "ovirt-sj-06.ictv.com" ]
    % = modified in '/etc/icinga2/zones.conf', lines 11:1-11:34
  * type = "Zone"
  * zone = ""

Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 16:1-16:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 17:2-17:14
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 16
    * last_column = 30
    * last_line = 16
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 16:1-16:30
  * type = "Zone"
  * zone = ""

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 20:1-20:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 21:2-21:14
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 20
    * last_column = 29
    * last_line = 20
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 20:1-20:29
  * type = "Zone"
  * zone = ""

Please let me know if more information is required.

2

In the WebUI I can clearly see the following:

Screenshot 2021-06-03 at 10.39.06
Screenshot 2021-06-03 at 10.39.061306×340 27.4 KB

And the sign request is pending on the icinga2-sanjose-satellite-1.ictv.com server.

3

Solved it by upgrading master-satellite(s) version to latest stable and making sure they are all same version OS and icinga2.
Once this was completed - all nonsigned newly added endpoints were instantly finalized and properly checked.