Failed to fetch signed certificate from parent node

Icinga 2
1

I have a primary master and secondary master , for some reason the client setup happens every two hours when the jenkins job run which in turn runs the playbook for client setup.
Recently we are facing issue related to certificate , secondary is not able to fetch the certificate details from the primary.

On checking the debug logs it looks like secondary could not connect to primary , but I was checking the connection manually during the setup process and there was not any network drop.

Kindly assist.

  • Version used (icinga2 --version)
    Primary:
master-icinga ~ # icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.4-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.6.1.el7.x86_64
  Architecture: x86_64

Secondary:

icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.4-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: CentOS Linux
  Platform version: 7 (Core)
  Kernel: Linux
  Kernel version: 3.10.0-1160.6.1.el7.x86_64
  Architecture: x86_64
  • Operating System and version
master2-icinga ~ # cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
  • Enabled features (icinga2 feature list)
master2-icinga ~ # icinga2 feature list
Disabled features: command compatlog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker debuglog ido-mysql mainlog notification

  • Icinga Web 2 version and modules (System - About)

  • Config validation (icinga2 daemon -C)

master2-icinga ~ # icinga2 daemon -C
[2022-08-18 10:34:18 +0000] information/cli: Icinga application loader (version: r2.13.4-1)
[2022-08-18 10:34:18 +0000] information/cli: Loading configuration file(s).
[2022-08-18 10:34:18 +0000] information/ConfigItem: Committing config item(s).
[2022-08-18 10:34:18 +0000] information/ApiListener: My API identity: master2-icinga.int.cloud.ruckuswireless.com
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 5 Users.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 3005 Services.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 9 ServiceGroups.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 4 Zones.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 4 NotificationCommands.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 6922 Notifications.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 456 Hosts.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 24 HostGroups.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 4 Endpoints.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 6 ApiUsers.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 269 CheckCommands.
[2022-08-18 10:34:22 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2022-08-18 10:34:22 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-08-18 10:34:22 +0000] information/cli: Finished validating the configuration file(s).
master2-icinga ~ #
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes
Summary

Zones.conf Primary:

object Endpoint “XXX” {
# Primary master will connect to secondary
# host = “hostvars[host][‘ansible_default_ipv4’][‘address’]}}”
host = “XXX”
}

object Endpoint NodeName {
}

object Zone “master” {
endpoints = [
NodeName,
“XXX”,
]
}

object Zone “global-templates” {

global = true

}

object Zone “global-commands” {
global = true
}
You have new mail in /var/spool/mail/root

secondary

##

object Endpoint "XXX" {
    }
##



object Endpoint NodeName {
}

object Zone "master" {
    endpoints = [
        NodeName,
        "XXX",
                            ]
}

# object Zone "global-templates" {
#     global = true
# }

object Zone "global-commands" {
    global = true
}

and the agent.conf:

object Endpoint "XXX" {
  host = "XXX" //the master actively tries to connect to the client, not viceversa
}

object Zone "XXX" {
  endpoints = [ "XXX" ]
  parent = "master"
}
object Endpoint "XXX" {
  host = XXX" //the master actively tries to connect to the client, not viceversa
}
Summary

object Zone “XXX” {
endpoints = [ “XXX” ]
parent = “master”