Icinga Master Upgrade from 2.13.9-1 to 2.14.2-1 fails encountering sync errors

Icinga 2
1

I am currently working on upgrading our distributed monitoring setup from Icinga version 2.13.9-1+ubuntu20.04 to 2.14.2-1+ubuntu20.04. Our setup consists of 2 master nodes, over 25 satellite nodes, and more than 4,000 hosts.

All nodes and satellites have been successfully upgraded to version 2.14.2-1+ubuntu20.04. The only nodes remaining are our master nodes, master01 and master02.

I am encountering an issue when attempting to upgrade master02. While following the upgrade procedure, master02 updates to version 2.14.2-1+ubuntu20.04. However, it is unable to sync and retrieve the same data as master01 in the api folder.

Steps I Followed:

  1. Backup current Icinga directories:
    cp -rp /var/lib/icinga2/ /home/saad/var-lib-icinga
    cp -rp /etc/icinga2/ /home/saad/etc-icinga

  2. Update Icinga version in preferences:
    vim /etc/apt/preferences.d/icinga
    (Changed version to 2.14.2-1+ubuntu20.04)

  3. Update and remove old Icinga packages:
    apt update
    apt remove icinga2 icinga2-bin icinga2-common icinga2-doc icinga2-ido-mysql

  4. Remove old API folder:
    cd /var/lib/icinga2/ && rm -r api/

  5. Install new Icinga packages:
    apt install icinga2-common icinga2-bin icinga2 icinga2-ido-mysql

  6. Validate the configuration:
    icinga2 daemon -C

Current Configuration:

OS Version:
Ubuntu 20.04.6 LTS (Focal Fossa)

Icinga Packages:
ii icinga2 2.13.9-1+ubuntu20.04 amd64
ii icinga2-bin 2.13.9-1+ubuntu20.04 amd64
ii icinga2-common 2.13.9-1+ubuntu20.04 all
ii icinga2-doc 2.14.2-1+ubuntu20.04 all
ii icinga2-ido-mysql 2.13.9-1+ubuntu20.04 amd64
ii vim-icinga2 2.14.2-1+ubuntu20.04 all

Icinga web details:

Icinga Web 2 Version 2.12.1
PHP Version 7.4.3-4ubuntu2.23

Loaded Libraries
|icinga/icinga-php-library. : 0.13.2
|icinga/icinga-php-thirdparty : 0.12.1

Loaded Modules
director : 1.10.2
fileshipper : 1.2.0
idoreports : 0.10.0
incubator : 0.22.0
ipl : v0.5.0
monitoring : 2.12.1
pdfexport. : 0.10.2
reactbundle : 0.9.0
reporting : 0.10.0

Before Upgrade Icinga daemon -C shows:
information/cli: Icinga application loader (version: r2.13.9-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: master02.mon01.hostname.com
information/WorkQueue: #5 (DaemonUtility::LoadConfigFiles) items: 0, rate: 129.2/s (7752/min 7752/5min 7752/15min);
information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 63268 Dependencies.
information/ConfigItem: Instantiated 337 HostGroups.
information/ConfigItem: Instantiated 4193 Hosts.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 341 Downtimes.
information/ConfigItem: Instantiated 920 Comments.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 115617 Notifications.
information/ConfigItem: Instantiated 1746 Zones.
information/ConfigItem: Instantiated 1759 Endpoints.
information/ConfigItem: Instantiated 19 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 374 CheckCommands.
information/ConfigItem: Instantiated 6 ServiceGroups.
information/ConfigItem: Instantiated 5 TimePeriods.
information/ConfigItem: Instantiated 20 Users.
information/ConfigItem: Instantiated 78249 Services.
information/ConfigItem: Instantiated 11 NotificationCommands.
information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
information/cli: Finished validating the configuration file(s).

Current Issue

Despite these steps, master02 is not able to sync the same data as master01 in the api folder, and this prevents proper synchronization between the two masters.

Getting the following Error:
warning/ApiListener: Removing API client for endpoint ‘satellite01’.
information/ApiListener: New client connection for identity ‘satellite01’ from [::ffff:1.1.1.1]:490
warning/JsonRpcConnection: API client disconnected for identity ‘satellite01’
warning/ApiListener: Removing API client for endpoint ‘satellite01’. 0 API clients left.
information/ApiListener: Sending config updates for endpoint ‘satellite01’ in zone ‘nyc11’.
information/ApiListener: Syncing configuration files for global zone ‘director-global’ to endpoint ‘satellite01’.
information/ApiListener: Syncing configuration files for zone ‘lax11’ to endpoint ‘satellite01’.

Service ‘RX10 AS_1!snmp-interface-rene-2’
→ Dependency ‘RX10 AS_1!snmp-interface-rene-2!NETOPS - snmp-agent-dependency’
→ Service ‘RX10 AS_1!SNMP Agent’
→ Dependency ‘RX10 AS_1!SNMP Agent!NETOPS - snmp-agent-dependency’
→ Service ‘RX10 AS_1!SNMP Agent’
critical/cli: Config validation failed. Re-run with ‘icinga2 daemon -C’ after fixing the config.

Need Some Suggestions

I am seeking advice on how to resolve this issue and ensure that master02 can sync properly with master01. Any insights or suggestions on what might be going wrong or additional steps I could take would be greatly appreciated.

Thank you in advance for your assistance!

2

Hi, this is a circular dependency error and starting with Icinga 2 >= 2.14 this is considered to be a fatal error and you need to fix it yourself. Somehow your RX10 AS_1!SNMP Agent service is depending on itself. Please check the assign where ... condition of your NETOPS - snmp-agent-dependency dependency object and make sure that the condition does not apply to your RX10 AS_1!SNMP Agent service. See also How to resolve circular dependency in Icinga2 for more details.

1 Like