Icinga2 Agent Communication Issues After Debian 10→12 Upgrade

Icinga 2
,
1

Hello everyone!

After upgrading the server’s OS from Debian 10 to Debian 12, I’ve been facing an issue with two Nagios plugins: check_apt and a custom plugin that queries a database to check active users. The other plugins appear to be working fine.

check_apt

image
image669×245 10.6 KB

But then:

image

custom plugin

It shows 12 users

image

But then:

Is not sending anything to the database where the plugin output is collected.

image
image891×228 12.2 KB

What we have tested so far:
1- The plugin works correctly when executed locally as nagios user

sudo -u nagios /usr/lib/nagios/plugins/check_apt

2- Certificates are properly signed by the CA:
icinga2 pki verify --cert / --cacert

3- Firewall allows port 5665 (also there are plugins that work)

4- Reinstalled the node using the wizard.

5- Checked the permissions for the user that queries the database, and they are correct.

Thank you in advance!

Version used :

  • In node:
    version: r2.13.6-1

  • In master:
    version: r2.13.6-1

Operating System and version:

  • In node:
    Debian GNU/Linux 12 (bookworm)

  • In master:
    Debian GNU/Linux 12 (bookworm)

Enabled features :

  • In node:
Disabled features: command compatlog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker debuglog mainlog
  • In master:
Disabled features: compatlog elasticsearch gelf graphite icingadb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command debuglog ido-mysql influxdb mainlog notification

Icinga Web 2 version and modules:

image
image602×525 15.4 KB

Config validation:

  • In node:
[2025-03-25 15:39:43 +0100] information/cli: Icinga application loader (version: r2.13.6-1)
[2025-03-25 15:39:43 +0100] information/cli: Loading configuration file(s).
[2025-03-25 15:39:43 +0100] information/ConfigItem: Committing config item(s).
[2025-03-25 15:39:43 +0100] information/ApiListener: My API identity: xx.xx.net
[2025-03-25 15:39:43 +0100] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /var/lib/icinga2/api/zones/global-templates/_etc/notifications.conf: 21:1-21:48) for type 'Notification' does not match anywhere!
[2025-03-25 15:39:43 +0100] warning/ApplyRule: Apply rule 'telegram-notification' (in /var/lib/icinga2/api/zones/global-templates/_etc/notifications.conf: 41:1-41:53) for type 'Notification' does not match anywhere!
[2025-03-25 15:39:43 +0100] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /var/lib/icinga2/api/zones/global-templates/_etc/notifications.conf: 11:1-11:45) for type 'Notification' does not match anywhere!
[2025-03-25 15:39:43 +0100] warning/ApplyRule: Apply rule 'telegram-notification' (in /var/lib/icinga2/api/zones/global-templates/_etc/notifications.conf: 32:1-32:50) for type 'Notification' does not match anywhere!
[2025-03-25 15:39:43 +0100] warning/ApplyRule: Apply rule 'backup-downtime' (in /var/lib/icinga2/api/zones/global-templates/_etc/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere!
[2025-03-25 15:39:43 +0100] warning/ApplyRule: Apply rule 'load-downtime' (in /var/lib/icinga2/api/zones/global-templates/_etc/downtimes.conf: 26:1-26:50) for type 'ScheduledDowntime' does not match anywhere!
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 2 Users.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 6 ServiceGroups.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 4 Zones.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 4 NotificationCommands.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 4 HostGroups.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 2 FileLoggers.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 3 ApiUsers.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 248 CheckCommands.
[2025-03-25 15:39:43 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2025-03-25 15:39:43 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-03-25 15:39:43 +0100] information/cli: Finished validating the configuration file(s).
  • In master:
[2025-03-25 15:41:04 +0100] information/cli: Icinga application loader (version: r2.13.6-1)
[2025-03-25 15:41:04 +0100] information/cli: Loading configuration file(s).
[2025-03-25 15:41:04 +0100] information/ConfigItem: Committing config item(s).
[2025-03-25 15:41:04 +0100] information/ApiListener: My API identity: monitor.com
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 InfluxdbWriter.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 3 TimePeriods.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 2 Users.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 97 Services.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 6 ServiceGroups.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 44 ScheduledDowntimes.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 67 Zones.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 4 NotificationCommands.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 194 Notifications.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 15 Hosts.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 4 HostGroups.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 Comment.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 44 Downtimes.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 66 Endpoints.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 2 FileLoggers.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 3 ApiUsers.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 248 CheckCommands.
[2025-03-25 15:41:04 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2025-03-25 15:41:04 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-03-25 15:41:04 +0100] information/cli: Finished validating the configuration file(s).

If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes:

  • In node:

/*
 * Generated by Icinga 2 node setup commands
 * on 2025-03-25 13:39:18 +0100
 */

object Endpoint "monitor.com" {
        host = "xx.xx.xx.xx"
        port = "5665"
}

object Zone "master" {
        endpoints = [ "monitor.com" ]
}

object Endpoint "xx.xx.net" {
}

object Zone "xx.xx.net" {
        endpoints = [ "xx.xx.net" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}
  • In master:
Object 'xx.xx.net' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 523:1-523:39
  * __name = "xx.xx.net"
  * host = ""
  * log_duration = 86400
  * name = "xx.xx.net"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 523
    * last_column = 39
    * last_line = 523
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "xx.xx.net" ]
    % = modified in '/etc/icinga2/zones.conf', lines 523:1-523:39
  * type = "Endpoint"
  * zone = ""
2

Is check_apt really running on the same machine?

The performance data may not be properly formatted as described here as you have multiple pipe symbols within one line.

I’d recommend to enable debuglog and search for perfdata in debug/IdoMysqlConnection e.g.:

perfdata = ‘rta=0.163000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0’

With that you could identify how icinga is evaluating the perfdata from your custom plugin.

3

Hello,

Thanks for your reply. The thing is, I have multiple nodes with these same plugins, and they all work fine. However, on this specific node, these two plugins started failing after the upgrade, so I think it’s not a formatting issue

Also, regarding the “check_apt” output, I feel like it might be displaying sth cached bc, on the problematic node, I have 0 packages to update. And on the master, I have 58, so I don’t know where it’s getting the information about 56 packages. It also can’t be that the plugin is pointing to another node since no node has anything to update, so I have no idea where that number 56 is coming from :thinking:

4

I use inspect to get the full command and then try to run them manually on the command line to figure out why checks misbehave.

If I get different behavior, it’s probably the environment.

You should also be able to find the command_endpoint while inspecting.

5

I realized that the directory structure /var/lib/icinga2/api/zones/global-templates/_etc/ doesn’t exist on my faulty node. Instead, it only goes up to* /var/lib/icinga2/api/zones/ .

However, the zones.conf files on both the master and the node appear to be correctly configured. I’ve tried reinstalling the node multiple times, but it still doesn’t receive the templates. Additionally, the web interface keeps displaying data that I can’t trace back to its source

6

Did you miss the global-templates zone in the faulty nodes zone.conf?

7

Try to clean up on your faulty node:

systemctl stop icinga2
rm /var/lib/icinga2/icinga2.state
rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*
systemctl start icinga2

and check icinga2.log for sync errors/ignores.

8

Hello, a small update:

On one hand, I’d say the zones.conf of the faulty node looks correct—it’s this one:

/*
 * Generated by Icinga 2 node setup commands
 * on 2025-03-26 13:41:39 +0100
 */

object Endpoint "monitor.com" {
}

object Zone "master" {
        endpoints = [ "monitor.com" ]
}

object Endpoint "xx.xx.net" {
}

object Zone "xx.xx.net" {
        endpoints = [ "xx.xx.net" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

On the other hand, I tried manually sending the templates from the master to the node using scp and setting the permissions to see if it would work as a temporary fix or provide clues about where the error might be. This recreation didn’t work either.

I’ll now try your advice.

9

Well, sorry for the trouble

We finally figured out the issue. Turns out there was a clone of the server still active (we had duplicated it for the OS upgrade)

Once we powered it off and reinstalled the node, everything worked instantly :sweat_smile:

1 Like