I have a (mostly) fresh install of icinga2 installed on Ubuntu 20.04, and am trying to get my first agent check working properly. I’m currently able to run remote checks (like ssh and ping), but when I try to use the icinga2 agent check, the check remains in a pending status.
In all of the below output, “icinga.example.com” has been changed from the name of my main / master icinga server.
oot@icinga:/etc/icinga2/zones.d/master# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.2-1)
Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Ubuntu
Platform version: 20.04.3 LTS (Focal Fossa)
Kernel: Linux
Kernel version: 5.4.0-90-generic
Architecture: x86_64
Build information:
Compiler: GNU 9.3.0
Build host: runner-hh8q3bz2-project-298-concurrent-0
OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
with the following features:
root@icinga:/etc/icinga2/zones.d/master# icinga2 feature list
Disabled features: command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker ido-pgsql mainlog notification
Here is my config validation:
root@icinga:/etc/icinga2/zones.d/master# icinga2 daemon -C
[2021-11-30 18:30:13 +0000] information/cli: Icinga application loader (version: r2.13.2-1)
[2021-11-30 18:30:13 +0000] information/cli: Loading configuration file(s).
[2021-11-30 18:30:14 +0000] information/ConfigItem: Committing config item(s).
[2021-11-30 18:30:14 +0000] information/ApiListener: My API identity: icinga-new
[2021-11-30 18:30:14 +0000] warning/ApplyRule: Apply rule 'vnc' (in /etc/icinga2/conf.d/services.conf: 112:1-112:19) for type 'Service' does not match anywhere!
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 13 Notifications.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 157 Hosts.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 5 Zones.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 3 Endpoints.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 IdoPgsqlConnection.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 2 ApiUsers.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 244 CheckCommands.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 User.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2021-11-30 18:30:14 +0000] information/ConfigItem: Instantiated 285 Services.
[2021-11-30 18:30:14 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-11-30 18:30:14 +0000] information/cli: Finished validating the configuration file(s).
Here’s my /etc/icinga/zones.conf file (commented sections have been redacted):
object Endpoint NodeName {
host = NodeName
}
object Zone ZoneName {
endpoints = [ NodeName ]
}
object Zone "global-templates" {
global = true
}
object Zone "director-global" {
global = true
}
object Zone "master" {
endpoints = [ "icinga.example.com" ]
}
object Zone "davidw-test-icinga-agent" {
endpoints = [ "davidw-test-icinga-agent" ]
parent = "master"
}
I have 2 files in /etc/icinga2/zones.d/master/ :
root@icinga:/etc/icinga2# ls /etc/icinga2/zones.d/master/
davidw-test-icinga-agent.conf
icinga.example.com.conf
Here are their contents:
root@icinga:/etc/icinga2# cat zones.d/master/icinga.example.com.conf
object Endpoint "icinga.example.com" {
}
root@icinga:/etc/icinga2# cat zones.d/master/davidw-test-icinga-agent.conf
object Endpoint "davidw-test-icinga-agent" {
host = "172.x.x.72"
}
object Host "davidw-test-icinga-agent" {
import "generic-host"
address = "172.x.x.72"
check_command = "hostalive"
vars.os = "Linux"
vars.client_endpoint = name
}
I have verified that the icinga2 agent is running on the davidw-test-icinga-agent server on port 5665. Here’s some information on the server with the agent:
root@davidw-test-icinga-agent:/home/dwhite# icinga2 object list --type endpoint
Object 'icinga.example.com' of type 'Endpoint':
% declared in '/etc/icinga2/zones.conf', lines 6:1-6:47
* __name = "icinga.example.com"
* host = "icinga.example.com"
% = modified in '/etc/icinga2/zones.conf', lines 7:2-7:39
* log_duration = 86400
* name = "icinga.example.com"
* package = "_etc"
* port = "5665"
% = modified in '/etc/icinga2/zones.conf', lines 8:2-8:14
* source_location
* first_column = 1
* first_line = 6
* last_column = 47
* last_line = 6
* path = "/etc/icinga2/zones.conf"
* templates = [ "icinga.example.com" ]
% = modified in '/etc/icinga2/zones.conf', lines 6:1-6:47
* type = "Endpoint"
* zone = ""
Object 'davidw-test-icinga-agent' of type 'Endpoint':
% declared in '/etc/icinga2/zones.conf', lines 15:1-15:42
* __name = "davidw-test-icinga-agent"
* host = ""
* log_duration = 86400
* name = "davidw-test-icinga-agent"
* package = "_etc"
* port = "5665"
* source_location
* first_column = 1
* first_line = 15
* last_column = 42
* last_line = 15
* path = "/etc/icinga2/zones.conf"
* templates = [ "davidw-test-icinga-agent" ]
% = modified in '/etc/icinga2/zones.conf', lines 15:1-15:42
* type = "Endpoint"
* zone = ""
What am I missing, and/or what other information can I provide to help troubleshoot this problem? It may be worth noting that IIRC, the hostname of the master server (icinga.example.com) was something else prior to my initial installation & configuration of the icinga2 server software.