Backend seems to be not running correctly / No hosts showing up

chr.pie · June 9, 2022, 6:11pm

I have installed Icinga2, Icingaweb2 and Director on an Ubuntu 18.04 LTS. I chose MySQL for DB. Everything was setup and all checks I saw (like DB connection or API for Director calls) are working correctly. But there are no host showing up and Icingaweb tells me that Icinga is currently not operational:
grafik

I can add things in director and deploy them. And there are there after a restart of the Icinga Service.

I don’t know what are the right questions to ask as everything seems to be fine…

Give as much information as you can, e.g.

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.3-1)

Copyright (c) 2012-2022 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 18.04.6 LTS (Bionic Beaver)
  Kernel: Linux
  Kernel version: 5.4.0-42-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 8.4.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
root@Ubuntu-Template:/etc/icinga2/conf.d#

feature list
Disabled features: command compatlog elasticsearch gelf graphite influxdb influxdb2
Enabled features: api checker debuglog icingadb ido-mysql livestatus mainlog notification opentsdb perfdata statusdata syslog

icinga2 daemon -C
[2022-06-09 20:08:33 +0200] information/cli: Icinga application loader (version: r2.13.3-1)
[2022-06-09 20:08:33 +0200] information/cli: Loading configuration file(s).
[2022-06-09 20:08:33 +0200] information/ConfigItem: Committing config item(s).
[2022-06-09 20:08:33 +0200] information/ApiListener: My API identity: Ubuntu-Template
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 2 HostGroups.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 3 Hosts.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 13 Notifications.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 Downtime.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 IcingaDB.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 StatusDataWriter.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 3 Zones.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 OpenTsdbWriter.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 3 ApiUsers.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 LivestatusListener.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 244 CheckCommands.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 User.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 3 TimePeriods.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 PerfdataWriter.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 15 Services.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 3 ServiceGroups.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2022-06-09 20:08:33 +0200] information/ConfigItem: Instantiated 1 SyslogLogger.
[2022-06-09 20:08:33 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-06-09 20:08:33 +0200] information/cli: Finished validating the configuration file(s).

icinga2 object list --type Endpoint
Object 'Ubuntu-Template' of type 'Endpoint':
  % declared in '/etc/icinga2/zones.conf', lines 7:1-7:24
  * __name = "Ubuntu-Template"
  * host = "Ubuntu-Template"
    % = modified in '/etc/icinga2/zones.conf', lines 8:3-8:17
  * log_duration = 86400
  * name = "Ubuntu-Template"
  * package = "_etc"
  * port = "5665"
  * source_location
    * first_column = 1
    * first_line = 7
    * last_column = 24
    * last_line = 7
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "Ubuntu-Template" ]
    % = modified in '/etc/icinga2/zones.conf', lines 7:1-7:24
  * type = "Endpoint"
  * zone = ""

icinga2 object list --type Zone
Object 'global-templates' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 24:1-24:30
  * __name = "global-templates"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 25:3-25:15
  * name = "global-templates"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 24
    * last_column = 30
    * last_line = 24
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "global-templates" ]
    % = modified in '/etc/icinga2/zones.conf', lines 24:1-24:30
  * type = "Zone"
  * zone = ""

Object 'director-global' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 36:1-36:29
  * __name = "director-global"
  * endpoints = null
  * global = true
    % = modified in '/etc/icinga2/zones.conf', lines 37:3-37:15
  * name = "director-global"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 36
    * last_column = 29
    * last_line = 36
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "director-global" ]
    % = modified in '/etc/icinga2/zones.conf', lines 36:1-36:29
  * type = "Zone"
  * zone = ""

Object 'Ubuntu-Template' of type 'Zone':
  % declared in '/etc/icinga2/zones.conf', lines 11:1-11:20
  * __name = "Ubuntu-Template"
  * endpoints = [ "Ubuntu-Template" ]
    % = modified in '/etc/icinga2/zones.conf', lines 12:3-12:26
  * global = false
  * name = "Ubuntu-Template"
  * package = "_etc"
  * parent = ""
  * source_location
    * first_column = 1
    * first_line = 11
    * last_column = 20
    * last_line = 11
    * path = "/etc/icinga2/zones.conf"
  * templates = [ "Ubuntu-Template" ]
    % = modified in '/etc/icinga2/zones.conf', lines 11:1-11:20
  * type = "Zone"
  * zone = ""

nilmerg · June 10, 2022, 6:46am

Icinga Web detects this by looking at the database (IDO). If Icinga runs and the director works fine, it seems that Icinga does not write to the IDO. Or, not to the IDO which also Icinga Web is using.

Is there no error regarding the ido-mysql connection in the Icinga log? Have you maybe configured the wrong IDO for Icinga Web?

chr.pie · June 10, 2022, 2:32pm

This is the content of /var/log/icinga2/icinga2.log after restart:

[2022-06-10 16:27:08 +0200] information/IcingaDB: 'icingadb' started.
[2022-06-10 16:27:08 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-06-10 16:27:08 +0200] information/DbConnection: 'ido-mysql' started.
[2022-06-10 16:27:08 +0200] information/LivestatusListener: 'livestatus' started.
[2022-06-10 16:27:08 +0200] information/LivestatusListener: Created UNIX socket in '/run/icinga2/cmd/livestatus'.
[2022-06-10 16:27:08 +0200] information/PerfdataWriter: 'perfdata' resumed.
[2022-06-10 16:27:08 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-06-10 16:27:08 +0200] information/OpentsdbWriter: 'opentsdb' resumed.
[2022-06-10 16:27:08 +0200] information/NotificationComponent: 'notification' started.
[2022-06-10 16:27:08 +0200] information/CheckerComponent: 'checker' started.
[2022-06-10 16:27:08 +0200] information/ConfigItem: Activated all objects.
[2022-06-10 16:27:08 +0200] warning/OpenTsdbWriter: Can't connect to OpenTSDB on host '127.0.0.1' port '4242.'
[2022-06-10 16:27:08 +0200] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2022-06-10 16:27:08 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2022-06-10 16:27:08 +0200] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.15.1')
[2022-06-10 16:27:08 +0200] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga2' in 0.0883908 second(s).
[2022-06-10 16:27:13 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-06-10 16:27:13 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-06-10 16:27:18 +0200] information/WorkQueue: #7 (ApiListener, RelayQueue) items: 0, rate: 0.283333/s (17/min 17/5min 17/15min);
[2022-06-10 16:27:18 +0200] information/WorkQueue: #8 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2022-06-10 16:27:18 +0200] warning/OpenTsdbWriter: Can't connect to OpenTSDB on host '127.0.0.1' port '4242.'
[2022-06-10 16:27:18 +0200] information/IdoMysqlConnection: Pending queries: 7 (Input: 3/s; Output: 3/s)
[2022-06-10 16:27:18 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-06-10 16:27:18 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-06-10 16:27:23 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-06-10 16:27:23 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-06-10 16:27:28 +0200] warning/OpenTsdbWriter: Can't connect to OpenTSDB on host '127.0.0.1' port '4242.'
[2022-06-10 16:27:28 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-06-10 16:27:28 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-06-10 16:27:33 +0200] information/IcingaDB: Trying to connect to Redis server (async) on host '127.0.0.1:6380'
[2022-06-10 16:27:33 +0200] critical/IcingaDB: Cannot connect to 127.0.0.1:6380: Connection refused
[2022-06-10 16:27:38 +0200] warning/OpenTsdbWriter: Can't connect to OpenTSDB on host '127.0.0.1' port '4242.'

I don’t see any errors regarding “ido-mysql”. I don’t know if OpenTSDB or Redis is a problem?