It seems that Icinga DB is not running. Make sure Icinga DB is running and writing into the database

Icinga 2
1

I finally made it through all the errors in the gui wizard
when I log in to the web interface I get this

image
image1185×917 58.6 KB

seems weird because I validated configuration for different databases numerous times during the wizard
I do see this but I am unable to determine what module or ??? is not able to connect

netadmin@monitor:~$ sudo -i
[sudo] password for netadmin:
root@monitor:~# systemctl enable --now icingadb
root@monitor:~# systemctl status icingadb
● icingadb.service - Icinga DB
Loaded: loaded (/usr/lib/systemd/system/icingadb.service; enabled; preset: enabled)
Active: active (running) since Thu 2025-09-25 14:50:13 UTC; 4min 48s ago
Main PID: 1304 (icingadb)
Tasks: 6 (limit: 14258)
Memory: 12.7M (peak: 13.1M)
CPU: 22ms
CGroup: /system.slice/icingadb.service
└─1304 /usr/sbin/icingadb --config /etc/icingadb/config.yml

Sep 25 14:50:13 monitor systemd[1]: Started icingadb.service - Icinga DB.
Sep 25 14:50:13 monitor icingadb[1304]: Connecting to database at ‘mysql://icingadb@localhost:5432/icingadb’
Sep 25 14:51:13 monitor icingadb[1304]: database: [packets.go:58 unexpected EOF]
Sep 25 14:51:13 monitor icingadb[1304]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 14:52:13 monitor icingadb[1304]: database: [packets.go:58 unexpected EOF]
Sep 25 14:52:13 monitor icingadb[1304]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 14:53:13 monitor icingadb[1304]: database: [packets.go:58 unexpected EOF]
Sep 25 14:53:13 monitor icingadb[1304]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 14:54:14 monitor icingadb[1304]: database: [packets.go:58 unexpected EOF]
Sep 25 14:54:14 monitor icingadb[1304]: database: Can’t connect to database. Retrying error=“invalid connection”

I see this below in one of the logs but unsure what to do with it
thanks for your patience with me and your help and time

[2025-09-25 10:55:36 -0500] information/ApiListener: ‘api’ started.
[2025-09-25 10:55:36 -0500] information/ApiListener: Started new listener on ‘[::]:5665’
[2025-09-25 10:55:36 -0500] information/DbConnection: ‘ido-pgsql’ started.
[2025-09-25 10:55:36 -0500] information/IcingaDB: ‘icingadb’ started.
[2025-09-25 10:55:36 -0500] information/NotificationComponent: ‘notification’ started.
[2025-09-25 10:55:36 -0500] information/CheckerComponent: ‘checker’ started.
[2025-09-25 10:55:36 -0500] information/ConfigItem: Activated all objects.
[2025-09-25 10:55:36 -0500] information/IdoPgsqlConnection: ‘ido-pgsql’ resumed.
[2025-09-25 10:55:36 -0500] information/DbConnection: Resuming IDO connection: ido-pgsql
[2025-09-25 10:55:36 -0500] information/IcingaDB: Trying to connect to Redis server (async) on host ‘127.0.0.1:6380’
[2025-09-25 10:55:36 -0500] information/IcingaDB: Connected to Redis server
[2025-09-25 10:55:36 -0500] information/IcingaDB: Starting initial config/status dump
[2025-09-25 10:55:36 -0500] information/IdoPgsqlConnection: PGSQL IDO instance id: 1 (schema version: ‘1.14.3’)
[2025-09-25 10:55:36 -0500] information/IcingaDB: Initial config/status dump finished in 0.0997579 seconds.
[2025-09-25 10:55:36 -0500] information/IdoPgsqlConnection: Finished reconnecting to ‘ido-pgsql’ database ‘icinga2’ in 0.139461 second(s).
[2025-09-25 10:55:46 -0500] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.233333/s (14/min 14/5min 14/15min);
[2025-09-25 10:55:46 -0500] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2025-09-25 10:55:46 -0500] information/IdoPgsqlConnection: Pending queries: 2 (Input: 3/s; Output: 3/s)
[2025-09-25 10:55:46 -0500] information/IcingaDB: Pending queries: 0 (Input: 1/s; Output: 1/s)
[2025-09-25 10:58:02 -0500] information/Checkable: Checkable ‘monitor!procs’ has 1 notification(s). Checking filters for type ‘Problem’, sends will be logged.
[2025-09-25 10:58:02 -0500] information/Notification: Sending ‘Problem’ notification ‘monitor!procs!mail-icingaadmin’ for user ‘icingaadmin’
[2025-09-25 10:58:02 -0500] information/Notification: Completed sending ‘Problem’ notification ‘monitor!procs!mail-icingaadmin’ for checkable ‘monitor!procs’ and user ‘icingaadmin’ using command 'mail-serv>
[2025-09-25 10:58:02 -0500] warning/PluginNotificationTask: Notification command for object ‘monitor!procs’ (PID: 1908, arguments: ‘/etc/icinga2/scripts/mail-service-notification.sh’ ‘-4’ ‘127.0.0.1’ ‘-6’ >
[2025-09-25 11:00:36 -0500] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2025-09-25 11:00:46 -0500] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.966667/s (58/min 283/5min 297/15min);
[2025-09-25 11:00:46 -0500] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2025-09-25 11:00:46 -0500] information/IcingaDB: Pending queries: 0 (Input: 1/s; Output: 1/s)
[2025-09-25 11:00:56 -0500] information/IdoPgsqlConnection: Pending queries: 0 (Input: 4/s; Output: 4/s)
[2025-09-25 11:01:54 -0500] information/Checkable: Checkable ‘monitor!procs’ has 1 notification(s). Checking filters for type ‘Recovery’, sends will be logged.
[2025-09-25 11:01:54 -0500] information/Notification: Sending ‘Recovery’ notification ‘monitor!procs!mail-icingaadmin’ for user ‘icingaadmin’
[2025-09-25 11:01:54 -0500] information/Notification: Completed sending ‘Recovery’ notification ‘monitor!procs!mail-icingaadmin’ for checkable ‘monitor!procs’ and user ‘icingaadmin’ using command 'mail-ser>
[2025-09-25 11:01:54 -0500] warning/PluginNotificationTask: Notification command for object ‘monitor!procs’ (PID: 2125, arguments: ‘/etc/icinga2/scripts/mail-service-notification.sh’ ‘-4’ ‘127.0.0.1’ ‘-6’ >
[2025-09-25 11:05:36 -0500] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2025-09-25 11:05:46 -0500] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2025-09-25 11:05:46 -0500] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.933333/s (56/min 269/5min 572/15min);
[2025-09-25 11:05:46 -0500] information/IcingaDB: Pending queries: 0 (Input: 3/s; Output: 3/s)
[2025-09-25 11:05:56 -0500] information/IdoPgsqlConnection: Pending queries: 0 (Input: 3/s; Output: 3/s)
[2025-09-25 11:10:36 -0500] information/ConfigObject: Dumping program state to file ‘/var/lib/icinga2/icinga2.state’
[2025-09-25 11:10:46 -0500] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.866667/s (52/min 256/5min 814/15min);
[2025-09-25 11:10:46 -0500] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2025-09-25 11:10:46 -0500] information/IcingaDB: Pending queries: 0 (Input: 2/s; Output: 2/s)
[2025-09-25 11:10:56 -0500] information/IdoPgsqlConnection: Pending queries: 0 (Input: 4/s; Output: 4/s)

2

enabled icinga2 debug
tailing the log I do not see any errors or failures

3

Did you set up the icingadb database?

4

yes and “validated configuration” of several databases including icingadb during the gui setup wizard
schema update ran

Name | Owner | Encoding | Locale Provider | Collate | Ctype | ICU Locale | ICU Rules | Access privileges | Size | Tablespace | Description
------------±-----------±---------±----------------±------------±------------±-----------±----------±----------------------±--------±-----------±-------------------------------------------
director | director | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | | | 7345 kB | pg_default |
icinga2 | icinga2 | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | | | 14 MB | pg_default |
icingadb | icingadb | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | | | 10 MB | pg_default |
icingaweb2 | icingaweb2 | UTF8 | libc | en_US.UTF-8 | en_US.UTF-8 | | | | 7908 kB | pg_default |

Role name | Attributes
------------±-----------------------------------------------------------
icingadb | Superuser

6

icinga2 debug log
looks like it is talking to the database

[2025-09-25 13:29:49 -0500] debug/IcingaDB: Firing and forgetting query: ‘HSET’ ‘icinga:service:state’ ‘bfce42f62d06e353d46bfed58f2d413ab7ceaa36’ ‘{“affects_children”:false,“check_attempt”:1,"check_commandlin…’
[2025-09-25 13:29:49 -0500] debug/IcingaDB: Firing and forgetting query: ‘HSET’ ‘icinga:checksum:service:state’ ‘bfce42f62d06e353d46bfed58f2d413ab7ceaa36’ ‘{“checksum”:“6bb00a1235fc881a329903269ee9b3a9a1a676f5”}’
[2025-09-25 13:29:49 -0500] debug/IcingaDB: Firing and forgetting query: ‘ZADD’ ‘icinga:nextupdate:service’ ‘1758825117.201694’ ‘bfce42f62d06e353d46bfed58f2d413ab7ceaa36’
[2025-09-25 13:29:49 -0500] debug/IdoPgsqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = ‘0’, active_checks_enabled = ‘1’, check_command = ‘ping4’, check_source = ‘monitor’, check_timeperiod_object_id = NULL, check_type = ‘0’, current_check_attempt = ‘1’, current_notification_number = ‘0’, current_state = ‘0’, endpoint_object_id = 250, event_handler_enabled = ‘1’, execution_time = ‘4.123524’, flap_detection_enabled = ‘0’, has_been_checked = ‘1’, instance_id = 1, is_flapping = ‘0’, is_reachable = ‘1’, last_check = TO_TIMESTAMP(1758824990) AT TIME ZONE ‘UTC’, last_hard_state = ‘0’, last_hard_state_change = TO_TIMESTAMP(1758214968) AT TIME ZONE ‘UTC’, last_notification = NULL, last_state_change = TO_TIMESTAMP(1758214968) AT TIME ZONE ‘UTC’, last_time_critical = NULL, last_time_ok = TO_TIMESTAMP(1758824990) AT TIME ZONE ‘UTC’, last_time_unknown = NULL, last_time_warning = NULL, latency = ‘0.000411’, long_output = ‘’, max_check_attempts = ‘5’, next_check = TO_TIMESTAMP(1758825049) AT TIME ZONE ‘UTC’, next_notification = TO_TIMESTAMP(1758825398) AT TIME ZONE ‘UTC’, normal_check_interval = ‘1’, notifications_enabled = ‘1’, original_attributes = ‘null’, output = ‘PING OK - Packet loss = 0%, RTA = 0.03 ms’, passive_checks_enabled = ‘1’, percent_state_change = ‘0’, perfdata = ‘rta=0.031000ms;100.000000;200.000000;0.000000 pl=0%;5;15;0;’, problem_has_been_acknowledged = ‘0’, process_performance_data = ‘1’, retry_check_interval = ‘0.500000’, scheduled_downtime_depth = ‘0’, service_object_id = 264, should_be_scheduled = ‘1’, state_type = ‘1’, status_update_time = TO_TIMESTAMP(1758824990) AT TIME ZONE ‘UTC’ WHERE service_object_id = 264
[2025-09-25 13:29:49 -0500] debug/IdoPgsqlConnection: Query: COMMIT
[2025-09-25 13:29:49 -0500] debug/IcingaDB: Firing and forgetting query: ‘XADD’ ‘icinga:stats’ ‘MAXLEN’ ‘1’ ‘*’ ‘IcingaApplication’ ‘{“status”:{“icingaapplication”:{“app”:{"enable_event_handlers…’ …
[2025-09-25 13:29:49 -0500] debug/IdoPgsqlConnection: Query: BEGIN

7

is there a way to determine what cant connect to a postgresql database?
sudo journalctl -xeu icingadb

Sep 25 11:54:53 monitor icingadb[1323]: Connecting to database at ‘mysql://icingadb@localhost:5432/icingadb’
Sep 25 11:55:53 monitor icingadb[1323]: database: [packets.go:58 unexpected EOF]
Sep 25 11:55:53 monitor icingadb[1323]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 11:56:53 monitor icingadb[1323]: database: [packets.go:58 unexpected EOF]
Sep 25 11:56:53 monitor icingadb[1323]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 11:57:54 monitor icingadb[1323]: database: [packets.go:58 unexpected EOF]
Sep 25 11:57:54 monitor icingadb[1323]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 11:58:55 monitor icingadb[1323]: database: [packets.go:58 unexpected EOF]
Sep 25 11:58:55 monitor icingadb[1323]: database: Can’t connect to database. Retrying error=“invalid connection”
Sep 25 11:59:56 monitor icingadb[1323]: database: [packets.go:58 unexpected EOF]
Sep 25 11:59:56 monitor icingadb[1323]: Can’t connect to database error=“can’t connect to database: retry deadline exceeded: invalid connection”
Sep 25 11:59:56 monitor systemd[1]: icingadb.service: Main process exited, code=exited, status=1/FAILURE

root@monitor:/etc/postgresql/16/main# sudo netstat -tlnp | grep 5432
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1191/postgres

9

image
image1653×694 69.5 KB

11

Not sure if firing and forgetting counts as a working DB connection.

On the other hand the IDO looks good to me.

12

This screenshot shows the configuration of the web frontend part of icingadb.

The icingadb backend, especially the DB, is configured in /etc/icingadb/config.yml .

13

Your screenshot says PostgreSQL, but it seems like you have configured MySQL in the Icinga DB configuration. Try setting database.type to pgsql, https://icinga.com/docs/icinga-db/latest/doc/03-Configuration/#database-configuration.

Btw, you have created a few threads about Icinga database in the last days, about both IDO and Icinga DB. Unless you have very special reasons, you should only need one of those - and that should be Icinga DB. IDO is deprecated and Icinga DB should be used for new setups.

1 Like
14

Regarding the second Redis error in your initial post: Have you set up and configured Redis?

Please share your Icinga 2 IcingaDB object configuration - usually defined in /etc/icinga2/features-enabled/icingadb.conf - and the redis section of the Icinga DB configuration - located in /etc/icingadb/config.yml - with us. Please redact any passwords or other sensitive information.

1 Like