Icinga2 issue - critical/Socket: accept() failed with error code 9, "Bad file descriptor"

Icinga 2
1

Hi,

I’ve Icinga2 2.15.1-1 running with Icingaweb2 2.12.5 on Ubuntu 24.04 LTS. The command transport is Icinga2 API. Using IDO/Mysql with MariaDB-10.11.13. Also Livestatus enabled for “unix” socket and on a tcp port. The cluster hass a few satellite zones. It is not using director or icinga2 agent. LimitNOFILE=524288

This issue came up for the first time since the cluster was put in production 8 months ago. There was a service config change which required the icinga2 service reload. After reload, the service failed to accept connections even it was running on the config master.

[2026-08-28 15:03:48 -0700] information/Application: Received request to shut down.
[2026-08-28 15:03:48 -0700] information/Application: Shutting down…
[2026-08-28 15:03:48 -0700] information/CheckerComponent: ‘checker’ stopped.
[2026-08-28 15:03:48 -0700] information/NotificationComponent: ‘notification’ stopped.
[2026-08-28 15:03:48 -0700] information/DbConnection: ‘ido-mysql’ stopped.
[2026-08-28 15:03:48 -0700] information/LivestatusListener: ‘livestatus-tcp’ stopped.
[2026-08-28 15:03:50 -0700] critical/Socket: accept() failed with error code 9, “Bad file descriptor”
[2026-08-28 15:03:50 -0700] critical/LivestatusListener: Cannot accept new connection.
[2026-08-28 15:03:50 -0700] information/LivestatusListener: ‘livestatus’ stopped.[2026-08-28 15:03:50 -0700] critical/LivestatusListener: Cannot accept new connection.
[2026-08-28 15:03:50 -0700] information/ApiListener: ‘api’ stopped.

2026-08-28T15:03:15.951785-07:00 <maser1.com> safe-reload[2748274]: Validating config files: Done
2026-08-28T15:03:15.979336-07:00 <master1.com> safe-reload[2748274]: Reloading Icinga 2: Done

The secondary master reported this

[2026-08-28 15:03:50 -0700] information/ApiListener: Removing API client for endpoint <master1.com> 0 API clients left.
API client disconnected for identity <master1.com>
[2026-08-28 15:03:58 -0700] information/ApiListener: Reconnecting to endpoint <master1.com> via host <master1.com> and port ‘5665’
[2026-08-28 15:03:58 -0700] critical/ApiListener: Cannot connect to host <master1.com> on port ‘5665’: Connection refused [system:111 at /usr/include/boost/asi
o/detail/reactive_socket_connect_op.hpp:98 in function ‘do_complete’]

Any idea what could have gone wrong here? I understand that the livestatus is deprecated and we may move on to RestApi once the cluster gets updated to 2.16.5.

Thanks

2

The secondary master also reported this message repeatedly until I restarted icinga2 service on the master1. The cluster has been working normal since then.

critical/ApiListener: Error while syncing endpoint <master1.com> : Error: Cannot send message to already disconnected API client <master1.com>

3

Exactly.

Yes, please!

4

Thanks @Al2Klimov Any comment on a possible reason(s) for the Icinga2 service on the master1 which just stopped responding at the time?

One week on the above incident, the Icinga2 service reload timed out after the validation was completed after a config change. I suspect this is what happened with the above incident as well. I don’t think the livestatus has anything to do with it.

I’ve 10K hosts and 57K services configured on the cluster. Thanks