Icinga2 crashed unexpected

lobr · October 7, 2021, 6:57am

Hello Icinga-community,

for some reason my Icinga system crashed yesterday. It is the first time I experienced this in more than a year.

Because of this the influxDB has a data gap.

What might has caused this?

At 12:40 it received the request to shutdown (wasn’t done by a human interaction):

[2021-10-06 12:40:07 +0200] information/Application: Received request to shut down.
[2021-10-06 12:40:07 +0200] information/Application: Shutting down...
[2021-10-06 12:40:07 +0200] information/CheckerComponent: 'checker' stopped.
[2021-10-06 12:40:07 +0200] information/NotificationComponent: 'notification' stopped.
[2021-10-06 12:40:07 +0200] information/ExternalCommandListener: 'command' stopped.
[2021-10-06 12:40:07 +0200] information/DbConnection: Pausing IDO connection: ido-mysql
[2021-10-06 12:40:07 +0200] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga_ido'.
[2021-10-06 12:40:07 +0200] information/IdoMysqlConnection: 'ido-mysql' paused.
[2021-10-06 12:40:07 +0200] information/DbConnection: 'ido-mysql' stopped.
[2021-10-06 12:40:08 +0200] information/InfluxdbWriter: 'influxdb' paused.
[2021-10-06 12:40:08 +0200] information/ApiListener: 'api' stopped.
[2021-10-06 12:40:08 +0200] information/FileLogger: 'main-log' started.
[2021-10-06 12:40:08 +0200] information/ApiListener: 'api' started.
[2021-10-06 12:40:08 +0200] information/ApiListener: Copying 5 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
[2021-10-06 12:40:08 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/archive_citrix_custom_commands.conf
[2021-10-06 12:40:08 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/archive_citrix_services.conf
[2021-10-06 12:40:08 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
[2021-10-06 12:40:08 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/services.conf
[2021-10-06 12:40:08 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates.conf
[2021-10-06 12:40:08 +0200] information/ApiListener: Copying 7 zone configuration files for zone 'otn-icinga-sat-hedvig' to '/var/lib/icinga2/api/zones/otn-icinga-sat-hedvig'.

Then it was back up exactly 5 minutes before it has been shutdown again at 12:45:

[2021-10-06 12:45:08 +0200] information/Application: Received request to shut down.
[2021-10-06 12:45:08 +0200] information/Application: Shutting down...
[2021-10-06 12:45:08 +0200] information/CheckerComponent: 'checker' stopped.
[2021-10-06 12:45:08 +0200] information/NotificationComponent: 'notification' stopped.
[2021-10-06 12:45:08 +0200] information/ExternalCommandListener: 'command' stopped.
[2021-10-06 12:45:08 +0200] information/DbConnection: Pausing IDO connection: ido-mysql
[2021-10-06 12:45:08 +0200] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga_ido'.
[2021-10-06 12:45:08 +0200] information/IdoMysqlConnection: 'ido-mysql' paused.
[2021-10-06 12:45:08 +0200] information/DbConnection: 'ido-mysql' stopped.
[2021-10-06 12:45:08 +0200] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-10-06 12:45:08 +0200] critical/Application: Icinga 2 has terminated unexpectedly. Additional information can be found in '/var/log/icinga2/crash/report.1633517108.859343'

For the next 5h these messages iterated:

[2021-10-06 14:24:11 +0200] information/ApiListener: New client connection for identity 'otn-icinga-sat-hedvig.mon.infra.domain.com' from [10.116.219.179]:33766
[2021-10-06 14:24:13 +0200] information/ApiListener: New client connection for identity 'k8s-icinga.domain.com' from [10.116.219.121]:32842
[2021-10-06 14:24:18 +0200] information/ApiListener: New client connection for identity 'otn-ac-ics-hfip.infra.domain.com' from [10.116.219.177]:59510

At 17:45 it came back:

[2021-10-06 17:45:40 +0200] information/ApiListener: 'api' started.
[2021-10-06 17:45:40 +0200] information/ApiListener: Copying 5 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/archive_citrix_custom_commands.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/archive_citrix_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Copying 3 zone configuration files for zone 'ac-data-hln' to '/var/lib/icinga2/api/zones/ac-data-hln'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-data-hln//_etc/helion_fIP_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-data-hln//_etc/hln_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-data-hln//_etc/hln_templates.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Copying 1 zone configuration files for zone 'icinga2' to '/var/lib/icinga2/api/zones/icinga2'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga2//_etc/hosts/hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Copying 7 zone configuration files for zone 'otn-icinga-sat-hedvig' to '/var/lib/icinga2/api/zones/otn-icinga-sat-hedvig'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/archived_hedvig_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/archived_hedvig_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_custom_commands.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_hosts_dcc2.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_templates.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Copying 5 zone configuration files for zone 'ac-citrix-zone' to '/var/lib/icinga2/api/zones/ac-citrix-zone'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/DEL_citrix_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/archive_citrix_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/citrix_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/citrix_hv.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/citrix_templates.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Copying 7 zone configuration files for zone 'k8s-icinga' to '/var/lib/icinga2/api/zones/k8s-icinga'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/archive_k8s_custom_checks.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/archive_k8s_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/archive_k8s_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_custom_checks.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_hosts.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_services.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_templates.conf
[2021-10-06 17:45:40 +0200] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-10-06 17:45:40 +0200] information/ExternalCommandListener: 'command' started.
[2021-10-06 17:45:40 +0200] warning/ExternalCommandListener: This feature is DEPRECATED and may be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/mile$
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'

[2021-10-06 17:45:40 +0200] information/ApiListener: Reconnecting to endpoint 'k8s-icinga.domain.com' via host 'k8s-icinga.domain.com' and port '566$
[2021-10-06 17:45:40 +0200] information/ApiListener: Reconnecting to endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' via host 'otn-icinga-sat-citrix.mon.infra.ai$
[2021-10-06 17:45:40 +0200] information/ApiListener: Reconnecting to endpoint 'otn-ac-ics-hfip.infra.domain.com' via host '[ "otn-ac-ics-hfip.infra.domain.com
[2021-10-06 17:45:40 +0200] information/ApiListener: Reconnecting to endpoint 'otn-icinga-sat-hedvig.mon.infra.domain.com' via host 'otn-icinga-sat-hedvig.mon.infra.ai$
[2021-10-06 17:45:40 +0200] information/DbConnection: 'ido-mysql' started.
[2021-10-06 17:45:40 +0200] critical/ApiListener: Cannot connect to host '[ "otn-ac-ics-hfip.infra.domain.com" ]' on port '5665': Host not found (authoritative)
[2021-10-06 17:45:40 +0200] information/InfluxdbWriter: 'influxdb' resumed.
[2021-10-06 17:45:40 +0200] information/NotificationComponent: 'notification' started.
[2021-10-06 17:45:40 +0200] information/CheckerComponent: 'checker' started.
[2021-10-06 17:45:40 +0200] information/ConfigItem: Activated all objects.
[2021-10-06 17:45:40 +0200] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2021-10-06 17:45:40 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2021-10-06 17:45:40 +0200] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2021-10-06 17:45:40 +0200] information/ApiListener: New client connection for identity 'otn-icinga-sat-citrix.mon.infra.domain.com' to [10.116.219.65]:5665
[2021-10-06 17:45:40 +0200] information/ApiListener: Finished reconnecting to endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' via host 'otn-icinga-sat-citrix.mon$
[2021-10-06 17:45:40 +0200] information/ApiListener: Sending config updates for endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' in zone 'ac-citrix-zone'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com
[2021-10-06 17:45:40 +0200] information/ApiListener: Syncing configuration files for zone 'ac-citrix-zone' to endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Finished sending config file updates for endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' in zone 'ac-citrix-$
[2021-10-06 17:45:40 +0200] information/ApiListener: Syncing runtime objects to endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' in zone 'ac-citr$
[2021-10-06 17:45:40 +0200] information/ApiListener: Sending replay log for endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' in zone 'ac-citrix-zone'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Finished sending replay log for endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' in zone 'ac-citrix-zone'.
[2021-10-06 17:45:40 +0200] information/ApiListener: Finished syncing endpoint 'otn-icinga-sat-citrix.mon.infra.domain.com' in zone 'ac-citrix-zone'.
[2021-10-06 17:45:40 +0200] information/JsonRpcConnection: Received certificate request for CN 'otn-icinga-sat-citrix.mon.infra.domain.com' signed by our CA.
[2021-10-06 17:45:40 +0200] information/JsonRpcConnection: The certificate for CN 'otn-icinga-sat-citrix.mon.infra.domain.com' is valid and uptodate. Skipping automate$
[2021-10-06 17:45:40 +0200] information/ApiListener: New client connection for identity 'otn-icinga-sat-hedvig.mon.infra.domain.com' to [10.116.219.179]:5665
[2021-10-06 17:45:40 +0200] information/ApiListener: Sending config updates for endpoint 'otn-icinga-sat-hedvig.mon.infra.domain.com' in zone 'otn-icinga-sat-he

/var/log/icinga2/crash/report.1633517108.859343

Caught SIGABRT.
Current time: 2021-10-06 12:45:08 +0200

  Application version: 2.13.1-1

System information:
  Platform: Red Hat Enterprise Linux Server
  Platform version: 7.9 (Maipo)
  Kernel: Linux
  Kernel version: 3.10.0-1127.10.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-507-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Stacktrace:
 0# icinga::Application::SigAbrtHandler(int) in /usr/lib64/icinga2/sbin/icinga2
 1# 0x00007F2AC1778630 in /lib64/libpthread.so.0
 2# gsignal in /lib64/libc.so.6
 3# abort in /lib64/libc.so.6
 4# 0x00007F2AC1413F67 in /lib64/libc.so.6
 5# 0x00007F2AC141A474 in /lib64/libc.so.6
 6# 0x00007F2AC141C56B in /lib64/libc.so.6
 7# icinga::InfluxdbCommonWriter::Flush() in /usr/lib64/icinga2/sbin/icinga2
 8# icinga::InfluxdbCommonWriter::FlushTimeoutWQ() in /usr/lib64/icinga2/sbin/icinga2
 9# icinga::WorkQueue::RunTaskFunction(std::function<void ()> const&) in /usr/lib64/icinga2/sbin/icinga2
10# icinga::WorkQueue::WorkerThreadProc() in /usr/lib64/icinga2/sbin/icinga2
11# 0x00007F2AC3678954 in /lib64/libboost_thread.so.1.69.0
12# 0x00007F2AC1770EA5 in /lib64/libpthread.so.0
13# clone in /lib64/libc.so.6

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://github.com/Icinga/icinga2 and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***

steaksauce · October 7, 2021, 2:47pm

You might check outside of the Icinga logs (ie /var/log/messages) and see what sent the request. Doesn’t look like a crash, looks like a normal restart/stop&start.

We have some watchdog daemon’s in our environment that will do this from time to time. For instance, one daemon says that “if the Icinga2 API stops responding, restart Icinga2”

lobr · October 14, 2021, 3:27pm

Hi @steaksauce ,

those restarts do happen without any exception every 30 minutes. The messages log does not show any suspicious action to me.

/var/log/messages

Oct 14 16:59:00 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:00 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 436f1846-2cff-11ec-9dd8-005056a1d026 45018
Oct 14 16:59:10 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:10 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 4964ba06-2cff-11ec-9dd9-005056a1d026 4479
Oct 14 16:59:20 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:20 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 4f5aaeb5-2cff-11ec-9dda-005056a1d026 3086
Oct 14 16:59:30 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:30 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 5550a4eb-2cff-11ec-9ddb-005056a1d026 6319
Oct 14 16:59:40 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:40 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 5b4675dd-2cff-11ec-9ddc-005056a1d026 3115
Oct 14 16:59:50 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:50 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 613e6491-2cff-11ec-9ddd-005056a1d026 11741
Oct 14 16:59:55 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:16:59:55 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 64936792-2cff-11ec-9dde-005056a1d026 6091
Oct 14 17:00:00 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:00 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 67329a7a-2cff-11ec-9ddf-005056a1d026 5876
Oct 14 17:00:05 icinga2 systemd: Reloading Icinga host/service/network monitoring system.
Oct 14 17:00:06 icinga2 safe-reload: Validating config files: Done
Oct 14 17:00:06 icinga2 safe-reload: Reloading Icinga 2: Done
Oct 14 17:00:08 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:08 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 6c2c735b-2cff-11ec-9de0-005056a1d026 3169
Oct 14 17:00:09 icinga2 systemd: Reloaded Icinga host/service/network monitoring system.
Oct 14 17:00:09 icinga2 icinga2: This feature is DEPRECATED and may be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones#012Context:#012#011(0) Activating object 'command' of type 'ExternalCommandListener'
Oct 14 17:00:09 icinga2 icinga2: Cannot connect to host '[ "otn-ac-ics-hfip.infra.domain.com" ]' on port '5665': Host not found (authoritative)
Oct 14 17:00:19 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:19 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 72a0593a-2cff-11ec-9de1-005056a1d026 5326
Oct 14 17:00:21 icinga2 icinga2: API client disconnected for identity 'otn-ac-ics-hfip.infra.domain.com'
Oct 14 17:00:21 icinga2 icinga2: Removing API client for endpoint 'otn-ac-ics-hfip.infra.domain.com'. 0 API clients left.
Oct 14 17:00:24 icinga2 icinga2: Timeout while reconnecting to endpoint 'k8s-icinga.domain.com' via host 'k8s-icinga.domain.com' and port '5665', cancelling attempt
Oct 14 17:00:27 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:27 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 77ae4dfe-2cff-11ec-9de2-005056a1d026 45249
Oct 14 17:00:29 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:29 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 789663e2-2cff-11ec-9de3-005056a1d026 2528
Oct 14 17:00:37 icinga2 icinga2: Ignoring invalid perfdata for checkable 'VSA - ac-VSA01!vsa_network' and command 'check_vsa_network' with value: sent={sent}#012Context:#012#011(0) Processing check result for 'VSA - ac-VSA01!vsa_network'
Oct 14 17:00:37 icinga2 icinga2: Ignoring invalid perfdata for checkable 'VSA - ac-VSA01!vsa_network' and command 'check_vsa_network' with value: received={received}".format(received=calc_data_recv(),#012Context:#012#011(0) Processing check result for 'VSA - ac-VSA01!vsa_network'
Oct 14 17:00:37 icinga2 icinga2: Ignoring invalid perfdata for checkable 'VSA - ac-VSA01!vsa_network' and command 'check_vsa_network' with value: sent=calc_data_sent())#012Context:#012#011(0) Processing check result for 'VSA - ac-VSA01!vsa_network'
Oct 14 17:00:39 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:39 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 7e8c3238-2cff-11ec-9de4-005056a1d026 9111
Oct 14 17:00:46 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:46 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 82f52e6f-2cff-11ec-9de5-005056a1d026 5955
Oct 14 17:00:49 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:49 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 848287a6-2cff-11ec-9de6-005056a1d026 11383
Oct 14 17:00:58 icinga2 icinga2: Ignoring invalid perfdata for checkable 'pnode09-1-m3!ncpa_network_hedvig_600' and command 'check_hedvig_ncpa_network' with value: sent={sent}#012Context:#012#011(0) Processing check result for 'pnode09-1-m3!ncpa_network_hedvig_600'
Oct 14 17:00:58 icinga2 icinga2: Ignoring invalid perfdata for checkable 'pnode09-1-m3!ncpa_network_hedvig_600' and command 'check_hedvig_ncpa_network' with value: received={received}".format(received=calc_data_recv(),#012Context:#012#011(0) Processing check result for 'pnode09-1-m3!ncpa_network_hedvig_600'
Oct 14 17:00:58 icinga2 icinga2: Ignoring invalid perfdata for checkable 'pnode09-1-m3!ncpa_network_hedvig_600' and command 'check_hedvig_ncpa_network' with value: sent=calc_data_sent())#012Context:#012#011(0) Processing check result for 'pnode09-1-m3!ncpa_network_hedvig_600'
Oct 14 17:00:59 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:00:59 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 8a78336f-2cff-11ec-9de7-005056a1d026 13363
Oct 14 17:01:09 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:01:09 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 906e78c0-2cff-11ec-9de8-005056a1d026 3108
Oct 14 17:01:19 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:01:19 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 966417b3-2cff-11ec-9de9-005056a1d026 4264
Oct 14 17:01:29 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:01:29 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" 9c5a0c63-2cff-11ec-9dea-005056a1d026 3816
Oct 14 17:01:39 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:01:39 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" a25064a8-2cff-11ec-9deb-005056a1d026 12357
Oct 14 17:01:49 icinga2 influxd: [httpd] 127.0.0.1 - icinga [14/Oct/2021:17:01:49 +0200] "POST /write?db=icinga2&p=%5BREDACTED%5D&precision=s&u=icinga HTTP/1.0" 204 0 "-" "Icinga/2.13.1-1" a846077d-2cff-11ec-9dec-005056a1d026 13368

/var/log/icinga2/icinga2.log

[2021-10-14 16:58:47 +0200] information/Checkable: Notifications are disabled for checkable 'Helion MGMT 10.116.236.24 - otncloud-ccp-nn-m2!ping4'.
[2021-10-14 16:58:47 +0200] information/Checkable: Notifications are disabled for checkable 'Helion MGMT 10.116.236.25 - otncloud-ccp-nn-m3!ping4'.
[2021-10-14 17:00:07 +0200] information/Application: Received request to shut down.
[2021-10-14 17:00:08 +0200] information/Application: Shutting down...
[2021-10-14 17:00:08 +0200] information/CheckerComponent: 'checker' stopped.
[2021-10-14 17:00:08 +0200] information/NotificationComponent: 'notification' stopped.
[2021-10-14 17:00:08 +0200] information/ExternalCommandListener: 'command' stopped.
[2021-10-14 17:00:08 +0200] information/DbConnection: Pausing IDO connection: ido-mysql
[2021-10-14 17:00:08 +0200] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga_ido'.
[2021-10-14 17:00:08 +0200] information/IdoMysqlConnection: 'ido-mysql' paused.
[2021-10-14 17:00:08 +0200] information/DbConnection: 'ido-mysql' stopped.
[2021-10-14 17:00:08 +0200] information/InfluxdbWriter: 'influxdb' paused.
[2021-10-14 17:00:08 +0200] information/ApiListener: 'api' stopped.
[2021-10-14 17:00:09 +0200] information/FileLogger: 'main-log' started.
[2021-10-14 17:00:09 +0200] information/ApiListener: 'api' started.
[2021-10-14 17:00:09 +0200] information/ApiListener: Copying 5 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/archive_citrix_custom_commands.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/archive_citrix_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Copying 3 zone configuration files for zone 'ac-data-hln' to '/var/lib/icinga2/api/zones/ac-data-hln'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-data-hln//_etc/helion_fIP_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-data-hln//_etc/hln_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-data-hln//_etc/hln_templates.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Copying 1 zone configuration files for zone 'icinga2' to '/var/lib/icinga2/api/zones/icinga2'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga2//_etc/hosts/hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Copying 7 zone configuration files for zone 'otn-icinga-sat-hedvig' to '/var/lib/icinga2/api/zones/otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/archived_hedvig_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/archived_hedvig_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_custom_commands.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_hosts_dcc2.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/otn-icinga-sat-hedvig//_etc/hedvig_templates.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Copying 5 zone configuration files for zone 'ac-citrix-zone' to '/var/lib/icinga2/api/zones/ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/DEL_citrix_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/archive_citrix_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/citrix_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/citrix_hv.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/ac-citrix-zone//_etc/citrix_templates.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Copying 7 zone configuration files for zone 'k8s-icinga' to '/var/lib/icinga2/api/zones/k8s-icinga'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/archive_k8s_custom_checks.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/archive_k8s_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/archive_k8s_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_custom_checks.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_hosts.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_services.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/k8s-icinga//_etc/k8s_templates.conf
[2021-10-14 17:00:09 +0200] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-10-14 17:00:09 +0200] information/ApiListener: Reconnecting to endpoint 'otn-ac-ics-hfip.infra.ac.domain' via host '[ "otn-ac-ics-hfip.infra.ac.domain" ]' and port '5665'
[2021-10-14 17:00:09 +0200] information/ApiListener: Reconnecting to endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' via host 'otn-icinga-sat-hedvig.mon.infra.ac.domain' and port '5665'
[2021-10-14 17:00:09 +0200] information/ApiListener: Reconnecting to endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' via host 'otn-icinga-sat-citrix.mon.infra.ac.domain' and port '5665'
[2021-10-14 17:00:09 +0200] information/ApiListener: Reconnecting to endpoint 'k8s-icinga.ac.domain' via host 'k8s-icinga.ac.domain' and port '5665'
[2021-10-14 17:00:09 +0200] information/ExternalCommandListener: 'command' started.
[2021-10-14 17:00:09 +0200] warning/ExternalCommandListener: This feature is DEPRECATED and may be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'

[2021-10-14 17:00:09 +0200] critical/ApiListener: Cannot connect to host '[ "otn-ac-ics-hfip.infra.ac.domain" ]' on port '5665': Host not found (authoritative)
[2021-10-14 17:00:09 +0200] information/DbConnection: 'ido-mysql' started.
[2021-10-14 17:00:09 +0200] information/InfluxdbWriter: 'influxdb' resumed.
[2021-10-14 17:00:09 +0200] information/NotificationComponent: 'notification' started.
[2021-10-14 17:00:09 +0200] information/CheckerComponent: 'checker' started.
[2021-10-14 17:00:09 +0200] information/ConfigItem: Activated all objects.
[2021-10-14 17:00:09 +0200] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2021-10-14 17:00:09 +0200] information/DbConnection: Resuming IDO connection: ido-mysql
[2021-10-14 17:00:09 +0200] information/ApiListener: New client connection for identity 'otn-icinga-sat-citrix.mon.infra.ac.domain' to [10.116.219.65]:5665
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished reconnecting to endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' via host 'otn-icinga-sat-citrix.mon.infra.ac.domain' and port '5665'
[2021-10-14 17:00:09 +0200] information/ApiListener: Sending config updates for endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' in zone 'ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Syncing configuration files for zone 'ac-citrix-zone' to endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished sending config file updates for endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' in zone 'ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Syncing runtime objects to endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' in zone 'ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Sending replay log for endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' in zone 'ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished sending replay log for endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' in zone 'ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished syncing endpoint 'otn-icinga-sat-citrix.mon.infra.ac.domain' in zone 'ac-citrix-zone'.
[2021-10-14 17:00:09 +0200] information/JsonRpcConnection: Received certificate request for CN 'otn-icinga-sat-citrix.mon.infra.ac.domain' signed by our CA.
[2021-10-14 17:00:09 +0200] information/JsonRpcConnection: The certificate for CN 'otn-icinga-sat-citrix.mon.infra.ac.domain' is valid and uptodate. Skipping automated renewal.
[2021-10-14 17:00:09 +0200] information/ApiListener: New client connection for identity 'otn-icinga-sat-hedvig.mon.infra.ac.domain' to [10.116.219.179]:5665
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished reconnecting to endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' via host 'otn-icinga-sat-hedvig.mon.infra.ac.domain' and port '5665'
[2021-10-14 17:00:09 +0200] information/ApiListener: Sending config updates for endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' in zone 'otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Syncing configuration files for zone 'otn-icinga-sat-hedvig' to endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished sending config file updates for endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' in zone 'otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Syncing runtime objects to endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' in zone 'otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Sending replay log for endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' in zone 'otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished sending replay log for endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' in zone 'otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/ApiListener: Finished syncing endpoint 'otn-icinga-sat-hedvig.mon.infra.ac.domain' in zone 'otn-icinga-sat-hedvig'.
[2021-10-14 17:00:09 +0200] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2021-10-14 17:00:09 +0200] information/JsonRpcConnection: Received certificate request for CN 'otn-icinga-sat-hedvig.mon.infra.ac.domain' signed by our CA.
[2021-10-14 17:00:09 +0200] information/JsonRpcConnection: The certificate for CN 'otn-icinga-sat-hedvig.mon.infra.ac.domain' is valid and uptodate. Skipping automated renewal.
[2021-10-14 17:00:11 +0200] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga_ido' in 1.6876 second(s).
[2021-10-14 17:00:13 +0200] information/ApiListener: New client connection for identity 'k8s-icinga.ac.domain' from [10.116.219.121]:37158
[2021-10-14 17:00:13 +0200] information/ApiListener: Sending config updates for endpoint 'k8s-icinga.ac.domain' in zone 'k8s-icinga'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'k8s-icinga.ac.domain'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Syncing configuration files for zone 'k8s-icinga' to endpoint 'k8s-icinga.ac.domain'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Finished sending config file updates for endpoint 'k8s-icinga.ac.domain' in zone 'k8s-icinga'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Syncing runtime objects to endpoint 'k8s-icinga.ac.domain'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'k8s-icinga.ac.domain'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'k8s-icinga.ac.domain' in zone 'k8s-icinga'.
[2021-10-14 17:00:13 +0200] information/ApiListener: Sending replay log for endpoint 'k8s-icinga.ac.domain' in zone 'k8s-icinga'.

leeclemens · October 14, 2021, 4:54pm

The default unit file has:

TimeoutStartSec=30m

Does it ever start up and function properly? Maybe disable influxdb since that’s in the stacktrace?

lobr · October 15, 2021, 1:21pm

Hi,

Yes, it does start up in almost all cases when it restarts. I just experienced two times were it couldn’t.
Disabling influxdb is not possible, because it is critical for production.

lobr · April 20, 2022, 3:22pm

Hello,

i occurs more often that the icinga2 service stops but does not recover anymore…

[root@otn-ac-monp-ma01 icinga2]# tail -f /var/log/icinga2/icinga2.log 
[2022-04-20 17:16:22 +0200] information/Application: Received request to shut down.
[2022-04-20 17:16:22 +0200] information/Application: Shutting down...
[2022-04-20 17:16:22 +0200] information/CheckerComponent: 'checker' stopped.
[2022-04-20 17:16:22 +0200] information/NotificationComponent: 'notification' stopped.
[2022-04-20 17:16:22 +0200] information/DbConnection: Pausing IDO connection: ido-pgsql
[2022-04-20 17:16:22 +0200] information/IdoPgsqlConnection: Disconnected from 'ido-pgsql' database 'icinga'.
[2022-04-20 17:16:22 +0200] information/IdoPgsqlConnection: 'ido-pgsql' paused.
[2022-04-20 17:16:22 +0200] information/DbConnection: 'ido-pgsql' stopped.
[2022-04-20 17:16:22 +0200] information/InfluxdbWriter: 'influxdb' paused.
[2022-04-20 17:16:22 +0200] information/ApiListener: 'api' stopped.

/var/log/messages

Apr 20 17:16:34 otn-ac-monp-ma01 systemd: start request repeated too quickly for icinga2.service
Apr 20 17:16:34 otn-ac-monp-ma01 systemd: Failed to start Icinga host/service/network monitoring system.
Apr 20 17:16:34 otn-ac-monp-ma01 systemd: icinga2.service failed.
Apr 20 17:16:35 otn-ac-monp-ma01 ansible-stat: Invoked with checksum_algorithm=sha1 get_checksum=True follow=False path=/etc/icinga2/zones.d/otn-ac-monp-sa01.local/aircloud/vSphere67/otn-div-ac/vm/Openshift/Openshift.json get_md5=False get_mime=True get_attributes=True
Apr 20 17:16:35 otn-ac-monp-ma01 ansible-file: Invoked with dest=/etc/icinga2/zones.d/otn-ac-monp-sa01.local/aircloud/vSphere67/otn-div-ac/vm/Openshift/Openshift.json recurse=False _original_basename=tmpcKb7xg state=file path=/etc/icinga2/zones.d/otn-ac-monp-sa01.local/aircloud/vSphere67/otn-div-ac/vm/Openshift/Openshift.json force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None content=NOT_LOGGING_PARAMETER backup=None remote_src=None regexp=None delimiter=None directory_mode=None
Apr 20 17:16:35 otn-ac-monp-ma01 ansible-command: Invoked with _raw_params=python3 /root/ansible/infrastructure-monitoring-dev-ansible/sync/sync.py tm2 /etc/icinga2/zones.d/otn-ac-monp-sa01.local/aircloud/vSphere67/otn-div-ac/vm/Openshift/Openshift.json /etc/icinga2/zones.d/otn-ac-monp-sa01.local/aircloud/vSphere67/otn-div-ac/vm/Openshift/Openshift.conf warn=True _uses_shell=False stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Apr 20 17:16:36 otn-ac-monp-ma01 ansible-command: Invoked with _uses_shell=True _raw_params=systemctl restart icinga2.service warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None
Apr 20 17:16:36 otn-ac-monp-ma01 systemd: start request repeated too quickly for icinga2.service
Apr 20 17:16:36 otn-ac-monp-ma01 systemd: Failed to start Icinga host/service/network monitoring system.
Apr 20 17:16:36 otn-ac-monp-ma01 systemd: icinga2.service failed.

What might this be related to?