Hi,
My icinga master has crashed twice on segmentation fault, as described on the dmesg:
[Mon Apr 27 09:31:50 2020] icinga2[6969]: segfault at 68 ip 00007f4a02f2bcb8 sp 00007f495c2d4a30 error 4 in libstdc++.so.6.0.19[7f4a02e6d000+e9000]
[Thu Apr 30 06:51:15 2020] icinga2[6618]: segfault at 68 ip 00007f2b97020cb8 sp 00007f2b13ecb8e0 error 4 in libstdc++.so.6.0.19[7f2b96f62000+e9000]
Can you help provide troubleshooting steps, advice on how to find out what has happened, or if this is a known issue?
I have checked for similar issues without luck,
Icinga2 Logs prior to crash
[2020-04-30 06:51:19 -0400] information/WorkQueue: #8 (IdoMysqlConnection, ido-mysql) items: 275, rate: 22230.2/s (1333814/min 2534266/5min 2534266/15min);
[2020-04-30 06:51:19 -0400] warning/InfluxdbWriter: Ignoring invalid perfdata for checkable ‘database-01.internal!mssql-dead-locks’ and command ‘mssql-dead-locks’ with value: dead_locks=0/s;0:2;0:10;0;
[2020-04-30 06:51:19 -0400] warning/InfluxdbWriter: Ignoring invalid perfdata fo
r checkable ‘database-01.internal!mssql-dead-locks’ and command ‘ms
sql-dead-locks’ with value: dead_locks=0/s;0:2;0:10;0;
Context:
(0) Processing check result for ‘database-01.internal!mssql
-dead-locks’[2020-04-30 06:51:19 -0400] warning/InfluxdbWriter: Ignoring invalid perfdata fo
r checkable ‘database-02.internal!mssql-transactions’ and command ’
mssql-transactions’ with value: transactions=20.83/s;;;0;
Context:
(0) Processing check result for ‘database-02.internal!mssql
-transactions’
Infrastructure details:
pedro@icinga-master.internal ~$ uname -a
Linux monitoring-icinga-1.lat.corp-apps.com 3.10.0-1062.18.1.el7.x86_64 #1 SMP Tue Mar 17 23:49:17 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
pedro@icinga-master.internal ~$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)
pedro@icinga-master.internal ~$ free -h
total used free shared buff/cache available
Mem: 15G 4.7G 7.2G 429M 3.6G 10G
Swap: 2.0G 3.5M 2.0G
pedro@icinga-master.internal ~$ uptime
09:58:07 up 8 days, 5:49, 1 user, load average: 3.47, 3.71, 3.73
pedro@icinga-master.internal ~$ cat /dev/cpu/
0/ 1/ 2/ 3/ 4/ 5/ 6/ 7/
rpm -qa | egrep "(icinga2|libcstd|glib|boost)" | sort
boost169-chrono-1.69.0-2.el7.x86_64
boost169-context-1.69.0-2.el7.x86_64
boost169-coroutine-1.69.0-2.el7.x86_64
boost169-date-time-1.69.0-2.el7.x86_64
boost169-filesystem-1.69.0-2.el7.x86_64
boost169-program-options-1.69.0-2.el7.x86_64
boost169-regex-1.69.0-2.el7.x86_64
boost169-system-1.69.0-2.el7.x86_64
boost169-thread-1.69.0-2.el7.x86_64
boost-program-options-1.53.0-27.el7.x86_64
boost-regex-1.53.0-27.el7.x86_64
boost-system-1.53.0-27.el7.x86_64
boost-thread-1.53.0-27.el7.x86_64
dbus-glib-0.100-7.el7.x86_64
glib2-2.56.1-5.el7.x86_64
glibc-2.17-292.el7.i686
glibc-2.17-292.el7.x86_64
glibc-common-2.17-292.el7.x86_64
glibc-devel-2.17-292.el7.x86_64
glibc-headers-2.17-292.el7.x86_64
glib-networking-2.56.1-1.el7.x86_64
icinga2-2.11.3-1.el7.icinga.x86_64
icinga2-bin-2.11.3-1.el7.icinga.x86_64
icinga2-common-2.11.3-1.el7.icinga.x86_64
icinga2-ido-mysql-2.11.3-1.el7.icinga.x86_64
Validation Command (executed locally):
[2020-04-30 13:50:49 +0000] information/cli: Icinga application loader (version: 2.11.3-1)
[2020-04-30 13:50:49 +0000] information/cli: Loading configuration file(s).
[2020-04-30 13:50:50 +0000] information/ConfigItem: Committing config item(s).
[2020-04-30 13:51:00 +0000] information/WorkQueue: #4 (DaemonUtility::LoadConfigFiles) items: 4, rate: 6.06667/s (364/min 364/5min 364/15min); empty in 18382 days, 13 hours and 51 minutes
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'disable-checks-domain1' (in /etc/icinga2/zones.d/global-external/services.d/seo/dependency.conf: 1:0-1:56) for type 'Dependency' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'disable-checks-domain2' (in /etc/icinga2/zones.d/global-external/services.d/seo/dependency.conf: 8:1-8:58) for type 'Dependency' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'disable-checks-domain3' (in /etc/icinga2/zones.d/global-external/services.d/seo/dependency.conf: 15:1-15:62) for type 'Dependency' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'disable-checks-domain4' (in /etc/icinga2/zones.d/global-external/services.d/seo/dependency.conf: 22:1-22:66) for type 'Dependency' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'disable-checks-domain5' (in /etc/icinga2/zones.d/global-external/services.d/seo/dependency.conf: 29:1-29:57) for type 'Dependency' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global/services.d/ito/solutions/vedge/health_check.conf: 1:0-1:107) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'sensorcheck_' (in /etc/icinga2/zones.d/global-internal/services.d/roomalert.conf: 1:0-1:76) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-internal/services.d/services-windows.conf: 222:1-222:58) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-internal/services.d/services.conf: 12:1-12:64) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'pacheck' (in /etc/icinga2/zones.d/global-internal/services.d/services.conf: 20:1-20:23) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-internal/services.d/services.conf: 135:1-135:76) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'active_passive_' (in /etc/icinga2/zones.d/global-internal/services.d/cluster/service_state.conf: 6:1-6:97) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'check-snmp-humidity' (in /etc/icinga2/zones.d/global-internal/services.d/exchange.nagios.org/check-snmp-humidity.conf: 1:0-1:34) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-internal/services.d/ito/dba/mssql/services-mssql.conf: 143:1-143:76) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-bigip.conf: 57:1-57:62) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-bigip.conf: 81:1-81:80) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'CPU-cisco-asa' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-cisco.conf: 83:1-83:29) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'memory-cisco-asa' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-cisco.conf: 91:1-91:32) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'failover-cisco-asa' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-cisco.conf: 99:1-99:34) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'ipsec-tunnel-cisco-asa' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-cisco.conf: 107:1-107:38) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'sessions-cisco-asa' (in /etc/icinga2/zones.d/global-internal/services.d/ito/network/services-cisco.conf: 115:1-115:34) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] warning/ApplyRule: Apply rule 'cassandra-core-service' (in /etc/icinga2/zones.d/global-internal/services.d/ito/support/cassandra-core.conf: 1:0-1:43) for type 'Service' does not match anywhere!
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 62 ScheduledDowntimes.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 285 HostGroups.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 EventCommand.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 3 NotificationCommands.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 106573 Notifications.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 6616 Hosts.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 116312 Dependencies.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 46 Zones.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 65 Endpoints.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 ApiUser.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 415 CheckCommands.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 3 TimePeriods.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 1 UserGroup.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 3 Users.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 99395 Services.
[2020-04-30 13:51:57 +0000] information/ConfigItem: Instantiated 52 ServiceGroups.
[2020-04-30 13:51:57 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2020-04-30 13:51:57 +0000] information/cli: Finished validating the configuration file(s).
Thanks in advance,