Today, I have noticed that some checks show were outdated/overdue but I didn’t really see why. Eventually, I have rebooted first the second server then the first icinga2 server and now the second server crashes with a segfault 1-2 minutes after starting.
# fgrep segfault /var/log/messages
Jan 25 08:54:00 icinga2-ha2 kernel: icinga2[3192]: segfault at 48 ip 00000000009e4cee sp 00007f530a3ed5e0 error 4 in icinga2[400000+df9000]
Jan 25 08:56:47 icinga2-ha2 kernel: icinga2[8850]: segfault at 48 ip 00000000009e4cee sp 00007f433c48d5e0 error 4 in icinga2[400000+df9000]
Jan 25 09:00:01 icinga2-ha2 kernel: icinga2[11760]: segfault at 48 ip 00000000009e4cee sp 00007f55490fd5e0 error 4 in icinga2[400000+df9000]
Jan 25 09:03:36 icinga2-ha2 kernel: icinga2[16139]: segfault at 48 ip 00000000009e4cee sp 00007f31816ae5e0 error 4 in icinga2[400000+df9000]
Jan 25 09:04:26 icinga2-ha2 kernel: icinga2[16989]: segfault at 48 ip 00000000009e4cee sp 00007f1a854175e0 error 4 in icinga2[400000+df9000]
Jan 25 09:13:33 icinga2-ha2 kernel: icinga2[28574]: segfault at 48 ip 00000000009e4cee sp 00007fc1e010d5e0 error 4 in icinga2[400000+df9000]
We didn’t really do anything nor changed any configuration for a couple of days now.
- Version used 2.13.2-1
icinga2-2.13.2-1.el7.icinga.x86_64
icinga2-bin-2.13.2-1.el7.icinga.x86_64
icinga2-common-2.13.2-1.el7.icinga.x86_64
icinga2-ido-mysql-2.13.2-1.el7.icinga.x86_64
icinga2-selinux-2.13.2-1.el7.icinga.x86_64
- Operating System and version: CentOS Linux release 7.9.2009 (Core)
- Enabled features: api checker command ido-mysql mainlog notification
- Icinga Web 2 version and modules 2.9.5
- Config validation (
icinga2 daemon -C)
# icinga2 daemon -C
[2022-01-25 09:27:03 +0100] information/cli: Icinga application loader (version: 2.13.2-1)
[2022-01-25 09:27:03 +0100] information/cli: Loading configuration file(s).
[2022-01-25 09:27:03 +0100] information/ConfigItem: Committing config item(s).
[2022-01-25 09:27:03 +0100] information/ApiListener: My API identity: icinga2-ha2.example.com
[2022-01-25 09:27:05 +0100] warning/ApplyRule: Apply rule 'swift-cluster-' (in /var/lib/icinga2/api/zones/master/_etc/services.conf: 1189:1-1189:103) for type 'Dependency' does not match anywhere!
[2022-01-25 09:27:05 +0100] warning/ApplyRule: Apply rule 'sms-host-rufbereitschaft' (in /var/lib/icinga2/api/zones/master/_etc/notifications-rufbereitschaft.conf: 49:1-49:53) for type 'Notification' does not match anywhere!
[2022-01-25 09:27:05 +0100] warning/ApplyRule: Apply rule 'sms-host' (in /var/lib/icinga2/api/zones/master/_etc/notifications.conf: 133:1-133:37) for type 'Notification' does not match anywhere!
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 54 Users.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 29 UserGroups.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 17 ServiceGroups.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 13 TimePeriods.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 4391 Services.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 8 Zones.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 4787 ScheduledDowntimes.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 4 NotificationCommands.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 69 HostGroups.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 4880 Notifications.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 8 Downtimes.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 2898 Dependencies.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 396 Hosts.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 7 Endpoints.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 12 Comments.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 7 ApiUsers.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 338 CheckCommands.
[2022-01-25 09:27:05 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2022-01-25 09:27:05 +0100] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2022-01-25 09:27:05 +0100] information/cli: Finished validating the configuration file(s).
- If you run multiple Icinga 2 instances, the
zones.conffile (oricinga2 object list --type Endpointandicinga2 object list --type Zone) from all affected nodes
zones.conf from both instances is identical:
/*
* Generated by Icinga 2 node setup commands
* on 2018-01-11 09:00:01 +0100
*/
object Endpoint "icinga2-ha1.example.com" {
host = "icinga2-ha1.example.com"
}
object Endpoint "icinga2-ha2.example.com" {
host = "icinga2-ha2.example.com"
}
object Zone "master" {
endpoints = [ "icinga2-ha1.example.com", "icinga2-ha2.example.com" ]
}
object Endpoint "status.example.com" {
host = "status.example.com"
}
object Endpoint "status-test.example.com" {
host = "status-test.example.com"
}
object Zone "status" {
endpoints = [ "status.example.com" ]
parent = "master"
}
object Zone "status-test" {
endpoints = [ "status-test.example.com" ]
parent = "master"
}
object Zone "global-templates" {
global = true
}
object Zone "director-global" {
global = true
}