Hello,
This issue happened twice on the config master and it wasn’t an active endpoint at the time so far since the deployment of this new clusters. No Icinga2 agents.
Give as much information as you can, e.g.
/var/log# lscpu | grep On-line
On-line CPU(s) list: 0-7
# free -k
total used free shared buff/cache available
Mem: 16375740 601864 15768096 11920 309112 15773876
Swap: 4194300 91932 4102368
- Version used (
icinga2 --version)
icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.15.0-1)
Copyright (c) 2012-2025 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
System information:
Platform: Ubuntu
Platform version: 24.04.3 LTS (Noble Numbat)
Kernel: Linux
Kernel version: 6.8.0-78-generic
Architecture: x86_64
Build information:
Compiler: GNU 13.3.0
Build host: runner-hh8q3bz2-project-575-concurrent-0
OpenSSL version: OpenSSL 3.0.13 30 Jan 2024
Application information:
General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2
Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var
Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
- Enabled features (
icinga2 feature list)
# icinga2 feature list
Disabled features: checker command compatlog debuglog elasticsearch gelf graphite icingadb influxdb influxdb2 journald livestatus notification opentsdb perfdata syslog
Enabled features: api ido-mysql mainlog
- Icinga Web 2 version and modules (System - About)
2.12.5
- Config validation (
icinga2 daemon -C)
# icinga2 daemon -C
[2025-09-05 13:16:18 -0700] information/cli: Icinga application loader (version: r2.15.0-1)
[2025-09-05 13:16:18 -0700] information/cli: Loading configuration file(s).
[2025-09-05 13:16:18 -0700] information/ConfigItem: Committing config item(s).
[2025-09-05 13:16:18 -0700] information/ApiListener: My API identity: x.y.com
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 126526 Notifications.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 7 NotificationCommands.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 IcingaApplication.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 57 HostGroups.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 8017 Hosts.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 3 Downtimes.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 FileLogger.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 9 Zones.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 14 Endpoints.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 3 ApiUsers.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 1 ApiListener.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 383 CheckCommands.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 133 Users.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 110 UserGroups.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 108 ServiceGroups.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 14 TimePeriods.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 48190 Services.
[2025-09-05 13:16:28 -0700] information/ConfigItem: Instantiated 4 ScheduledDowntimes.
[2025-09-05 13:16:28 -0700] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2025-09-05 13:16:28 -0700] information/cli: Finished validating the configuration file(s).
From kern.log
2025-09-05T12:12:53.613161-07:00 x.y.com kernel: icinga2 invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
2025-09-05T12:12:53.628775-07:00 x.y.com kernel: CPU: 3 PID: 742676 Comm: icinga2 Not tainted 6.8.0-78-generic #78-Ubuntu
2025-09-05T12:12:53.628777-07:00 x.y.com kernel: Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9 04/01/2014
2025-09-05T12:12:53.628778-07:00 x.y.com kernel: Call Trace:
2025-09-05T12:12:53.634179-07:00 x.y.com kernel: <TASK>
2025-09-05T12:12:53.635783-07:00 x.y.com kernel: dump_stack_lvl+0x76/0xa0
2025-09-05T12:12:53.635784-07:00 x.y.com kernel: dump_stack+0x10/0x20
2025-09-05T12:12:53.640106-07:00 x.y.com kernel: dump_header+0x49/0x210
2025-09-05T12:12:53.640107-07:00 x.y.com kernel: oom_kill_process+0x118/0x280
2025-09-05T12:12:53.640108-07:00 x.y.com kernel: ? oom_evaluate_task+0x143/0x1e0
2025-09-05T12:12:53.640108-07:00 x.y.com kernel: out_of_memory+0x103/0x350
2025-09-05T12:12:53.640109-07:00 x.y.com kernel: __alloc_pages_may_oom+0x10c/0x1d0
2025-09-05T12:12:53.640110-07:00 x.y.com kernel: __alloc_pages_slowpath.constprop.0+0x420/0x9f0
2025-09-05T12:12:53.640110-07:00 x.y.com kernel: __alloc_pages+0x31f/0x350
2025-09-05T12:12:53.640111-07:00 x.y.com kernel: alloc_pages_mpol+0x91/0x210
2025-09-05T12:12:53.650367-07:00 x.y.com kernel: alloc_pages+0x5b/0xd0
2025-09-05T12:12:53.650369-07:00 x.y.com kernel: folio_alloc+0x15/0x40
2025-09-05T12:12:53.650370-07:00 x.y.com kernel: filemap_alloc_folio+0xf4/0x100
2025-09-05T12:12:53.650370-07:00 x.y.com kernel: __filemap_get_folio+0x195/0x2d0
2025-09-05T12:12:53.650371-07:00 x.y.com kernel: filemap_fault+0x15c/0x8e0
2025-09-05T12:12:53.650371-07:00 x.y.com kernel: __do_fault+0x3a/0x190
2025-09-05T12:12:53.650372-07:00 x.y.com kernel: do_read_fault+0x133/0x200
2025-09-05T12:12:53.650372-07:00 x.y.com kernel: do_fault+0xf0/0x260
2025-09-05T12:12:53.660507-07:00 x.y.com kernel: handle_pte_fault+0x114/0x1d0
2025-09-05T12:12:53.660509-07:00 x.y.com kernel: __handle_mm_fault+0x654/0x800
2025-09-05T12:12:53.660510-07:00 x.y.com kernel: handle_mm_fault+0x18a/0x380
2025-09-05T12:12:53.660510-07:00 x.y.com kernel: do_user_addr_fault+0x169/0x670
2025-09-05T12:12:53.660511-07:00 x.y.com kernel: exc_page_fault+0x83/0x1b0
2025-09-05T12:12:53.660511-07:00 x.y.com kernel: asm_exc_page_fault+0x27/0x30
2025-09-05T12:12:53.660512-07:00 x.y.com kernel: RIP: 0033:0x7146fc223630
2025-09-05T12:12:53.660513-07:00 x.y.com kernel: Code: Unable to access opcode bytes at 0x7146fc223606.
2025-09-05T12:12:53.661857-07:00 x.y.com kernel: RSP: 002b:00007146fa533288 EFLAGS: 00010246
2025-09-05T12:12:53.661859-07:00 x.y.com kernel: RAX: 000071468c029190 RBX: 0000000000000000 RCX: 00007146fba05780
2025-09-05T12:12:53.661859-07:00 x.y.com kernel: RDX: 0000000000000056 RSI: 00007146fc3ecb41 RDI: 0000000000000000
2025-09-05T12:12:53.661860-07:00 x.y.com kernel: RBP: 00007146fa5332b0 R08: 0000000000000000 R09: 0000000000000000
2025-09-05T12:12:53.661861-07:00 x.y.com kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 000071468c029190
2025-09-05T12:12:53.661861-07:00 x.y.com kernel: R13: 00007146fc3ecb41 R14: 0000000000000000 R15: 00005657042427a0
2025-09-05T12:12:53.661862-07:00 x.y.com kernel: </TASK>
2025-09-05T12:12:53.661862-07:00 x.y.com kernel: Mem-Info:
2025-09-05T12:12:53.668219-07:00 x.y.com kernel: active_anon:1592644 inactive_anon:2403968 isolated_anon:0
2025-09-05T12:12:53.668221-07:00 x.y.com kernel: active_file:0 inactive_file:50 isolated_file:0
2025-09-05T12:12:53.668222-07:00 x.y.com kernel: unevictable:6821 dirty:0 writeback:0
2025-09-05T12:12:53.668222-07:00 x.y.com kernel: slab_reclaimable:6913 slab_unreclaimable:24893
2025-09-05T12:12:53.668223-07:00 x.y.com kernel: mapped:4082 shmem:2032 pagetables:11134
2025-09-05T12:12:53.668223-07:00 x.y.com kernel: sec_pagetables:0 bounce:0
2025-09-05T12:12:53.668224-07:00 x.y.com kernel: kernel_misc_reclaimable:0
2025-09-05T12:12:53.668224-07:00 x.y.com kernel: free:34160 free_pcp:0 free_cma:0
Using MySQL IDO (MariaDB 10.11.13) with HA enabled. The secondary master apperared working normal at the time. No files were generated in /var/log/icinga2/crash
Please advise
Thanks