How to clean cache info from icinga2 and icinga2 satellite servers

Icinga 2
1

Before you ask a question, you can check the troubleshooting documentation first, maybe you can find an answer here.

Please describe your problem as detailed as possible and don’t forget to use a meaningful title :slight_smile:
We also have a markdown formatting guide to help you make your topics more readable!

Give as much information as you can, e.g.

  • Version used (icinga2 --version) master: version: r2.13.5-1, satellite: version: 2.13.2-1
  • Operating System and version master: opensuse 15.4 with latest patches satellite: AlmaLinux release 8.7 with latest patches
  • Enabled features (icinga2 feature list) master: api checker debuglog icingadb ido-mysql mainlog notification satellite: api checker debuglog mainlog
  • Icinga Web 2 version and modules (System - About) Icinga Web 2 Version 2.11.1
    Refer to attachment
  • Config validation (icinga2 daemon -C) → Refer to attachments
  • If you run multiple Icinga 2 instances, the zones.conf file (or icinga2 object list --type Endpoint and icinga2 object list --type Zone) from all affected nodes

icinga2:~ # 次世代 → icinga2 object list --type Endpoint
Object ‘icinga2.test.net’ of type ‘Endpoint’:
% declared in ‘/etc/icinga2/zones.conf’, lines 59:1-59:34

  • __name = “icinga2.test.net”
  • host = “”
  • log_duration = 86400
  • name = “icinga2.test.net”
  • package = “_etc”
  • port = “5665”
  • source_location
    • first_column = 1
    • first_line = 59
    • last_column = 34
    • last_line = 59
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “icinga2.test.net” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 59:1-59:34
  • type = “Endpoint”
  • zone = “”

Object ‘icinga2-sat1.test.net’ of type ‘Endpoint’:
% declared in ‘/etc/icinga2/zones.conf’, lines 51:1-51:39

  • __name = “icinga2-sat1.test.net”
  • host = “192.168.8.111”
    % = modified in ‘/etc/icinga2/zones.conf’, lines 52:3-52:24
  • log_duration = 86400
  • name = “icinga2-sat1.test.net”
  • package = “_etc”
  • port = “5665”
  • source_location
    • first_column = 1
    • first_line = 51
    • last_column = 39
    • last_line = 51
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “icinga2-sat1.test.net” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 51:1-51:39
  • type = “Endpoint”
  • zone = “”

Object ‘icinga2-agent2’ of type ‘Endpoint’:
% declared in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 2:1-2:32

  • __name = “icinga2-agent2”
  • host = “192.168.31.113”
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 3:1-3:23
  • log_duration = 0
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 4:1-4:16
  • name = “icinga2-agent2”
  • package = “_etc”
  • port = “5665”
  • source_location
    • first_column = 1
    • first_line = 2
    • last_column = 32
    • last_line = 2
    • path = “/etc/icinga2/zones.d/cn/icinga2-agent2.conf”
  • templates = [ “icinga2-agent2” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 2:1-2:32
  • type = “Endpoint”
  • zone = “cn”

Object ‘icinga2-agent3’ of type ‘Endpoint’:
% declared in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 2:1-2:32

  • __name = “icinga2-agent3”
  • host = “192.168.8.190”
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 3:1-3:22
  • log_duration = 0
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 4:1-4:16
  • name = “icinga2-agent3”
  • package = “_etc”
  • port = “5665”
  • source_location
    • first_column = 1
    • first_line = 2
    • last_column = 32
    • last_line = 2
    • path = “/etc/icinga2/zones.d/cn/icinga2-agent3.conf”
  • templates = [ “icinga2-agent3” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 2:1-2:32
  • type = “Endpoint”
  • zone = “cn”

Object ‘icinga2-agent1’ of type ‘Endpoint’:
% declared in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 2:1-2:32

  • __name = “icinga2-agent1”
  • host = “192.168.31.112”
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 3:1-3:23
  • log_duration = 0
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 4:1-4:16
  • name = “icinga2-agent1”
  • package = “_etc”
  • port = “5665”
  • source_location
    • first_column = 1
    • first_line = 2
    • last_column = 32
    • last_line = 2
    • path = “/etc/icinga2/zones.d/cn/icinga2-agent1.conf”
  • templates = [ “icinga2-agent1” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 2:1-2:32
  • type = “Endpoint”
  • zone = “cn”
    ======================

icinga2:~ # 次世代 → icinga2 object list --type Zone
Object ‘icinga2-agent1’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 7:1-7:28

  • __name = “icinga2-agent1”
  • endpoints = [ “icinga2-agent1” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 8:1-8:32
  • global = false
  • name = “icinga2-agent1”
  • package = “_etc”
  • parent = “cn”
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 9:1-9:13
  • source_location
    • first_column = 1
    • first_line = 7
    • last_column = 28
    • last_line = 7
    • path = “/etc/icinga2/zones.d/cn/icinga2-agent1.conf”
  • templates = [ “icinga2-agent1” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent1.conf’, lines 7:1-7:28
  • type = “Zone”
  • zone = “cn”

Object ‘global-templates’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.conf’, lines 26:1-26:30

  • __name = “global-templates”
  • endpoints = null
  • global = true
    % = modified in ‘/etc/icinga2/zones.conf’, lines 27:3-27:15
  • name = “global-templates”
  • package = “_etc”
  • parent = “”
  • source_location
    • first_column = 1
    • first_line = 26
    • last_column = 30
    • last_line = 26
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “global-templates” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 26:1-26:30
  • type = “Zone”
  • zone = “”

Object ‘icinga2-agent2’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 7:1-7:28

  • __name = “icinga2-agent2”
  • endpoints = [ “icinga2-agent2” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 8:1-8:32
  • global = false
  • name = “icinga2-agent2”
  • package = “_etc”
  • parent = “cn”
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 9:1-9:13
  • source_location
    • first_column = 1
    • first_line = 7
    • last_column = 28
    • last_line = 7
    • path = “/etc/icinga2/zones.d/cn/icinga2-agent2.conf”
  • templates = [ “icinga2-agent2” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent2.conf’, lines 7:1-7:28
  • type = “Zone”
  • zone = “cn”

Object ‘global-commands’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.conf’, lines 42:1-42:29

  • __name = “global-commands”
  • endpoints = null
  • global = true
    % = modified in ‘/etc/icinga2/zones.conf’, lines 43:3-43:15
  • name = “global-commands”
  • package = “_etc”
  • parent = “”
  • source_location
    • first_column = 1
    • first_line = 42
    • last_column = 29
    • last_line = 42
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “global-commands” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 42:1-42:29
  • type = “Zone”
  • zone = “”

Object ‘cn’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.conf’, lines 62:1-62:16

  • __name = “cn”
  • endpoints = [ “icinga2-sat1.test.net” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 63:2-63:40
  • global = false
  • name = “cn”
  • package = “_etc”
  • parent = “master”
    % = modified in ‘/etc/icinga2/zones.conf’, lines 65:2-65:18
  • source_location
    • first_column = 1
    • first_line = 62
    • last_column = 16
    • last_line = 62
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “cn” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 62:1-62:16
  • type = “Zone”
  • zone = “”

Object ‘master’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.conf’, lines 55:1-55:20

  • __name = “master”
  • endpoints = [ “icinga2.test.net” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 56:3-56:36
  • global = false
  • name = “master”
  • package = “_etc”
  • parent = “”
  • source_location
    • first_column = 1
    • first_line = 55
    • last_column = 20
    • last_line = 55
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “master” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 55:1-55:20
  • type = “Zone”
  • zone = “”

Object ‘director-global’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.conf’, lines 38:1-38:29

  • __name = “director-global”
  • endpoints = null
  • global = true
    % = modified in ‘/etc/icinga2/zones.conf’, lines 39:3-39:15
  • name = “director-global”
  • package = “_etc”
  • parent = “”
  • source_location
    • first_column = 1
    • first_line = 38
    • last_column = 29
    • last_line = 38
    • path = “/etc/icinga2/zones.conf”
  • templates = [ “director-global” ]
    % = modified in ‘/etc/icinga2/zones.conf’, lines 38:1-38:29
  • type = “Zone”
  • zone = “”

Object ‘icinga2-agent3’ of type ‘Zone’:
% declared in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 7:1-7:28

  • __name = “icinga2-agent3”
  • endpoints = [ “icinga2-agent3” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 8:1-8:32
  • global = false
  • name = “icinga2-agent3”
  • package = “_etc”
  • parent = “cn”
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 9:1-9:13
  • source_location
    • first_column = 1
    • first_line = 7
    • last_column = 28
    • last_line = 7
    • path = “/etc/icinga2/zones.d/cn/icinga2-agent3.conf”
  • templates = [ “icinga2-agent3” ]
    % = modified in ‘/etc/icinga2/zones.d/cn/icinga2-agent3.conf’, lines 7:1-7:28
  • type = “Zone”
  • zone = “cn”

Here our monitored agent: icinga2-agent3 with IP: 192.168.8.190. But from its ping4 result, always shows with ip: 192.168.8.114. I grep master and satellite servers /var/lib/icinga2 and /etc/icinga2, with no IP of 192.168.8.114. But I still can filter out many records of IP: 192.168.8.114 from icinga2 debug.log. From records, it link to “icinga2-agent3.test.net” endpoint. But it are not configured in our current inciga2 master and sat. Maybe old configuration. So how to clean the cache info from icinga2 system?

[2023-04-21 08:32:36 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:32:39 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:32:46 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:32:49 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:32:53 +0800] notice/Process: Running command ‘/usr/lib//nagios/plugins/check_ping’ ‘-H’ ‘192.168.8.114’ ‘-c’ ‘300,100%’ ‘-w’ ‘250,80%’: PID 4971

[2023-04-21 08:32:56 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:32:56 +0800] notice/Process: PID 4971 (‘/usr/lib//nagios/plugins/check_ping’ ‘-H’ ‘192.168.8.114’ ‘-c’ ‘300,100%’ ‘-w’ ‘250,80%’) terminated with exit code 2

[2023-04-21 08:32:56 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:32:56 +0800] debug/IdoMysqlConnection: Query: UPDATE icinga_hoststatus SET acknowledgement_type = ‘0’, active_checks_enabled = ‘1’, check_command = ‘hostalive’, check_source = ‘icinga2.test.net’, check_timeperiod_object_id = NULL, check_type = ‘0’, current_check_attempt = ‘1’, current_notification_number = ‘0’, current_state = ‘1’, endpoint_object_id = 247, event_handler_enabled = ‘1’, execution_time = ‘3.087183’, flap_detection_enabled = ‘0’, has_been_checked = ‘1’, host_object_id = 617, instance_id = 1, is_flapping = ‘0’, is_reachable = ‘1’, last_check = FROM_UNIXTIME(1682037176), last_hard_state = ‘1’, last_hard_state_change = FROM_UNIXTIME(1681804164), last_notification = NULL, last_state_change = NULL, last_time_down = FROM_UNIXTIME(1682037176), last_time_unreachable = NULL, last_time_up = NULL, latency = ‘0.000631’, long_output = ‘’, max_check_attempts = ‘3’, next_check = FROM_UNIXTIME(1682037464), next_notification = NULL, normal_check_interval = ‘5’, notifications_enabled = ‘1’, original_attributes = ‘null’, output = ‘CRITICAL - Host Unreachable (192.168.8.114)’, passive_checks_enabled = ‘1’, percent_state_change = ‘0’, perfdata = ‘’, problem_has_been_acknowledged = ‘0’, process_performance_data = ‘1’, retry_check_interval = ‘1’, scheduled_downtime_depth = ‘0’, should_be_scheduled = ‘1’, state_type = ‘1’, status_update_time = FROM_UNIXTIME(1682037176) WHERE host_object_id = 617

[2023-04-21 08:33:06 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:33:09 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:33:16 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:33:19 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:33:26 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:33:27 +0800] notice/Process: Running command ‘/usr/lib//nagios/plugins/check_ping’ ‘-4’ ‘-H’ ‘192.168.8.114’ ‘-c’ ‘300,15%’ ‘-w’ ‘250,5%’: PID 5040

[2023-04-21 08:33:29 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:33:29 +0800] notice/Process: PID 5040 (‘/usr/lib//nagios/plugins/check_ping’ ‘-4’ ‘-H’ ‘192.168.8.114’ ‘-c’ ‘300,15%’ ‘-w’ ‘250,5%’) terminated with exit code 2

[2023-04-21 08:33:29 +0800] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = ‘0’, active_checks_enabled = ‘1’, check_command = ‘ping4’, check_source = ‘icinga2.test.net’, check_timeperiod_object_id = NULL, check_type = ‘0’, current_check_attempt = ‘1’, current_notification_number = ‘0’, current_state = ‘2’, endpoint_object_id = 247, event_handler_enabled = ‘1’, execution_time = ‘2.199668’, flap_detection_enabled = ‘0’, has_been_checked = ‘1’, instance_id = 1, is_flapping = ‘0’, is_reachable = ‘0’, last_check = FROM_UNIXTIME(1682037209), last_hard_state = ‘2’, last_hard_state_change = FROM_UNIXTIME(1681804470), last_notification = NULL, last_state_change = FROM_UNIXTIME(1681804349), last_time_critical = FROM_UNIXTIME(1682037209), last_time_ok = FROM_UNIXTIME(1681804286), last_time_unknown = NULL, last_time_warning = NULL, latency = ‘0.000595’, long_output = ‘’, max_check_attempts = ‘5’, next_check = FROM_UNIXTIME(1682037267), next_notification = NULL, normal_check_interval = ‘1’, notifications_enabled = ‘1’, original_attributes = ‘null’, output = ‘CRITICAL - Host Unreachable (192.168.8.114)’, passive_checks_enabled = ‘1’, percent_state_change = ‘0’, perfdata = ‘’, problem_has_been_acknowledged = ‘0’, process_performance_data = ‘1’, retry_check_interval = ‘0.500000’, scheduled_downtime_depth = ‘0’, service_object_id = 619, should_be_scheduled = ‘1’, state_type = ‘1’, status_update_time = FROM_UNIXTIME(1682037209) WHERE service_object_id = 619

[2023-04-21 08:33:36 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:33:39 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:33:46 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:33:49 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:33:56 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:33:59 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

[2023-04-21 08:34:06 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’

[2023-04-21 08:34:09 +0800] critical/ApiListener: Cannot connect to host ‘192.168.8.114’ on port ‘5665’: No route to host

截屏2023-04-21 09.16.43
截屏2023-04-21 09.16.432368×1386 189 KB

截屏2023-04-21 09.16.06
截屏2023-04-21 09.16.062360×766 202 KB

截屏2023-04-21 09.15.46
截屏2023-04-21 09.15.461920×864 309 KB

2

From master’s debug.log " [2023-04-21 08:28:36 +0800] information/ApiListener: Reconnecting to endpoint ‘icinga2-agent3.test.net’ via host ‘192.168.8.114’ and port ‘5665’". But from “icinga2 object list --type Endpoint”, I didn’t see the record. So how to fix the obsolete endpoint: icinga2-agent3.test.net and remove it.