Icinga2 Distributed monitoring checks by satellites are stuck in"pending"

lobr · March 23, 2021, 5:45pm

Hello,

I am starting my first thread here since I don’t have any clue anymore how I might be able to solve this issue.
I already read and tried the Distributed monitoring section of the documentation, without any success: Distributed Monitoring - Icinga 2

The situation: I am trying to run a master - satellite architecture to monitor hosts that are only reachable on the network by the specific satellite.Yet there is no redunancy implemented. The master shouldn’t do anything than pushing does check commands to the satellite. Only the satellite should execute the checks.

The issue: No matter what config I do try, the service checks by the satellite do always stuck on “PENDING”.

I would really appreciate help.

Here is my configuration:

########## Configuration of Master: ##########

/etc/icinga2/zones.conf

object Endpoint "icinga-qa.domain" {
}

object Zone "master" {
        endpoints = [ "icinga-qa.domain" ]

}

object Endpoint "icinga-qa-satellite-01.domain" {
        host = "10.116.220.140"
}

object Zone "icinga-qa-satellite-01.domain" {
        endpoints = [ "icinga-qa-satellite-01.domain" ]
        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

/etc/icinga2/zones.d/global-templates/services.conf

apply Service "test_ping" {
  import "generic-service"
  check_command = "ping4"
  assign where host.vars.type == "test"
  command_endpoint = host.vars.agent_endpoint
}

/etc/icinga2/zones.d/icinga-qa-satellite-01.domain

object Host "Test 192.168.1.22" { //this host is only available in the net of the satellite
import "generic-host"
address = "192.168.1.22"
vars.type = "test"
vars.agent_endpoint = "icinga-qa-satellite-01.domain"
 zone = "icinga-qa-satellite-01.domain"
}

########## Configuration of the satellite: ##########

etc/icinga2/icinga2.conf

include “zones.conf”
…
// Disabled by the node setup CLI command on 2020-11-20 09:48:47 +0100
// include_recursive "conf.

etc/icinga2/zones.conf

##### MASTER  #####

object Endpoint "icinga-qa.domain" {
        host = "10.116.219.2"
}

object Zone "master" {
        endpoints = [ "icinga-qa.domain" ]

}

##### SATELLITE #####

object Endpoint "icinga-qa-satellite-01.domain" {
        host = "icinga-qa-satellite-01.domain"
}

object Zone "icinga-qa-satellite-01.domain" {
        endpoints = [ "icinga-qa-satellite-01.domain" ]
        parent = "master"
}

###### GLOBAL-TEMPLATES ######

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

rsx · March 24, 2021, 7:29am

Hi & welcome to the icinga community,

In this case you don’t need

command_endpoint = host.vars.agent_endpoint

vars.agent_endpoint = "icinga-qa-satellite-01.domain"
 zone = "icinga-qa-satellite-01.domain"

since the host object is defined in /etc/icinga2/zones.d/icinga-qa-satellite-01.domain.

Any hints in the logs? Certificate is signed?

lobr · March 24, 2021, 9:33am

Hello Roland,
thank you for the quick reply to this thread!
The three lines you stated and which seem to be obsolete have been removed.

Regarding the certificate:

Following Distributed Monitoring - Icinga 2 , the master’s ca.cert has been copied to the satellite at /var/lib/icinga2/certs/ca.crt
So this should be alright?

The debug.log doesn’t return anything suspicious (no warning or critical messages…)

root@icinga-qa:/etc/icinga2/zones.d/master>>tail -f /var/log/icinga2/debug.log | grep "icinga-qa-satellite-01"

[2021-03-24 10:26:57 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' messag
e from identity 'icinga-qa-satellite-01.domain'.
[2021-03-24 10:27:00 +0100] notice/ApiListener: Sending message 'event::SetForceNextCheck' 
to 'icinga-qa-satellite-01.domain'
[2021-03-24 10:27:00 +0100] notice/ApiListener: Sending message 'event::SetNextCheck' to 'i
cinga-qa-satellite-01.domain'
[2021-03-24 10:27:00 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SE
T acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'cluster-zone'
,  check_source = 'icinga-qa.domain',  check_type = '0',  current_ch
eck_attempt = '1',  current_notification_number = '0',  current_state = '0',  endpoint_obje
ct_id = 4101,  event_handler_enabled = '1',  execution_time = '0.000130',  flap_detection_e
nabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable 
= '1',  last_check = FROM_UNIXTIME(1616578020),  last_hard_state = '0',  last_hard_state_ch
ange = FROM_UNIXTIME(1616510117),  last_state_change = FROM_UNIXTIME(1616510117),  last_tim
e_ok = FROM_UNIXTIME(1616578020),  last_time_unknown = FROM_UNIXTIME(1616510068),  latency 
= '0.000140',  long_output = '',  max_check_attempts = '3',  next_check = FROM_UNIXTIME(161
6578050),  normal_check_interval = '0.500000',  notifications_enabled = '1',  original_attr
ibutes = 'null',  output = 'Zone \'icinga-qa-satellite-01.domain\' i
s connected. Log lag: less than 1 millisecond',  passive_checks_enabled = '1',  percent_sta
te_change = '0',  perfdata = 'slave_lag=0;0;0 last_messages_sent=1616578020.118525 last_mes
sages_received=1616578017.844264 sum_messages_sent_per_second=0.116667 sum_messages_receive
d_per_second=0.250000 sum_bytes_sent_per_second=12.133333 sum_bytes_received_per_second=21.
250000',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_chec
k_interval = '0.166667',  scheduled_downtime_depth = '0',  service_object_id = 4857,  shoul
d_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1616578020) WH
ERE service_object_id = 4857
[2021-03-24 10:27:01 +0100] debug/ApiListener: Not connecting to Endpoint 'icinga-qa-satell
ite-01.domain' because we're already connected to it.
[2021-03-24 10:27:01 +0100] notice/ApiListener: Connected endpoints: icinga-qa-satellite-01
.domain (1)
[2021-03-24 10:27:02 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' messag
e from identity 'icinga-qa-satellite-01.domain'.
[2021-03-24 10:27:07 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' messag
e from identity 'icinga-qa-satellite-01.domain'.
[2021-03-24 10:27:11 +0100] debug/ApiListener: Not connecting to Endpoint 'icinga-qa-satell
ite-01.domain' because we're already connected to it.
[2021-03-24 10:27:11 +0100] notice/ApiListener: Connected endpoints: icinga-qa-satellite-01
.domain (1)
[2021-03-24 10:27:12 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' messag
e from identity 'icinga-qa-satellite-01.domain'.
[2021-03-24 10:27:14 +0100] notice/JsonRpcConnection: Received 'event::Heartbeat' message f
rom identity 'icinga-qa-satellite-01.domain'.
[2021-03-24 10:27:17 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' messag
e from identity 'icinga-qa-satellite-01.domain'.

Tcpdump on the test host that should be monitored does not return any traffic on either the master, nor the satellite when force executing the check.

tcpdump -i ens192 host 192.168.1.22

rsx · March 24, 2021, 12:22pm

No, this is not necessary e.g. if you run icinga2 node wizard which is the recommended way. With that icinga creates a certificate for the satellite and send it to the master where is needs to be signed. This is called on-demand signing which is easier to understand especially for beginners.

I’d also recommend a service check based on cluster-zone to keep an eye on zone connections.

lobr · March 25, 2021, 8:32am

Hi Roland,

thanks for the hint to use a cluster zone service check. The satellite is up regarding that service.

But altough I did run that Icinga2 setup wizard already in the past, I gave it a second try. Sadly the check from the satellite is still in pending.

Here is the content of the setup screen:

Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is an agent/satellite setup ('n' installs a master setup) [Y/n]: Y

Starting the Agent/Satellite setup routine...

Please specify the common name (CN) [icinga-qa-satellite-01.domain]:

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): icinga-qa.domain

Do you want to establish a connection to the parent node from this node? [Y/n]: Y
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): icinga-qa.domain
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]: N
Parent certificate information:

 Version:             3
 Subject:             CN = icinga-qa.domain
 Issuer:              CN = Icinga CA
 Valid From:          Nov 18 14:41:20 2019 GMT
 Valid Until:         Nov 14 14:41:20 2034 GMT
 Serial:              70:3a:af:b3:e5:0e:e1:fa:c1:26:2f:58:a2:e2:23:8e:a3:1d:9d:98

 Signature Algorithm: sha256WithRSAEncryption
 Subject Alt Names:   icinga-qa.domain
 Fingerprint:         5F 4C D6 E9 B1 8D 5B 99 55 24 F4 1B BE E0 F5 48 26 33 58 E8 B4 D7 2E 39 4E DF EC C1 A6 F1 ED 9F

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'icinga-qa-satellite-01.domain'): c98d1922ca4f8a1ea1bf188be4141f197a01ed2e
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...

Local zone name [icinga-qa-satellite-01.domain]:
Parent zone name [master]: master

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]: N

Do you want to disable the inclusion of the conf.d directory [Y/n]: Y
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

After this setup I got the error message: critical/config: Error: Infinite recursion detected while resolving zone graph. Check your zone hierarchy.
Which was caused by a object Zone which hasn’t been a naming assigned by the wizard:

/etc/icinga2/zones.conf(18): object Zone "" {
                                              ^^^^^^^^^^^^^^
/etc/icinga2/zones.conf(19):  endpoints = [ "icinga-qa-satellite-01.domain" ]
/etc/icinga2/zones.conf(20):  parent = "master"

I filled the empty zone name which the naming it had before, so that the zones.conf on the satellite looks like that:

object Endpoint "icinga-qa.domain" {
}

object Zone "master" {
        endpoints = [ "icinga-qa.domain" ]

}


# 2021-03-12: Added Endpoint for Test-Satellite: icinga-qa-satellite-01.aircloud.common.ai$
object Endpoint "icinga-qa-satellite-01.domain" {
        host = "10.116.220.140"
}

# 2021-03-12: Added Endpoint for Test-Satellite: icinga-qa-satellite-01.aircloud.common.ai$
object Zone "icinga-qa-satellite-01.domain" {
        endpoints = [ "icinga-qa-satellite-01.domain" ]
        parent = "master"
}


object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

rsx · March 25, 2021, 12:42pm

Can you please run these commands on master and satellite and share their outputs?

icinga2 object list -n “Test 192.168.1.22”
icinga2 object list -n test_ping

lobr · March 25, 2021, 2:58pm

These are the outputs of the commands:

COMMAND: icinga2 object list -n "Test 192.168.1.22"

ON MASTER

root@icinga-qa:~>>icinga2 object list -n "Test 192.168.1.22" 
Object 'Test 192.168.1.22' of type 'Host':
% declared in '/etc/icinga2/zones.d/icinga-qa-satellite-01.domain/hosts.conf', lines 13:1-13:31
* __name = "Test 192.168.1.22"
* action_url = ""
* address = "192.168.1.22"
% = modified in '/etc/icinga2/zones.d/icinga-qa-satellite-01.domain/hosts.conf', lines 15:1-15:24
* address6 = ""
* check_command = "hostalive"
% = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 19:3-19:29
* check_interval = 300
% = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 16:3-16:21
* check_period = ""
* check_timeout = null
* command_endpoint = ""
* display_name = "Test 192.168.1.22"
* enable_active_checks = true
* enable_event_handler = true
* enable_flapping = false
* enable_notifications = true
* enable_passive_checks = true
* enable_perfdata = true
* event_command = ""
* flapping_threshold = 0
* flapping_threshold_high = 30
* flapping_threshold_low = 25
* groups = [ ]
* icon_image = ""
* icon_image_alt = ""
* max_check_attempts = 10
% = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 15:3-15:25
* name = "Test 192.168.1.22"
* notes = ""
* notes_url = ""
* package = "_etc"
* retry_interval = 120
% = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 17:3-17:21
* source_location
* first_column = 1
* first_line = 13
* last_column = 31
* last_line = 13
* path = "/etc/icinga2/zones.d/icinga-qa-satellite-01.domain/hosts.conf"
* templates = [ "Test 192.168.1.22", "generic-host" ]
% = modified in '/etc/icinga2/zones.d/icinga-qa-satellite-01.domain/hosts.conf', lines 13:1-13:31
% = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 14:1-14:28
* type = "Host"
* vars
* storage = "INFC-NFS_VMdataStore_DeDup-04"
  % = modified in '/etc/icinga2/zones.d/icinga-qa-satellite-01.domain/hosts.conf', lines 17:1-17:46
* type = "test"
  % = modified in '/etc/icinga2/zones.d/icinga-qa-satellite-01.domain/hosts.conf', lines 16:1-16:18
* volatile = false
* zone = "icinga-qa-satellite-01.domain"

ON SATELLITE

-> NO OUTPUT

COMMAND: icinga2 object list -n test_ping

ON MASTER

root@icinga-qa:~>>icinga2 object list -n test_ping
Object 'Test 192.168.1.22!test_ping' of type 'Service':
  % declared in '/etc/icinga2/zones.d/global-templates/services.conf', lines 98:1-98:31
  * __name = "Test 192.168.1.22!test_ping"
  * action_url = ""
  * check_command = "ping4"
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 100:3-100:25
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 103:3-103:25
  * check_interval = 300
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 39:3-39:21
  * check_period = ""
  * check_timeout = null
  * command_endpoint = ""
  * display_name = "test_ping"
  * enable_active_checks = true
  * enable_event_handler = true
  * enable_flapping = false
  * enable_notifications = true
  * enable_passive_checks = true
  * enable_perfdata = true
  * event_command = ""
  * flapping_threshold = 0
  * flapping_threshold_high = 30
  * flapping_threshold_low = 25
  * groups = [ ]
  * host_name = "Test 192.168.1.22"
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 98:1-98:31
  * icon_image = ""
  * icon_image_alt = ""
  * max_check_attempts = 10
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 38:3-38:25
  * name = "test_ping"
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 98:1-98:31
  * notes = ""
  * notes_url = ""
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 98:1-98:31
  * retry_interval = 120
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 40:3-40:21
  * source_location
    * first_column = 1
    * first_line = 98
    * last_column = 31
    * last_line = 98
    * path = "/etc/icinga2/zones.d/global-templates/services.conf"
  * templates = [ "test_ping", "generic-service" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 98:1-98:31
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 37:1-37:34
  * type = "Service"
  * vars = null
  * volatile = false
  * zone = "icinga-qa-satellite-01.domain"
    % = modified in '/etc/icinga2/zones.d/global-templates/services.conf', lines 98:1-98:31

ON SATELLITE

-> NO OUTPUT

rsx · March 26, 2021, 8:05am

That means sync is not working. Do you have accept_config = true in your satellite’s /etc/icinga2/features-enabled/api.conf? Is api enabled on your satellite? Check with icinga2 feature list.

lobr · March 26, 2021, 8:39am

Yes, accept_config is set to true on the satellite.
/etc/icinga2/features/enabled/api.conf:

/**
 * The API listener is used for distributed monitoring setups.
 */
object ApiListener "api" {
  accept_config = true
  accept_commands = true
}

The command icinga2 feature list returns:

[root@icinga-qa-satellite-01 features-enabled]# icinga2 feature list
Disabled features: command compatlog elasticsearch gelf graphite icingadb influxdb livestatus notification opentsdb perfdata statusdata syslog
Enabled features: api checker debuglog mainlog

rsx · March 26, 2021, 9:14am

That’s crazy. cluster-zone tells you the zone is connected and you have no hints in the logs? Sounds strange. Which versions of icinga are involved?

lobr · March 26, 2021, 10:34am

The version I am using is 2.12.3.

root@icinga-qa:~>>icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: 2.12.3)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Red Hat Enterprise Linux Server
  Platform version: 7.8 (Maipo)
  Kernel: Linux
  Kernel version: 3.10.0-1127.10.1.el7.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 4.8.5
  Build host: runner-hh8q3bz2-project-322-concurrent-0
  OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

This is the part of the debug.log when the service check “test_ping” on “Test 192.168.1.22” is manually triggered:

[2021-03-26 11:15:26 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:15:26 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:15:27 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:15:27 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:15:28 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:15:28 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:15:28 +0100] information/ExternalCommandListener: Executing external command: [1616753728] SCHEDULE_FORCED_SVC_CHECK;Test 192.168.1.22;test_ping;1616753728
[2021-03-26 11:15:28 +0100] notice/ExternalCommandProcessor: Rescheduling next check for service 'test_ping'
[2021-03-26 11:15:28 +0100] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2021-03-26 11:15:28 +0100] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'icinga-qa-satellite-01.domain'
[2021-03-26 11:15:28 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-03-26 11:15:28 +0100] notice/ApiListener: Sending message 'event::SetNextCheck' to 'icinga-qa-satellite-01.domain'
[2021-03-26 11:15:29 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET next_check = FROM_UNIXTIME(1616753728) WHERE service_object_id = 4856
[2021-03-26 11:15:29 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:15:29 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:15:29 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 189; Checks/s: 0.483333
[2021-03-26 11:15:30 +0100] debug/InfluxdbWriter: Timer expired writing 0 data points
[2021-03-26 11:15:30 +0100] notice/DbConnection: Updating programstatus table.
[2021-03-26 11:15:30 +0100] debug/ApiListener: Not connecting to Endpoint 'icinga-qa-satellite-01.domain' because we're already connected to it.
[2021-03-26 11:15:30 +0100] debug/ApiListener: Not connecting to Endpoint 'icinga-qa.domain' because that's us.
[2021-03-26 11:15:30 +0100] notice/ApiListener: Current zone master: icinga-qa.domain
[2021-03-26 11:15:30 +0100] notice/ApiListener: Connected endpoints: icinga-qa-satellite-01.domain (1)
[2021-03-26 11:15:30 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'icinga-qa.domain'.
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: DELETE FROM icinga_programstatus WHERE instance_id = 1
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES ('1', '1', '1', 'icinga-qa.domain', '1', '1', 1, '1', FROM_UNIXTIME(1616753730), '1', '1', '1', '12904', '1', FROM_UNIXTIME(1616734835), '2.12.3', FROM_UNIXTIME(1616753730))
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '150')
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '150')
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '55')
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '55')
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:15:30 +0100] debug/IdoMysqlConnection: Query: BEGIN
...

##############

...
[2021-03-26 11:16:14 +0100] information/ExternalCommandListener: Executing external command: [1616753774] SCHEDULE_FORCED_SVC_CHECK;Test 192.168.1.22;test_ping;1616753774
[2021-03-26 11:16:14 +0100] notice/ExternalCommandProcessor: Rescheduling next check for service 'test_ping'
[2021-03-26 11:16:14 +0100] notice/ApiListener: Relaying 'event::SetForceNextCheck' message
[2021-03-26 11:16:14 +0100] notice/ApiListener: Sending message 'event::SetForceNextCheck' to 'icinga-qa-satellite-01.domain'
[2021-03-26 11:16:14 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-03-26 11:16:14 +0100] notice/ApiListener: Sending message 'event::SetNextCheck' to 'icinga-qa-satellite-01.domain'
[2021-03-26 11:16:15 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from identity 'icinga-qa-satellite-01.domain'.
[2021-03-26 11:16:15 +0100] debug/CheckerComponent: Scheduling info for checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' (2021-03-26 11:16:15 +0100):
 Object 'icinga-qa-satellite-01.domain!satellite-zone-health', Next Check: 2021-03-26 11:16:15 +0100(1.61675e+09).
[2021-03-26 11:16:15 +0100] debug/CheckerComponent: Executing check for 'icinga-qa-satellite-01.domain!satellite-zone-health'
[2021-03-26 11:16:15 +0100] debug/Checkable: Update checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' with check interval '30' from last check time at
 2021-03-26 11:15:45 +0100 (1.61675e+09) to next check time at 2021-03-26 11:16:45 +0100 (1.61675e+09).
[2021-03-26 11:16:15 +0100] debug/Checkable: Update checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' with check interval '30' from last check time at
 2021-03-26 11:16:15 +0100 (1.61675e+09) to next check time at 2021-03-26 11:16:45 +0100 (1.61675e+09).
[2021-03-26 11:16:15 +0100] notice/ApiListener: Relaying 'event::SetLastCheckStarted' message
[2021-03-26 11:16:15 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-03-26 11:16:15 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2021-03-26 11:16:15 +0100] notice/ApiListener: Relaying 'event::CheckResult' message
[2021-03-26 11:16:15 +0100] debug/CheckerComponent: Check finished for object 'icinga-qa-satellite-01.domain!satellite-zone-health'
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-
qa-satellite-01.domain,service=satellite-zone-health,metric=slave_lag unit="s",value=0 1616753775'.
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-qa-satellite-01.domain,service=satellite-zone-health,metric=last_messages_sent value=1616753774.958799 1616753775'.
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-qa-satellite-01.domain,service=satellite-zone-health,metric=last_messages_received value=1616753775.284045 1616753775'.
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-qa-satellite-01.domain,service=satellite-zone-health,metric=sum_messages_sent_per_second value=0.116667 1616753775'.
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-qa-satellite-01.domain,service=satellite-zone-health,metric=sum_messages_received_per_second value=0.266667 1616753775'.
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-qa-satellite-01.domain,service=satellite-zone-health,metric=sum_bytes_sent_per_second value=14 1616753775'.
[2021-03-26 11:16:15 +0100] debug/InfluxdbWriter: Checkable 'icinga-qa-satellite-01.domain!satellite-zone-health' adds to metric list:'cluster-zone,hostname=icinga-qa-satellite-01.domain,service=satellite-zone-health,metric=sum_bytes_received_per_second value=22.766667 1616753775'.
[2021-03-26 11:16:15 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_hoststatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'hostalive',  check_source = 'icinga-qa.domain',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '0',  endpoint_object_id = 4101,  event_handler_enabled = '1',  execution_time = '4.006499',  flap_detection_enabled = '0',  has_been_checked = '1',  host_object_id = 4235,  instance_id = 1,  is_flapping = '0',  is_reachable = '1',  last_check = FROM_UNIXTIME(1616753774),  last_hard_state = '0',  last_hard_state_change = FROM_UNIXTIME(1616692718),  last_state_change = FROM_UNIXTIME(1616692718),  last_time_down = FROM_UNIXTIME(1616692597),  last_time_up = FROM_UNIXTIME(1616753774),  latency = '0.000429',  long_output = '',  max_check_attempts = '10',  next_check = FROM_UNIXTIME(1616754073),  normal_check_interval = '5',  notifications_enabled = '1',  original_attributes = 'null',  output = 'PING OK - Packet loss = 0%, RTA = 0.69 ms',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = 'rta=0.691000ms;3000.000000;5000.000000;0.000000 pl=0%;80;100;0',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '2',  scheduled_downtime_depth = '0',  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1616753774) WHERE host_object_id = 4235
[2021-03-26 11:16:15 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET next_check = FROM_UNIXTIME(1616753774) WHERE service_object_id = 4856
[2021-03-26 11:16:15 +0100] debug/IdoMysqlConnection: Query: UPDATE icinga_servicestatus SET acknowledgement_type = '0',  active_checks_enabled = '1',  check_command = 'cluster-zone',  check_source = 'icinga-qa.domain',  check_type = '0',  current_check_attempt = '1',  current_notification_number = '0',  current_state = '0',  endpoint_object_id = 4101,  event_handler_enabled = '1',  execution_time = '0.000115',  flap_detection_enabled = '0',  has_been_checked = '1',  instance_id = 1,  is_flapping = '0',  is_reachable = '0',  last_check = FROM_UNIXTIME(1616753775),  last_hard_state = '0',  last_hard_state_change = FROM_UNIXTIME(1616747358),  last_state_change = FROM_UNIXTIME(1616747358),  last_time_critical = FROM_UNIXTIME(1616747349),  last_time_ok = FROM_UNIXTIME(1616753775),  latency = '0.000101',  long_output = '',  max_check_attempts = '3',  next_check = FROM_UNIXTIME(1616753805),  normal_check_interval = '0.500000',  notifications_enabled = '1',  original_attributes = 'null',  output = 'Zone \'icinga-qa-satellite-01.domain\' is connected. Log lag: less than 1 millisecond',  passive_checks_enabled = '1',  percent_state_change = '0',  perfdata = 'slave_lag=0;0;0 last_messages_sent=1616753774.958799 last_messages_received=1616753775.284045 sum_messages_sent_per_second=0.116667 sum_messages_received_per_second=0.266667 sum_bytes_sent_per_second=14 sum_bytes_received_per_second=22.766667',  problem_has_been_acknowledged = '0',  process_performance_data = '1',  retry_check_interval = '0.166667',  scheduled_downtime_depth = '0',  service_object_id = 4866,  should_be_scheduled = '1',  state_type = '1',  status_update_time = FROM_UNIXTIME(1616753775) WHERE service_object_id = 4866
[2021-03-26 11:16:15 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:15 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:16 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:16 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:17 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:17 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:18 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:18 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:19 +0100] notice/CheckerComponent: Pending checkables: 0; Idle checkables: 189; Checks/s: 0.516667
[2021-03-26 11:16:19 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:19 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:20 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.6/s (36/min 226/5min 680/15min); empty in infinite time, your task handler isn't able to keep up
[2021-03-26 11:16:20 +0100] debug/InfluxdbWriter: Timer expired writing 9 data points
[2021-03-26 11:16:20 +0100] debug/InfluxdbWriter: Flushing data buffer to InfluxDB.
[2021-03-26 11:16:20 +0100] notice/InfluxdbWriter: Reconnecting to InfluxDB on host '127.0.0.1' port '8086'.
[2021-03-26 11:16:20 +0100] notice/DbConnection: Updating programstatus table.
[2021-03-26 11:16:20 +0100] notice/ApiListener: Updating object authority for objects at endpoint 'icinga-qa.domain'.
[2021-03-26 11:16:20 +0100] debug/ApiListener: Not connecting to Endpoint 'icinga-qa-satellite-01.domain' because we're already connected to it.
[2021-03-26 11:16:20 +0100] debug/ApiListener: Not connecting to Endpoint 'icinga-qa.domain' because that's us.
[2021-03-26 11:16:20 +0100] notice/ApiListener: Current zone master: icinga-qa.domain
[2021-03-26 11:16:20 +0100] notice/ApiListener: Connected endpoints: icinga-qa-satellite-01.domain (1)
[2021-03-26 11:16:20 +0100] notice/JsonRpcConnection: Received 'log::SetLogPosition' message from identity 'icinga-qa-satellite-01.domain'.
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: DELETE FROM icinga_programstatus WHERE instance_id = 1
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_programstatus (active_host_checks_enabled, active_service_checks_enabled, daemon_mode, endpoint_name, event_handlers_enabled, flap_detection_enabled, instance_id, is_currently_running, last_command_check, notifications_enabled, passive_host_checks_enabled, passive_service_checks_enabled, process_id, process_performance_data, program_start_time, program_version, status_update_time) VALUES ('1', '1', '1', 'icinga-qa.domain', '1', '1', 1, '1', FROM_UNIXTIME(1616753780), '1', '1', '1', '12904', '1', FROM_UNIXTIME(1616734835), '2.12.3', FROM_UNIXTIME(1616753780))
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '150')
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '150')
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '55')
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '55')
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:20 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:21 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:21 +0100] debug/IdoMysqlConnection: Query: BEGIN
[2021-03-26 11:16:22 +0100] debug/IdoMysqlConnection: Query: COMMIT
[2021-03-26 11:16:22 +0100] debug/IdoMysqlConnection: Query: BEGIN

rsx · March 26, 2021, 10:52am

When you do a reload on your master, its icinga2.log should have entries like this:

[2021-03-26 11:46:21 +0100] information/ApiListener: New client connection for identity 'icinga-qa-satellite-01.domain' to [10.116.220.140]:5665
[2021-03-26 11:46:21 +0100] information/ApiListener: Sending config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.

lobr · March 26, 2021, 11:41am

After a systemctl reload icinga2.service on the master, the two mentioned outputs were written to the icinga2.log. Here is the whole log:

[2021-03-26 12:35:16 +0100] information/Application: Received request to shut down.
[2021-03-26 12:35:16 +0100] information/Application: Shutting down...
[2021-03-26 12:35:16 +0100] information/CheckerComponent: 'checker' stopped.
[2021-03-26 12:35:16 +0100] information/NotificationComponent: 'notification' stopped.
[2021-03-26 12:35:16 +0100] information/ExternalCommandListener: 'command' stopped.
[2021-03-26 12:35:16 +0100] information/PerfdataWriter: 'perfdata' paused.
[2021-03-26 12:35:16 +0100] information/InfluxdbWriter: 'influxdb' paused.
[2021-03-26 12:35:16 +0100] information/DbConnection: Pausing IDO connection: ido-mysql
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga_ido'.
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: 'ido-mysql' paused.
[2021-03-26 12:35:16 +0100] information/DbConnection: 'ido-mysql' stopped.
[2021-03-26 12:35:16 +0100] information/ApiListener: 'api' stopped.
[2021-03-26 12:35:16 +0100] information/FileLogger: 'main-log' started.
[2021-03-26 12:35:16 +0100] information/FileLogger: 'debug-file' started.
[2021-03-26 12:35:16 +0100] information/ApiListener: 'api' started.
[2021-03-26 12:35:16 +0100] information/ApiListener: Copying 3 zone configuration files for zone 'icinga-qa-satellite-01.domain' to '/var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/health.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/hosts.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/pet.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Copying 3 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/services.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates.conf
[2021-03-26 12:35:16 +0100] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/master/hosts/.bigdata.conf.swp' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
Context:
        (0) Creating config update for file '/etc/icinga2/zones.d/master/hosts/.bigdata.conf.swp'
        (1) Activating object 'api' of type 'ApiListener'

[2021-03-26 12:35:16 +0100] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/master/hosts/.netscaler.conf.swp' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
Context:
        (0) Creating config update for file '/etc/icinga2/zones.d/master/hosts/.netscaler.conf.swp'
        (1) Activating object 'api' of type 'ApiListener'

[2021-03-26 12:35:16 +0100] information/ApiListener: Copying 18 zone configuration files for zone 'master' to '/var/lib/icinga2/api/zones/master'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/arista.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/arista.conf.save
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/brunoTest.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/cluster.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/dc.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/dummy.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/hosts.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/icinga.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/moonshot.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/openstack.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/ops.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/parentstest.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/pets.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/proxy.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/switch.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/vsa.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/services.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-03-26 12:35:16 +0100] information/ExternalCommandListener: 'command' started.
[2021-03-26 12:35:16 +0100] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'

[2021-03-26 12:35:16 +0100] information/ApiListener: Reconnecting to endpoint 'icinga-qa-satellite-01.domain' via host '10.116.220.140' and port '5665'
[2021-03-26 12:35:16 +0100] information/DbConnection: 'ido-mysql' started.
[2021-03-26 12:35:16 +0100] information/PerfdataWriter: 'perfdata' resumed.
[2021-03-26 12:35:16 +0100] information/InfluxdbWriter: 'influxdb' resumed.
[2021-03-26 12:35:16 +0100] information/NotificationComponent: 'notification' started.
[2021-03-26 12:35:16 +0100] information/CheckerComponent: 'checker' started.
[2021-03-26 12:35:16 +0100] information/ConfigItem: Activated all objects.
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2021-03-26 12:35:16 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2021-03-26 12:35:16 +0100] information/ApiListener: New client connection for identity 'icinga-qa-satellite-01.domain' to [10.116.220.140]:5665
[2021-03-26 12:35:16 +0100] information/ApiListener: Sending config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Syncing configuration files for zone 'icinga-qa-satellite-01.domain' to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished reconnecting to endpoint 'icinga-qa-satellite-01.domain' via host '10.116.220.140' and port '5665'
[2021-03-26 12:35:16 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/JsonRpcConnection: Received certificate request for CN 'icinga-qa-satellite-01.domain' signed by our CA.
[2021-03-26 12:35:16 +0100] information/JsonRpcConnection: The certificate for CN 'icinga-qa-satellite-01.domain' is valid and uptodate. Skipping automated renewal.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Sending replay log for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished syncing endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga_ido' in 0.091711 second(s).
[2021-03-26 12:35:26 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.916667/s (55/min 55/5min 55/15min);
[2021-03-26 12:35:26 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:35:26 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.05/s (3/min 3/5min 3/15min);
[2021-03-26 12:35:26 +0100] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)

rsx · March 26, 2021, 11:59am

Can you share icinga2.log from your satellite within the same time frame?

Your master is sending conf files, but obviuosly your satellite does not receive or activate them. You could try cleaning at your satellite:

systemctl stop icinga2
rm -rf /var/lib/icinga2/api/{packages,zones,zones-stage}/*
systemctl start icinga2

lobr · March 26, 2021, 1:09pm

Here are both logs from the master and the satellite after a restart.

Master

root@icinga-qa:~>>tail -f n70 /var/log/icinga2/icinga2.log
tail: cannot open ‘n70’ for reading: No such file or directory
==> /var/log/icinga2/icinga2.log <==
[2021-03-26 12:30:20 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.616667/s (37/min 226/5min 680/15min); empty in infinite time, your task handler isn't able to keep up
[2021-03-26 12:30:30 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.566667/s (34/min 226/5min 680/15min);
[2021-03-26 12:30:38 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 12:30:40 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.733333/s (44/min 225/5min 679/15min);
[2021-03-26 12:31:00 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.716667/s (43/min 226/5min 680/15min); empty in 9 seconds
[2021-03-26 12:33:20 +0100] information/IdoMysqlConnection: Pending queries: 1 (Input: 3/s; Output: 3/s)
[2021-03-26 12:34:10 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 3.2/s (192/min 782/5min 2362/15min);
[2021-03-26 12:34:10 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.866667/s (52/min 223/5min 678/15min);
[2021-03-26 12:34:10 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:34:20 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.883333/s (53/min 226/5min 680/15min);
[2021-03-26 12:34:28 +0100] information/ExternalCommandListener: Executing external command: [1616758467] SCHEDULE_FORCED_SVC_CHECK;Test 192.168.1.22;test_ping;1616758467
[2021-03-26 12:35:10 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 1, rate: 0.616667/s (37/min 225/5min 678/15min);
[2021-03-26 12:35:16 +0100] information/Application: Received request to shut down.
[2021-03-26 12:35:16 +0100] information/Application: Shutting down...
[2021-03-26 12:35:16 +0100] information/CheckerComponent: 'checker' stopped.
[2021-03-26 12:35:16 +0100] information/NotificationComponent: 'notification' stopped.
[2021-03-26 12:35:16 +0100] information/ExternalCommandListener: 'command' stopped.
[2021-03-26 12:35:16 +0100] information/PerfdataWriter: 'perfdata' paused.
[2021-03-26 12:35:16 +0100] information/InfluxdbWriter: 'influxdb' paused.
[2021-03-26 12:35:16 +0100] information/DbConnection: Pausing IDO connection: ido-mysql
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: Disconnected from 'ido-mysql' database 'icinga_ido'.
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: 'ido-mysql' paused.
[2021-03-26 12:35:16 +0100] information/DbConnection: 'ido-mysql' stopped.
[2021-03-26 12:35:16 +0100] information/ApiListener: 'api' stopped.
[2021-03-26 12:35:16 +0100] information/FileLogger: 'main-log' started.
[2021-03-26 12:35:16 +0100] information/FileLogger: 'debug-file' started.
[2021-03-26 12:35:16 +0100] information/ApiListener: 'api' started.
[2021-03-26 12:35:16 +0100] information/ApiListener: Copying 3 zone configuration files for zone 'icinga-qa-satellite-01.domain' to '/var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/health.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/hosts.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/pet.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Copying 3 zone configuration files for zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/commands.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/services.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates.conf
[2021-03-26 12:35:16 +0100] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/master/hosts/.bigdata.conf.swp' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
Context:
        (0) Creating config update for file '/etc/icinga2/zones.d/master/hosts/.bigdata.conf.swp'
        (1) Activating object 'api' of type 'ApiListener'

[2021-03-26 12:35:16 +0100] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/master/hosts/.netscaler.conf.swp' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
Context:
        (0) Creating config update for file '/etc/icinga2/zones.d/master/hosts/.netscaler.conf.swp'
        (1) Activating object 'api' of type 'ApiListener'

[2021-03-26 12:35:16 +0100] information/ApiListener: Copying 18 zone configuration files for zone 'master' to '/var/lib/icinga2/api/zones/master'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/arista.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/arista.conf.save
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/brunoTest.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/cluster.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/dc.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/dummy.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/hosts.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/icinga.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/moonshot.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/openstack.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/ops.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/parentstest.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/pets.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/proxy.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/switch.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/vsa.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/services.conf
[2021-03-26 12:35:16 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-03-26 12:35:16 +0100] information/ExternalCommandListener: 'command' started.
[2021-03-26 12:35:16 +0100] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'

[2021-03-26 12:35:16 +0100] information/ApiListener: Reconnecting to endpoint 'icinga-qa-satellite-01.domain' via host '10.116.220.140' and port '5665'
[2021-03-26 12:35:16 +0100] information/DbConnection: 'ido-mysql' started.
[2021-03-26 12:35:16 +0100] information/PerfdataWriter: 'perfdata' resumed.
[2021-03-26 12:35:16 +0100] information/InfluxdbWriter: 'influxdb' resumed.
[2021-03-26 12:35:16 +0100] information/NotificationComponent: 'notification' started.
[2021-03-26 12:35:16 +0100] information/CheckerComponent: 'checker' started.
[2021-03-26 12:35:16 +0100] information/ConfigItem: Activated all objects.
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2021-03-26 12:35:16 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2021-03-26 12:35:16 +0100] information/ApiListener: New client connection for identity 'icinga-qa-satellite-01.domain' to [10.116.220.140]:5665
[2021-03-26 12:35:16 +0100] information/ApiListener: Sending config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Syncing configuration files for zone 'icinga-qa-satellite-01.domain' to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished reconnecting to endpoint 'icinga-qa-satellite-01.domain' via host '10.116.220.140' and port '5665'
[2021-03-26 12:35:16 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/JsonRpcConnection: Received certificate request for CN 'icinga-qa-satellite-01.domain' signed by our CA.
[2021-03-26 12:35:16 +0100] information/JsonRpcConnection: The certificate for CN 'icinga-qa-satellite-01.domain' is valid and uptodate. Skipping automated renewal.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Sending replay log for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/ApiListener: Finished syncing endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 12:35:16 +0100] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga_ido' in 0.091711 second(s).
[2021-03-26 12:35:26 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 0.916667/s (55/min 55/5min 55/15min);
[2021-03-26 12:35:26 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:35:26 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.05/s (3/min 3/5min 3/15min);
[2021-03-26 12:35:26 +0100] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2021-03-26 12:38:26 +0100] information/IdoMysqlConnection: Pending queries: 10 (Input: 3/s; Output: 2/s)
[2021-03-26 12:40:16 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 12:40:36 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:40:36 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 2.03333/s (122/min 788/5min 865/15min);
[2021-03-26 12:40:36 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.583333/s (35/min 228/5min 236/15min);
[2021-03-26 12:40:36 +0100] information/IdoMysqlConnection: Pending queries: 11 (Input: 3/s; Output: 2/s)
[2021-03-26 12:42:36 +0100] information/IdoMysqlConnection: Pending queries: 10 (Input: 3/s; Output: 2/s)
[2021-03-26 12:45:16 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 12:45:36 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 2.03333/s (122/min 792/5min 1661/15min);
[2021-03-26 12:45:36 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:45:36 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.583333/s (35/min 227/5min 464/15min);
[2021-03-26 12:45:46 +0100] information/IdoMysqlConnection: Pending queries: 9 (Input: 4/s; Output: 4/s)
[2021-03-26 12:50:16 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 12:50:36 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 2.03333/s (122/min 792/5min 2380/15min);
[2021-03-26 12:50:36 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:50:46 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.516667/s (31/min 228/5min 685/15min);
[2021-03-26 12:50:46 +0100] information/IdoMysqlConnection: Pending queries: 10 (Input: 4/s; Output: 4/s)
[2021-03-26 12:55:16 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 12:55:36 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 12:55:36 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 1.86667/s (112/min 800/5min 2388/15min);
[2021-03-26 12:55:46 +0100] information/IdoMysqlConnection: Pending queries: 0 (Input: 4/s; Output: 4/s)
[2021-03-26 12:55:56 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.6/s (36/min 228/5min 685/15min);
[2021-03-26 13:00:16 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:00:46 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:00:46 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 1.86667/s (112/min 792/5min 2380/15min);
[2021-03-26 13:00:56 +0100] information/IdoMysqlConnection: Pending queries: 0 (Input: 4/s; Output: 4/s)
[2021-03-26 13:01:06 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.666667/s (40/min 233/5min 691/15min);
[2021-03-26 13:05:17 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:05:46 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:05:46 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 1.86667/s (112/min 796/5min 2382/15min);
[2021-03-26 13:05:56 +0100] information/IdoMysqlConnection: Pending queries: 9 (Input: 5/s; Output: 5/s)
[2021-03-26 13:06:06 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.7/s (42/min 228/5min 696/15min);
[2021-03-26 13:10:17 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:10:56 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 2.5/s (150/min 798/5min 2410/15min);
[2021-03-26 13:10:56 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:10:56 +0100] information/IdoMysqlConnection: Pending queries: 0 (Input: 5/s; Output: 5/s)
[2021-03-26 13:11:16 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.833333/s (50/min 229/5min 692/15min);
[2021-03-26 13:15:17 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:16:06 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:16:06 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 2.66667/s (160/min 790/5min 2390/15min);
[2021-03-26 13:16:06 +0100] information/IdoMysqlConnection: Pending queries: 6 (Input: 5/s; Output: 5/s)
[2021-03-26 13:16:26 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.85/s (51/min 228/5min 690/15min);
[2021-03-26 13:20:17 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:21:16 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 3.23333/s (194/min 798/5min 2408/15min);
[2021-03-26 13:21:16 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:21:16 +0100] information/IdoMysqlConnection: Pending queries: 5 (Input: 5/s; Output: 6/s)
[2021-03-26 13:21:36 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 1.05/s (63/min 229/5min 687/15min);
[2021-03-26 13:25:17 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:26:16 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:26:16 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 3.3/s (198/min 800/5min 2408/15min);
[2021-03-26 13:26:26 +0100] information/IdoMysqlConnection: Pending queries: 0 (Input: 4/s; Output: 4/s)
[2021-03-26 13:26:36 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.983333/s (59/min 227/5min 688/15min);
[2021-03-26 13:30:17 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:31:16 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 3.53333/s (212/min 800/5min 2398/15min);
[2021-03-26 13:31:16 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
root@icinga-qa:~>>tail -f n70 /var/log/icinga2/icinga2.log
tail: cannot open ‘n70’ for reading: No such file or directory
==> /var/log/icinga2/icinga2.log <==
[2021-03-26 13:47:06 +0100] information/IdoMysqlConnection: Pending queries: 11 (Input: 4/s;
 Output: 3/s)
[2021-03-26 13:49:16 +0100] information/IdoMysqlConnection: Pending queries: 13 (Input: 4/s;
 Output: 2/s)
[2021-03-26 13:50:17 +0100] information/ConfigObject: Dumping program state to file '/var/li
b/icinga2/icinga2.state'
[2021-03-26 13:51:45 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rat
e:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:51:45 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, ra
te: 3.16667/s (190/min 794/5min 2380/15min);
[2021-03-26 13:52:05 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, r
ate: 0.9/s (54/min 231/5min 687/15min);
[2021-03-26 13:52:06 +0100] information/IdoMysqlConnection: Pending queries: 0 (Input: 4/s; 
Output: 4/s)
[2021-03-26 13:55:17 +0100] information/ConfigObject: Dumping program state to file '/var/li
b/icinga2/icinga2.state'
[2021-03-26 13:56:55 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rat
e:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:56:55 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, ra
te: 3.36667/s (202/min 792/5min 2388/15min);
[2021-03-26 13:57:02 +0100] warning/JsonRpcConnection: API client disconnected for identity 
'icinga-qa-satellite-01.domain'
[2021-03-26 13:57:02 +0100] warning/ApiListener: Removing API client for endpoint 'icinga-qa
-satellite-01.domain'. 0 API clients left.
[2021-03-26 13:57:03 +0100] information/Application: Received request to shut down.
[2021-03-26 13:57:03 +0100] information/Application: Shutting down...
[2021-03-26 13:57:03 +0100] information/CheckerComponent: 'checker' stopped.
[2021-03-26 13:57:03 +0100] information/NotificationComponent: 'notification' stopped.
[2021-03-26 13:57:03 +0100] information/ExternalCommandListener: 'command' stopped.
[2021-03-26 13:57:03 +0100] information/PerfdataWriter: 'perfdata' paused.
[2021-03-26 13:57:03 +0100] information/InfluxdbWriter: 'influxdb' paused.
[2021-03-26 13:57:03 +0100] information/DbConnection: Pausing IDO connection: ido-mysql
[2021-03-26 13:57:03 +0100] information/IdoMysqlConnection: Disconnected from 'ido-mysql' da
tabase 'icinga_ido'.
[2021-03-26 13:57:03 +0100] information/IdoMysqlConnection: 'ido-mysql' paused.
[2021-03-26 13:57:03 +0100] information/DbConnection: 'ido-mysql' stopped.
[2021-03-26 13:57:03 +0100] information/ApiListener: 'api' stopped.
[2021-03-26 13:57:03 +0100] information/FileLogger: 'main-log' started.
[2021-03-26 13:57:03 +0100] information/FileLogger: 'debug-file' started.
[2021-03-26 13:57:03 +0100] information/ApiListener: 'api' started.
[2021-03-26 13:57:03 +0100] information/ApiListener: Copying 3 zone configuration files for 
zone 'global-templates' to '/var/lib/icinga2/api/zones/global-templates'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/i
cinga2/api/zones/global-templates//_etc/commands.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/i
cinga2/api/zones/global-templates//_etc/services.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/global-templates//_etc/templates.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Copying 3 zone configuration files for zone 'icinga-qa-satellite-01.domain' to '/var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/health.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/hosts.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/icinga-qa-satellite-01.domain//_etc/pet.conf
[2021-03-26 13:57:03 +0100] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/master/hosts/.bigdata.conf.swp' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
Context:
        (0) Creating config update for file '/etc/icinga2/zones.d/master/hosts/.bigdata.conf.swp'
        (1) Activating object 'api' of type 'ApiListener'

[2021-03-26 13:57:03 +0100] critical/ApiListener: Ignoring file '/etc/icinga2/zones.d/master/hosts/.netscaler.conf.swp' for cluster config sync: Does not contain valid UTF8. Binary files are not supported.
Context:
        (0) Creating config update for file '/etc/icinga2/zones.d/master/hosts/.netscaler.conf.swp'
        (1) Activating object 'api' of type 'ApiListener'

[2021-03-26 13:57:03 +0100] information/ApiListener: Copying 18 zone configuration files for zone 'master' to '/var/lib/icinga2/api/zones/master'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/arista.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/arista.conf.save
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/brunoTest.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/cluster.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/dc.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/dummy.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/hosts.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/icinga.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/moonshot.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/openstack.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/ops.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/parentstest.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/pets.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/proxy.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/switch.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/hosts/vsa.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Updating configuration file: /var/lib/icinga2/api/zones/master//_etc/services.conf
[2021-03-26 13:57:03 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-03-26 13:57:03 +0100] information/ExternalCommandListener: 'command' started.
[2021-03-26 13:57:03 +0100] warning/ExternalCommandListener: This feature is DEPRECATED and will be removed in future releases. Check the roadmap at https://github.com/Icinga/icinga2/milestones
Context:
        (0) Activating object 'command' of type 'ExternalCommandListener'

[2021-03-26 13:57:03 +0100] information/ApiListener: Reconnecting to endpoint 'icinga-qa-satellite-01.domain' via host '10.116.220.140' and port '5665'
[2021-03-26 13:57:03 +0100] information/DbConnection: 'ido-mysql' started.
[2021-03-26 13:57:03 +0100] information/PerfdataWriter: 'perfdata' resumed.
[2021-03-26 13:57:03 +0100] information/InfluxdbWriter: 'influxdb' resumed.
[2021-03-26 13:57:03 +0100] information/NotificationComponent: 'notification' started.
[2021-03-26 13:57:03 +0100] information/CheckerComponent: 'checker' started.
[2021-03-26 13:57:03 +0100] information/ConfigItem: Activated all objects.
[2021-03-26 13:57:03 +0100] information/IdoMysqlConnection: 'ido-mysql' resumed.
[2021-03-26 13:57:03 +0100] information/DbConnection: Resuming IDO connection: ido-mysql
[2021-03-26 13:57:03 +0100] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.14.3')
[2021-03-26 13:57:03 +0100] information/ApiListener: New client connection for identity 'icinga-qa-satellite-01.domain' to [10.116.220.140]:5665
[2021-03-26 13:57:03 +0100] information/ApiListener: Sending config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Syncing configuration files for global zone 'global-templates' to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Finished reconnecting to endpoint 'icinga-qa-satellite-01.domain' via host '10.116.220.140' and port '5665'
[2021-03-26 13:57:03 +0100] information/ApiListener: Syncing configuration files for zone 'icinga-qa-satellite-01.domain' to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/JsonRpcConnection: Received certificate request for CN 'icinga-qa-satellite-01.domain' signed by our CA.
[2021-03-26 13:57:03 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/JsonRpcConnection: The certificate for CN 'icinga-qa-satellite-01.domain' is valid and uptodate. Skipping automated renewal.
[2021-03-26 13:57:03 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Sending replay log for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/ApiListener: Finished syncing endpoint 'icinga-qa-satellite-01.domain' in zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:57:03 +0100] information/IdoMysqlConnection: Finished reconnecting to 'ido-mysql' database 'icinga_ido' in 0.112689 second(s).
[2021-03-26 13:57:08 +0100] information/ApiListener: New client connection for identity 'icinga-qa-satellite-01.domain' from [10.116.220.140]:56474
[2021-03-26 13:57:13 +0100] information/WorkQueue: #7 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:57:13 +0100] information/WorkQueue: #6 (ApiListener, RelayQueue) items: 0, rate: 1.16667/s (70/min 70/5min 70/15min);
[2021-03-26 13:57:13 +0100] information/WorkQueue: #8 (InfluxdbWriter, influxdb) items: 0, rate: 0.1/s (6/min 6/5min 6/15min);

lobr · March 26, 2021, 1:10pm

Satellite

[root@icinga-qa-satellite-01 features-enabled]# tail -f n70 /var/log/icinga2/icinga2.log
tail: cannot open ‘n70’ for reading: No such file or directory
==> /var/log/icinga2/icinga2.log <==
[2021-03-26 13:38:57 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:43:49 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:44:07 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:44:07 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:48:49 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:49:17 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:49:17 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:53:49 +0100] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-03-26 13:54:27 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:54:27 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:56:54 +0100] information/Application: Received request to shut down.
[2021-03-26 13:56:54 +0100] information/Application: Shutting down...
[2021-03-26 13:56:54 +0100] information/CheckerComponent: 'checker' stopped.
[2021-03-26 13:56:54 +0100] information/ApiListener: 'api' stopped.
[2021-03-26 13:56:54 +0100] information/FileLogger: 'main-log' started.
[2021-03-26 13:56:54 +0100] information/ApiListener: 'api' started.
[2021-03-26 13:56:54 +0100] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2021-03-26 13:56:54 +0100] information/CheckerComponent: 'checker' started.
[2021-03-26 13:56:54 +0100] information/ConfigItem: Activated all objects.
[2021-03-26 13:56:54 +0100] information/ApiListener: Reconnecting to endpoint 'icinga-qa.domain' via host 'icinga-qa.domain' and port '5665'
[2021-03-26 13:56:55 +0100] information/ApiListener: New client connection for identity 'icinga-qa.domain' from [10.116.219.2]:50366
[2021-03-26 13:56:55 +0100] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'icinga-qa.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Sending config updates for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished sending config file updates for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Syncing runtime objects to endpoint 'icinga-qa.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished syncing runtime objects to endpoint 'icinga-qa.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished sending runtime config updates for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Sending replay log for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished sending replay log for endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Finished syncing endpoint 'icinga-qa.domain' in zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Applying config update from endpoint 'icinga-qa.domain' of zone 'master'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Received configuration for zone 'global-templates' from endpoint 'icinga-qa.domain'. Comparing the timestamp and checksums.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/commands.conf' for zone 'global-templates'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/services.conf' for zone 'global-templates'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/templates.conf' for zone 'global-templates'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/global-templates' (39012 Bytes).
[2021-03-26 13:56:55 +0100] information/ApiListener: Received configuration for zone 'icinga-qa-satellite-01.domain' from endpoint 'icinga-qa.domain'. Comparing the timestamp and checksums.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain//_etc/health.conf' for zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain//_etc/hosts.conf' for zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain//_etc/pet.conf' for zone 'icinga-qa-satellite-01.domain'.
[2021-03-26 13:56:55 +0100] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/icinga-qa-satellite-01.domain' (1753 Bytes).
[2021-03-26 13:56:55 +0100] information/ApiListener: Received configuration updates (2) from endpoint 'icinga-qa.domain' are different to production, triggering validation and reload.
[2021-03-26 13:56:55 +0100] critical/ApiListener: Config validation failed for staged cluster config sync in '/var/lib/icinga2/api/zones-stage/'. Aborting. Logs: '/var/lib/icinga2/api/zones-stage//startup.log'
[2021-03-26 13:56:59 +0100] information/ApiListener: New client connection for identity 'icinga-qa.domain' to [10.116.219.2]:5665
[2021-03-26 13:56:59 +0100] information/ApiListener: Finished reconnecting to endpoint 'icinga-qa.domain' via host 'icinga-qa.domain' and port '5665'
[2021-03-26 13:57:04 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);
[2021-03-26 13:57:04 +0100] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate:  0/s (0/min 0/5min 0/15min);

After cleaning the satellite, the services still stay in “pending”.

rsx · March 26, 2021, 1:36pm

Here is the error:

[2021-03-26 13:56:55 +0100] critical/ApiListener: Config validation failed for staged cluster config sync in ‘/var/lib/icinga2/api/zones-stage/’. Aborting. Logs: ‘/var/lib/icinga2/api/zones-stage//startup.log’

Could you please share /var/lib/icinga2/api/zones-stage//startup.log?

Did you try the cleaning steps?

lobr · March 26, 2021, 1:41pm

Here is the startup.log from the satellite. On the master there isn’t any startup.log in this particular directory.

[2021-03-26 14:21:06 +0100] information/cli: Icinga application loader (version: 2.12.3)
[2021-03-26 14:21:06 +0100] information/cli: Loading configuration file(s).
[2021-03-26 14:21:06 +0100] information/ConfigItem: Committing config item(s).
[2021-03-26 14:21:06 +0100] information/ApiListener: My API identity: icinga-qa-satellite-01.domain
[2021-03-26 14:21:07 +0100] critical/config: Error: Error while evaluating expression: Tried to access undefined script variable 'api_token'
Location: in /var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf: 224:5-224:13
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(222):     PluginDir + "/ncpa_proxy_wrapper.sh",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(223):     "$address$",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(224):     api_token,
                                                                                ^^^^^^^^^
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(225):     "$remote_ncpa_metric$"
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(226):   ]

[2021-03-26 14:21:07 +0100] critical/config: Error: Error while evaluating expression: Tried to access undefined script variable 'api_token'
Location: in /var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf: 177:5-177:13
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(175):     PluginDir + "/ncpa_proxy_wrapper_q.sh",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(176):     "$address$",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(177):     api_token,
                                                                                ^^^^^^^^^
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(178):     "$remote_ncpa_metric$",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(179):     "$remote_ncpa_query$",

[2021-03-26 14:21:07 +0100] critical/config: Error: Error while evaluating expression: Tried to access undefined script variable 'api_token'
Location: in /var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf: 235:5-235:13
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(233):     PluginDir + "/ncpa_proxy_wrapper_w.sh",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(234):     "$address$",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(235):     api_token,
                                                                                ^^^^^^^^^
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(236):     "$remote_ncpa_metric$",
/var/lib/icinga2/api/zones-stage//global-templates/_etc/commands.conf(237):     "$remote_ncpa_warning$",

[2021-03-26 14:21:07 +0100] critical/config: 3 errors
[2021-03-26 14:21:07 +0100] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

Yes, I did try the clean up on the satellite.

rsx · March 26, 2021, 1:46pm

There is an error with /etc/icinga2/zones.d/global-templates/commands.conf. It looks like you have defined api_token at your master but not at your satellite. Hence, config validation on the satellite is failing.