/var/lib/icinga2/api/zones/sat1/director/hosts.conf
hosts listed in this file and hosts was checked by sat because icmp checks are lower then from masters.
connectiom from sat1 to both masters works (tested with telnet master1 5665)
how can i check api event stream?
ok found something, when i stop icinga2 on master2 then only graphs will be updated and host is visible in icingaweb2.
https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#checks-are-not-executed
That means that the IDO feature switches back to master1, and this endpoint knows about the host also being the config master.
So your next step is to find out why the secondary master2 doesn’t sync the configuration. Inspect the logs when such a sync happens, and check specifically for the sat1 zone.
https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#cluster-troubleshooting-config-sync
ok i checked https://icinga.com/docs/icinga2/latest/doc/15-troubleshooting/#cluster-troubleshooting-config-sync
when i stop master2 i got
critical/ApiListener: Cannot connect to host ‘master2’ on port ‘5665’
seems that sat1 try to connect to master2 and then master2 is running then i got:
Oct 25 13:04:25 sat1 icinga2[32534]: [2019-10-25 15:04:25 +0200] information/ApiListener: Finished sending replay log for endpoint ‘master2’ in zone ‘master’.
Oct 25 13:04:25 sat1 icinga2[32534]: [2019-10-25 15:04:25 +0200] information/ApiListener: Finished syncing endpoint ‘master2’ in zone ‘master’.
problem with icinga doc is not clear where to exectue on master(s) or sat endpoint
in debug log on master2:
notice/JsonRpcConnection: Received ‘event::Heartbeat’ message from identity ‘sat1’.
Specifically I’m looking for lines like this (example from a test for 2.11.2).
[2019-10-22 13:46:16 +0200] information/ApiListener: New client connection for identity 'master2' from [::ffff:127.0.0.1]:51946
[2019-10-22 13:46:16 +0200] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'master2'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Sending config updates for endpoint 'master2' in zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Finished sending config file updates for endpoint 'master2' in zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Syncing runtime objects to endpoint 'master2'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Finished syncing runtime objects to endpoint 'master2'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Finished sending runtime config updates for endpoint 'master2' in zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Sending replay log for endpoint 'master2' in zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Finished sending replay log for endpoint 'master2' in zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Finished syncing endpoint 'master2' in zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Applying config update from endpoint 'master2' of zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master2'. Comparing the checksums.
[2019-10-22 13:46:16 +0200] information/ApiListener: Stage: Updating received configuration file 'var-c/lib/icinga2/api/zones-stage/agent//_etc/host.conf' for zone 'agent'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Applying configuration file update for path 'var-c/lib/icinga2/api/zones-stage/agent' (220 Bytes).
[2019-10-22 13:46:16 +0200] information/ApiListener: Received configuration updates (1) from endpoint 'master2' are different to production, triggering validation and reload.
[2019-10-22 13:46:17 +0200] information/ApiListener: Config validation for stage 'var-c/lib/icinga2/api/zones-stage/' was OK, replacing into 'var-c/lib/icinga2/api/zones/' and triggering reload.
[2019-10-22 13:46:17 +0200] information/ApiListener: Copying file 'agent//.checksums' from config sync staging to production zones directory.
[2019-10-22 13:46:17 +0200] information/ApiListener: Copying file 'agent//.timestamp' from config sync staging to production zones directory.
[2019-10-22 13:46:17 +0200] information/ApiListener: Copying file 'agent//_etc/host.conf' from config sync staging to production zones directory.
[2019-10-22 13:46:18 +0200] information/Application: Got reload command: Starting new instance.
In your example, look for sat1 on master2 in the logs. Also, to again get a better picture, please show the zones.conf content from master2.
[2019-10-25 17:21:06 +0200] notice/ApiListener: Sending message ‘event::CheckResult’ to ‘master1’
[2019-10-25 17:21:06 +0200] notice/ApiListener: Relaying ‘event::SetNextCheck’ message
[2019-10-25 17:21:06 +0200] notice/ApiListener: Relaying ‘event::SetNextCheck’ message
[2019-10-25 17:21:06 +0200] notice/ApiListener: Relaying ‘event::CheckResult’ message
[2019-10-25 17:21:06 +0200] debug/ApiListener: Not connecting to Endpoint ‘sat1’ because we’re already connected to it.
[2019-10-25 17:21:06 +0200] debug/ApiListener: Not connecting to Endpoint ‘master1’ because we’re already connected to it.
[2019-10-25 17:21:06 +0200] debug/ApiListener: Not connecting to Endpoint ‘master2’ because that’s us.
[2019-10-25 17:21:06 +0200] notice/ApiListener: Current zone master: master1
[2019-10-25 17:21:06 +0200] notice/ApiListener: Connected endpoints: sat1 (1) and master1 (1)
[2019-10-25 17:21:06 +0200] notice/ApiListener: Updating object authority for objects at endpoint ‘master2’.
[2019-10-25 17:21:06 +0200] notice/ApiListener: Setting log position for identity ‘sat1’: 2019/10/25 17:20:47
[2019-10-25 17:21:06 +0200] notice/ApiListener: Setting log position for identity ‘master1’: 2019/10/25 17:21:06
[2019-10-25 17:21:06 +0200] notice/ApiListener: Relaying ‘event::SetNextCheck’ message
[2019-10-25 17:21:06 +0200] notice/ApiListener: Sending message ‘event::SetNextCheck’ to ‘master1’
zones.conf on master2:
object Endpoint “master1” {
host = “master1”
}
object Endpoint “master2” {
}object Zone “master” {
endpoints = [ “master1”,“master2” ]
}object Zone “global-templates” {
global = true
}object Zone “director-global” {
global = true
}object Endpoint “sat1” {
host = “sat1”
}
object Zone “sat1” {
endpoints = [ “sat1” ]
parent = “master”
}
That’s impossible that these log lines do not exist after you restart master2 with a config sync (or trigger one via director deployment).
How exactly are you searching the logs? cat, grep, less, awk or something else? Which file are you looking at?
icinga2 feature enable debuglog
service icinga2 restart
tail -f /var/log/icinga2/debug.log|grep ApiListener
why should these lines not exist?
Maybe your terminal buffer is limited and you only see the last 10 lines.
Especially this starts:
[2019-10-22 13:46:16 +0200] information/ApiListener: Applying config update from endpoint 'master2' of zone 'master'.
[2019-10-22 13:46:16 +0200] information/ApiListener: Received configuration for zone 'agent' from endpoint 'master2'. Comparing the checksums.
Hint from my daily work: I don’t use tail -f, but less.
less /var/log/icinga2/debug.log
/ApiListener allows me to search, SHIFT+F starts a live stream of the files.
If I really want to filter this:
less /var/log/icinga2/debug.log | grep -i Apilistener | less
Have a nice weekend,
Michael
Hi Jafi,
I have the same issue (two Masters in HA) and a satellite in a different zone is not visible on secondary Master that is Active Enpoint from icingaweb2 view. Logs and debug logs seems OK byt icinga2 object list --type host --name givensatelite shows nothing on Master2.
What only work is to put the satellite into master zone, but it is not goal…
Have you somehow solved that situation?
Thank you
Jan