I’m happy it worked out. But as long as you’re not using command_endpoint or sync the host config via the zones.d/agent-{01/02}.local directories your agents will never do any work. You can verify that looking at Check Source in Icinga Web 2.
That’s the way I was aiming for. The problem was, that your master doesn’t know about your agent zones. You can just add the zones and endpoints of your agents to your masters zones.conf and it will then also sync those directories.
The agents don’t belong to the sat-01 zone, they have their own zones which are just children of the sat-01 zone.
I’ll keep an eye on it as it is not consistent in it’s behavior since I’ve installed the release. If it persists I’ll open an issue.
Ok so just to be 100%, the /etc/icinga2/zones.d/agent-{01/02}.local directories are supposed to be on the MASTER, correct?
And where is this “check source” in Icinga Web 2? I cannot find it anywhere.
Thanks for the clarification. That makes sense. From this I evince that within the sat-01 directory I should never find any reference to any agent on any file within any satellite directory. Correct?
I just wanted to thank everyone who participated in this topic, as it was one of the most elaborate, productive and friendly exchanges I’ve seen in a bit.
This explains in a great detail how to read the documentation and set up the configuration the right way
Keep it up, stay safe and have a wonderful week,
Feu
I’ve applied a few changes following your instructions and it works.
Summerising:
added the following on zones.conf on master and satellite
I’m sure I’ve tried that before but the configuration on the sat-01 was still wrong so didn’t fix much and probably puppet removed that afterwards on the satellite and I didn’t notice :-@
object Endpoint “agent-02.local” {
}
object Endpoint “agent-01.local” {
}
object Zone “agent-02.local” {
endpoints = [ “agent-02.local” ]
}
object Zone “agent-01.local” {
endpoints = [ “agent-01.local” ]
}
removed the agents entries from sat-01 zone hosts.conf, created respective subdirs as you suggested:
object Host "agent-01.local" {
address = "X.X.X.X"
groups = [ linux-nodes, ]
display_name = "agent-01.local"
check_command = "hostalive"
vars.client_endpoint = name
zone = "master" //this trick specified in the manual fixed my agents from being permanently pending
}
(AGENT-02) same as agent-01.local just -02 hostname
modified services in sat-01 zone called sat-* as you suggested and fixed that too (only didn’t apply to the satellite as well but I’ll figure that out)
I think this is really it
Just a couple of things are still bothering me:
icinga2 object list --type host --name agent-01(02).local on the agents doesn’t give me any output now
icinga2 object list --type host --name agent-01(02).local on the master tells me on both: zone = “master”, I assume this is caused by the aforementioned trick to make the check done by the master zone
with such a need to “replicate” “object Endpoint” and “object Zone” on any zones.conf all the way through the agents (master → satellite) that file, on large deployments, may end up being very big. Won’t that affect performances in particular during startup and on the master which may have many satellites and agents registered in its file?
This still looks like the agents aren’t getting the config, which is kinda odd. Have a look at your agents logs, maybe they’re not accepting config or something like “we have an authoritative version of the zone’s config” again? Because without zone = “master” they should still not be pending, but checked locally by those agents. Which of course it not something you want with the hostalive command, but it should still work.
It will get big, yes. But in large environments zones and endpoints are a very small part of the problem. In environments with 250k+ host/services reloads can take a few minutes.
One think you can do to prevent this file getting messy, is to split it up in multiple files. But if you manage it with Puppet, it’s maybe not that important. You can just replace the content of zones.conf with something like include “zones.conf.d/*.conf” and put your zone/endpoint objects into multiple files in a zones.conf.d directory.
Did you leave those services in the zones.d/sat-01 directory? I’d suggest moving those to zones.d/global-templates, because your agents won’t get the config otherwise. Also make sure your services are checked by the right endpoint. The easiest way of doing this is by looking at the “Check Source” in Icinga Web 2. All your sat-* services should be checked by sat-01 or agent-01/02.
This is getting tough as in a fairly long career not many services/systems managed to make me feel stupid. Icinga2 just violently entered into that short list.
Anyway …
No, there is no “we have an authoritative version of the zone’s config” message in the logs.
In fact not much in the logs (right below).
I checked the agent and after an rm -rvf /var/lib/icinga2/api/{packages,zones,zones-stage}/*
I restarted agent-01.local and logs follows:
[2020-06-23 08:55:04 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-06-23 08:55:04 +0000] information/ConfigItem: Activated all objects.
[2020-06-23 08:55:04 +0000] information/ApiListener: Reconnecting to endpoint 'mon-02.local' via host 'X.X.X.X' and port '5665'
[2020-06-23 08:55:04 +0000] information/ApiListener: New client connection for identity 'mon-02.local' to [X.X.X.X]:5665
[2020-06-23 08:55:04 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'mon-02.local'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Sending config updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Finished sending config file updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Syncing runtime objects to endpoint 'mon-02.local'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'mon-02.local'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Sending replay log for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Finished sending replay log for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Finished syncing endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 08:55:04 +0000] information/ApiListener: Finished reconnecting to endpoint 'mon-02.local' via host 'X.X.X.X' and port '5665'
[2020-06-23 08:55:14 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2020-06-23 08:55:14 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
By the way I’ve noticed that Check Source value is mon-01.local(the master) for any node except for the satellite which has itself as source which is probably wrong. I’ve removed the zone = master to the hosts file of the agents (on the master) to verify the PENDING issue.
A quick glance on the Icingaweb2:
Check execution
Command hostalive Process check result
Check Source mon-01.local
Reachable yes
Last check 53m 24s ago (Check now)
Next check in -42m 46s Reschedule
Check attempts 1/3 (hard state)
Check execution time 4.12s
Check latency 0.001154s
Something looks wrong with the check and that’s why the pending thing. The only difference now is that it is not in PENDING (perhaps will become in few hours) but is UP but with a little clock icon.
Another difference is that if I do the check manually it will respond and the clock will go away. But it looks like there won’t be any automatic check at the end of the timer as the negative number keeps raising in value at the Next check field.
(ON THE MASTER)
All the nodes seem to be in the right zone (at least to me).
I feel like this is a secondary issue right now. I’d rather fix the core then I’ll spend time to tune the services on my own without bothering anyone here furthermore.
Yeah, Icinga can be quite complex and confusing at first, but very powerful and flexible when you understand it. You’re already dealing with one of the harder parts of Icinga, which means concepts aren’t getting much harder after that.
Okay, looking at your zone files helped a lot. You’re missing the parent = “sat-01” for your agent zones on your satellite and master. Without that those nodes don’t know that they have to sync config to them. That should fix your agents hosts being PENDING or overdue(little clock icon).
After fixing the issue mentioned above, you should end up with all 3 hosts being checked (Check Source) by their node. That means that all the syncing stuff finally worked. But you’ll still need to add the zone = "master" to make sure that the hostalive check isn’t executed locally on your agents/satellite.
Summary: After adding zone = "master", you should be able to see your hosts on your agents using icinga2 object list --type host and your check source should be your master. That will provide the right setup for your apply rules, so your services are finally checked locally on your agents.
And zone = master re-added on the agents hosts files. The only thing that changed is that the check was immediately done on the hosts and the little clock went away. So basically the condition got reverted to it’s previous state when zone = master was there.
Unfortunately there is still no output to the command on the agents: icinga2 object list --type host
only to the --type Zone variant and it follows for the agent-01.local
root@agent-01.local# icinga2 object list --type Zone
Object 'sat-01' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 19:1-19:27
* __name = "sat-01"
* endpoints = [ "mon-02.local" ]
% = modified in '/etc/icinga2/zones.conf', lines 20:3-20:52
* global = false
* name = "sat-01"
* package = "_etc"
* parent = ""
* source_location
* first_column = 1
* first_line = 19
* last_column = 27
* last_line = 19
* path = "/etc/icinga2/zones.conf"
* templates = [ "sat-01" ]
% = modified in '/etc/icinga2/zones.conf', lines 19:1-19:27
* type = "Zone"
* zone = ""
Object 'global-templates' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 15:1-15:30
* __name = "global-templates"
* endpoints = null
* global = true
% = modified in '/etc/icinga2/zones.conf', lines 16:3-16:15
* name = "global-templates"
* package = "_etc"
* parent = ""
* source_location
* first_column = 1
* first_line = 15
* last_column = 30
* last_line = 15
* path = "/etc/icinga2/zones.conf"
* templates = [ "global-templates" ]
% = modified in '/etc/icinga2/zones.conf', lines 15:1-15:30
* type = "Zone"
* zone = ""
Object 'agent-01.local' of type 'Zone':
% declared in '/etc/icinga2/zones.conf', lines 10:1-10:47
* __name = "agent-01.local"
* endpoints = [ "agent-01.local" ]
% = modified in '/etc/icinga2/zones.conf', lines 11:3-11:54
* global = false
* name = "agent-01.local"
* package = "_etc"
* parent = "sat-01"
% = modified in '/etc/icinga2/zones.conf', lines 12:3-12:26
* source_location
* first_column = 1
* first_line = 10
* last_column = 47
* last_line = 10
* path = "/etc/icinga2/zones.conf"
* templates = [ "agent-01.local" ]
% = modified in '/etc/icinga2/zones.conf', lines 10:1-10:47
* type = "Zone"
* zone = ""
BUT THEN
Out of desperation, I’ve tried to also add parent = "sat-01"
to the satellite zones.conf and it wasn’t very happy about that:
[2020-06-23 13:33:11 +0000] information/cli: Icinga application loader (version: r2.11.4-1)
[2020-06-23 13:33:11 +0000] information/cli: Loading configuration file(s).
[2020-06-23 13:33:11 +0000] information/ConfigItem: Committing config item(s).
[2020-06-23 13:33:11 +0000] information/ApiListener: My API identity: agent-01.local
[2020-06-23 13:33:12 +0000] critical/config: Error: Validation failed for object 'agent-01.local' of type 'Host'; Attribute 'zone': Object 'master' of type 'Zone' does not exist.
Location: in /var/lib/icinga2/api/zones-stage//agent-01.local/_etc/hosts.conf: 7:3-7:17
/var/lib/icinga2/api/zones-stage//agent-01.local/_etc/hosts.conf(5): display_name = "agent-01.local"
/var/lib/icinga2/api/zones-stage//agent-01.local/_etc/hosts.conf(6): check_command = "hostalive"
/var/lib/icinga2/api/zones-stage//agent-01.local/_etc/hosts.conf(7): zone = "master"
^^^^^^^^^^^^^^^
/var/lib/icinga2/api/zones-stage//agent-01.local/_etc/hosts.conf(8): vars.client_endpoint = name
/var/lib/icinga2/api/zones-stage//agent-01.local/_etc/hosts.conf(9): }
[2020-06-23 13:33:12 +0000] critical/config: 1 error
[2020-06-23 13:33:12 +0000] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
but then I’ve recognized that one as the entry (the trick) in the master.
Removed that from the master I got some real change:
[2020-06-23 13:37:01 +0000] information/ApiListener: Reconnecting to endpoint 'mon-02.local' via host 'X.X.X.X' and port '5665'
[2020-06-23 13:37:01 +0000] information/ApiListener: New client connection for identity 'mon-02.local' to [X.X.X.X]:5665
[2020-06-23 13:37:01 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'mon-02.local'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Sending config updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Finished sending config file updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Syncing runtime objects to endpoint 'mon-02.local'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'mon-02.local'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Sending replay log for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Finished sending replay log for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Finished syncing endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Finished reconnecting to endpoint 'mon-02.local' via host 'X.X.X.X' and port '5665'
[2020-06-23 13:37:01 +0000] information/ApiListener: Applying config update from endpoint 'mon-02.local' of zone 'sat-01'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Received configuration for zone 'agent-01.local' from endpoint 'mon-02.local'. Comparing the timestamp and checksums.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/agent-01.local//_etc/hosts.conf' for zone 'agent-01.local'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/agent-01.local' (362 Bytes).
[2020-06-23 13:37:01 +0000] information/ApiListener: Received configuration for zone 'global-templates' from endpoint 'mon-02.local'. Comparing the timestamp and checksums.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/commands.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/groups.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/notifications.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/services.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/templates.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/timeperiods.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/users.conf' for zone 'global-templates'.
[2020-06-23 13:37:01 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/global-templates' (11618 Bytes).
[2020-06-23 13:37:01 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'mon-02.local' are different to production, triggering validation and reload.
[2020-06-23 13:37:01 +0000] information/ApiListener: Config validation for stage '/var/lib/icinga2/api/zones-stage/' was OK, replacing into '/var/lib/icinga2/api/zones/' and triggering reload.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'agent-01.local//.checksums' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'agent-01.local//.timestamp' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'agent-01.local//_etc/hosts.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//.checksums' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//.timestamp' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/commands.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/groups.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/notifications.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/services.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/templates.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/timeperiods.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:01 +0000] information/ApiListener: Copying file 'global-templates//_etc/users.conf' from config sync staging to production zones directory.
[2020-06-23 13:37:02 +0000] information/Application: Received request to shut down.
[2020-06-23 13:37:02 +0000] information/Application: Shutting down...
[2020-06-23 13:37:02 +0000] information/ApiListener: 'api' stopped.
[2020-06-23 13:37:02 +0000] information/FileLogger: 'main-log' started.
[2020-06-23 13:37:02 +0000] information/ApiListener: 'api' started.
[2020-06-23 13:37:02 +0000] information/ApiListener: Started new listener on '[0.0.0.0]:5665'
[2020-06-23 13:37:02 +0000] information/ConfigItem: Activated all objects.
[2020-06-23 13:37:02 +0000] information/ApiListener: Reconnecting to endpoint 'mon-02.local' via host 'X.X.X.X' and port '5665'
[2020-06-23 13:37:02 +0000] information/ApiListener: New client connection for identity 'mon-02.local' to [X.X.X.X]:5665
[2020-06-23 13:37:02 +0000] information/ApiListener: Requesting new certificate for this Icinga instance from endpoint 'mon-02.local'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Sending config updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Finished sending config file updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Syncing runtime objects to endpoint 'mon-02.local'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Finished syncing runtime objects to endpoint 'mon-02.local'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Finished sending runtime config updates for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Sending replay log for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Finished sending replay log for endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Finished syncing endpoint 'mon-02.local' in zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Finished reconnecting to endpoint 'mon-02.local' via host 'X.X.X.X' and port '5665'
[2020-06-23 13:37:02 +0000] information/ApiListener: Applying config update from endpoint 'mon-02.local' of zone 'sat-01'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Received configuration for zone 'agent-01.local' from endpoint 'mon-02.local'. Comparing the timestamp and checksums.
[2020-06-23 13:37:02 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/agent-01.local'. Current timestamp '2020-06-23 13:36:44 +0000' (1592919404.360141) >= received timestamp '2020-06-23 13:36:44 +0000' (1592919404.360141).
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/agent-01.local//_etc/hosts.conf' for zone 'agent-01.local'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/agent-01.local' (362 Bytes).
[2020-06-23 13:37:02 +0000] information/ApiListener: Received configuration for zone 'global-templates' from endpoint 'mon-02.local'. Comparing the timestamp and checksums.
[2020-06-23 13:37:02 +0000] information/ApiListener: Our production configuration is more recent than the received configuration update. Ignoring configuration file update for path '/var/lib/icinga2/api/zones-stage/global-templates'. Current timestamp '2020-06-23 13:36:44 +0000' (1592919404.359671) >= received timestamp '2020-06-23 13:36:44 +0000' (1592919404.359671).
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/commands.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/groups.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/notifications.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/services.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/templates.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/timeperiods.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Stage: Updating received configuration file '/var/lib/icinga2/api/zones-stage/global-templates//_etc/users.conf' for zone 'global-templates'.
[2020-06-23 13:37:02 +0000] information/ApiListener: Applying configuration file update for path '/var/lib/icinga2/api/zones-stage/global-templates' (11618 Bytes).
[2020-06-23 13:37:02 +0000] information/ApiListener: Received configuration updates (2) from endpoint 'mon-02.local' do not qualify for production, not triggering reload.
[2020-06-23 13:37:12 +0000] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 0.1/s (6/min 6/5min 6/15min);
[2020-06-23 13:37:12 +0000] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
And that was a game changer (at least for the --type host command).
Now, for the first time I think, it works on the agent: root@agent-01.local# icinga2 object list --type host
but now I’ve got the “PENDING/CHECK DELAYED” issue back in place. Moreover:
it’s still the same.
So, to summerise, if I tell to the satellite that the agent nodes have SAT-01 as their parent zone(like is now set on the master) the agents finally understand who the heck they really are. But then I cannot use the zone = master trick to check them and, if I don’t, they do not get checked at all and the little clock shows up again.
If I revert this change I’ve done on the satellite the agents lose consciousness of themselves again.
I fix one thing and I screw up another. Damn it! : )
And I don’t get anything from any log which may suggest that there is some malfunctioning anywhere apart from the (somewhere previously mentioned) line in the master log which is probably nothing relevant but I post it anyway because I don’t have anything else to post.
2020-06-23 14:02:56 +0000] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 6, rate: 1.78333/s (107/min 567/5min 1508/15min); empty in infinite time, your task handler isn't able to keep up
[root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite
[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-agent1.localdomain.conf
object Endpoint "icinga2-agent1.localdomain" {
host = "192.168.56.111" // The satellite actively tries to connect to the agent
log_duration = 0 // Disable the replay log for command endpoint agents
}
object Zone "icinga2-agent1.localdomain" {
endpoints = [ "icinga2-agent1.localdomain" ]
parent = "satellite"
}
[root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite]# vim icinga2-agent2.localdomain.conf
object Endpoint "icinga2-agent2.localdomain" {
host = "192.168.56.112" // The satellite actively tries to connect to the agent
log_duration = 0 // Disable the replay log for command endpoint agents
}
object Zone "icinga2-agent2.localdomain" {
endpoints = [ "icinga2-agent2.localdomain" ]
parent = "satellite"
}
It is a clear instruction to edit an agent configuration icinga2-agent1.localdomain.conf within the satellite zone (which is its actual parent zone) /etc/icinga2/zones.d/satellite
instead, as you and Noah confirmed, it should be /etc/icinga2/zones.d/icinga2-agent1.localdomain/icinga2-agent1.localdomain.conf
Not that I would have fixed my issue (which I’m still working on) but If that information, misleading the least, wouldn’t be there I would have saved some time to myself and to you guys as well.
I’d suggest to review that.
I’ve referred to the missunderstanding with @nhilverling. Anyway, you need to place every zone and endpoint object in zones.conf only if you’re using V2.11.
I am using 2.11. But is there an error in the documentation?
I need to know otherwise I’m just trying all the possible combinations and nothing is fully working and that’s really not the way to troubleshoot anyway.
I’ve got configuration propagation but at the end checks works only if done manually, automatically don’t get triggered. So I still have the initial issue.
and this bloody error: Received configuration updates (2) from endpoint 'mon-02.local' do not qualify for production, not triggering reload.
How can I see in details those 2 updates which the agent is discarding?
(already tried: /usr/sbin/icinga2 daemon -x notice)
This is becoming too heavy to troubleshoot and the worthiness in terms of time/service is diminishing.
If I cannot rely on the documentation I’ll have to give up and go another way.
That’s actually nothing bad. It just means that the config is already up to date. The message is quite confusing and will be changed with version 2.12.
Both options can be used. If you configure your agents host objects in your satellite zone, you will have to use command_endpoint on services that should be executed locally on your agents. If you put those hosts in their own zones (agent1/2) you won’t need to do that.
I’m not aware of any big issues with the current documentation. It might still be confusing, because the docs are trying to explain both ways of configuring a cluster (config sync / command endpoint)
My configuration seems to be fully working now, the “check source” points to their respective parents and the config sync seems to be working very well.
My only problem is still the very first one which still persists: checks on agents works when triggered manually but otherwise don’t get done automatically.
The only services automatically checked (without the clocks) are the ones of mon-01 and mon-02 (master and satellite) All the agents are permanently stuck.
This is the typical detail of an agent service (taken today 29th of June):
Clearly shows that there is something wrong there.
QUICK UPDATE
I’ve noticed that relaunching all the checks manually works (as usual) but the date is still stuck on Jun 25 which is probably when the node was moved (for testing) into the satellite zone itself and then took back out on its own zone.
All agents are affected and, as usual, nor the master neither the satellite.
I finally have a working deployment. Don’t know if it’s of anyone’s interest but I may re-post my current configuration when I’ll have the time (I have more nodes now).
Anyway I think it’s important to say that one of the main issue still there which requires manual intervention resides on the puppet module which, when deploying new satellites, executes wrong/incomplete commands while proceeding with the keys/certificates initial retrieval.
(in the following puppet doesn’t put the --host argument for unknown reasons)
