I pretty much guess I already know the answer to the title question already (hint: they are), but I’ll still explain what we are trying to do
We have numerous checks against the SentinelOne API. These checks run on hosts in many different zones.
As SentinelOne might “randomly” decide when they will have a maintenance and therefore the API is not reachable, we decided that a dependency is our best option to suppress notifications.
For that I created a health check for the API that is run in “zone A”.
All the other checks using the API are run in “zone A” to “zone Z”
Now the problem is that the dependency causes the config validation to fail on the satellites of the various zones (This happens silently I might add, we are investigating that) and the new configuration isn’t activated.
Message is
Error: Validation failed for object 'zoneX-mgmt02!service-s1-fanotify!generic-sentinelone-dependency' of type 'Dependency'; Attribute 'parent_host_name': Object 'SentinelOne_Mgmt-Console' of type 'Host' does not exist.
Dependency config
zones.d/director-global/dependency_apply.conf
apply Dependency "generic-sentinelone-dependency" to Service {
import "sentinelone_agent_health-dependency"
parent_host_name = "SentinelOne_Mgmt-Console"
parent_service_name = "s1_mgmt-console_status"
assign where match("service-s1-*", service.name) || match("service-sentinel*", service.name) || match("sentinelone*", service.name)
states = [ OK ]
}
Parent host configuration:
zones.d/zoneA-azure/hosts.conf
object Host "SentinelOne_Mgmt-Console" {
import "zoneA-azure-host-template"
import "dummy_hosts"
address = "the API url of sentinelone"
vars.notification_period = "none"
}
Is there a possibility to get this setup running?
It seems like the dependency objects are created in the respective zone of the host and not in the global zone.
Do the dependency objects need to exist outside of the master zone?
In my case the masters are the only ones that do the notifying and if I’m correct the master needs to know all objects or it will discard information form satellites and agents about the object.
If you have multiple zones (like we do), objects are a “member” of a specific zone. And each zone only has knowledge about its own objects.
The master knows about all objects because it holds the entire config.
And that is the problem here, because the parent host is inside the master zone, which the objects of the other zones don’t know about.
Ping @theFeu. Is there any hope that I can get this config working^^? Maybe someone of the Icinga core team could share an insight.
No idea how to do the following in Director, but as a static config user I’d try to move the parent object definition to the child zone, but setting zone=“master”. That way I could move the Dependency to the child zone as well.
thanks for the suggestion.
Never knew I could create a host object in a child zone and pin it to a different zone.
Apart from that, wouldn’t I then have to do that for each child zone (currently around 15)?
I would have liked to have one host object that acts as a parent and the dependency objects not inside the different satellite zones, so they don’t care that the parent isn’t in “their” zone.
Would putting the dependency config in the master zone work, because the master should know about all objects?
That would be the nicer option, have you tried it out?
Oh, and while you’re on it do zone=“master” in the apply rule.
Yes. zone=“master” in an apply rule already being in a file of the master zone.
Well,
This actually works. I’m a bit surprised to be honest
Had the dependency config in the master zone once before, but without the zone = "master".
That was the last bit that made it work.
apply Dependency "generic-sentinelone-dependency" to Service {
import "sentinelone_agent_health-dependency"
parent_host_name = "SentinelOne_Mgmt-Console"
parent_service_name = "s1_mgmt-console_status"
states = [ OK ]
zone = "master"
assign where match("service-s1-*", service.name) || match("service-sentinel*", service.name) || match("sentinelone*", service.name)
}
And this would make it possible to put zone = "whatever" into a config that is sitting in a different zone than “whatever”, correct?
Due to a problem with our HA-Cluster (continuous sync loop) , which resulted in the need to shut down one of the nodes, we did some debugging and stumbled upon a “curious” behavior.
After I activated the dependency config above the IDO mysql-queries increased about 2x-3x to ~150-200k per 15min
Hi all, I would like to expand on this topic because we have a use case which requires that a dependency works for hosts which are in different child zones not the master zone. So basically we have a distributed Icinga cluster with 3 zones. Each of these zones contains a host with 2 services replicalag1 and replicalag2 which check the state of replication of a generic postgres cluster. What we are trying to achieve is that an alert should be notified only if at least one of the replicalag1 services is in error and a corresponding replicalag2 service is also in error on at least one of the slave postgres nodes. Also we only schedule a replicalag2 service check if one of the replicalag1 services is in error.
Now using the latest icinga-director 1.11 we have most of this working. The only problem we are experiencing is that the config validation fails because each host is in a different zone so it complains that it can’t find the host in the other zones.
I think for dependencies to make sense there must be some way to allow global visibility of hosts otherwise the service to host or host to host aspect looses a lot of potential. Has anyone got something similar working? We are using a host custom variable to pass the nodes of the postgres cluster for the dependency apply rule. So each of the three hosts references at least a node which is in another zone.
Hope something can work.
