Service to other host service dependency

staypekos · January 18, 2021, 5:56pm

Hi, we have icinga deployed in remote location which is directly connected over 192.168.255.0/30 network with the main location. I add the host interconnect with address 192.168.255.2 which is the main location side. The check for reachability and ping4 work good. Also there is no problem with host dependency of the hosts in main location on interconnect host.
But sometimes happens that the interconnect host is reachable, only the ping4 has to high RTA value and the service goes in critical (warning) state. In that moment also hosts in main location are UP, but their services goes critical (warning).

I try to set up service-to-service dependecy, but the notifications from services are stil sent.

First time I tried with general Apply rule (commented lines were tried uncommented)

apply Dependency "ping4-interconnect" to Service {
//parent_host_name = "interconnect"
  parent_service_name = "ping4"
  disable_checks = true
//ignore_soft_states = false
  disable_notifications = true
//states = [ OK ]
  assign where service.check_command == "ping4" //assign where host.vars.location == "main"
  ignore where service.name == "ping4-interconnect"
}

When this doesn’t work for me, I tried with basic dependecy object

object Dependency "ping4-interconnect" {
  parent_host_name = "interconnect"
  parent_service_name = "ping4"
  child_host_name = "testing.tacr.local"
  child_service_name = "ping4"
  states = [ OK ]
  disable_checks = true
//ignore_soft_states = false
}

But the result is still the same when testing. Service interconnect!ping4 goes critical, the noticifation is send, testing!ping4 goes critical and the notification is send.

Here is part of debuglog, I’m a bit confused about passed and filed dependecies

[2021-01-18 18:33:04 +0100] notice/Dependency: Dependency 'testing!ssh!disable-host-service-checks' passed: Parent host 'testing' matches state filter.
[2021-01-18 18:33:04 +0100] notice/Dependency: Dependency 'interconnect!ping4!disable-host-service-checks' passed: Parent host 'interconnect' matches state filter.
[2021-01-18 18:33:04 +0100] notice/Dependency: Dependency 'testing!ping4!ping4-interconnect' failed. Parent service 'interconnect!ping4' is CRITICAL
[2021-01-18 18:33:04 +0100] notice/Dependency: Dependency 'testing!ping4!disable-host-service-checks' passed: Parent host 'testing' matches state filter.
[2021-01-18 18:33:09 +0100] notice/Dependency: Dependency 'interconnect!ping4!disable-host-service-checks' passed: Parent host 'interconnect' matches state filter.
[2021-01-18 18:33:09 +0100] notice/Dependency: Dependency 'testing!ssh!disable-host-service-checks' passed: Parent host 'testing' matches state filter.
[2021-01-18 18:33:09 +0100] notice/Dependency: Dependency 'interconnect!ping4!disable-host-service-checks' passed: Parent host 'interconnect' matches state filter.
[2021-01-18 18:33:09 +0100] notice/Dependency: Dependency 'testing!ping4!ping4-interconnect' failed. Parent service 'interconnect!ping4' is CRITICAL
[2021-01-18 18:33:09 +0100] notice/Dependency: Dependency 'testing!ping4!disable-host-service-checks' passed: Parent host 'testing' matches state filter.

Is it even possible to make this type of dependency? If yes, have I forgot to set some parameter?
Thanks in advance

log1c · January 19, 2021, 2:31pm

To have a Service-to-Service dependency you need to set the name of the parent host and service. Otherwise the dependency will be treated as as “local” dependency for services to their own host (see https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#dependency @ “Service-to-Service-on-the-same-Host Dependency Example”)

So if you add the parent_host_name = “interconnect” to the apply rule, the config should work.

The second config should also work, if the host object is named “testing.tacr.local”.
The debuglog output let’s me think that the name is just “testing”.

staypekos · January 20, 2021, 11:13am

Thanks for the answer.
I already tried with parent_host_name = "interconnect" in apply rule, but it didn’t work.

Sorry for confusion, I removed the domain to shorter the log and forgot to remove it in shown config.

Could it possibly be the problem with some interference with host dependency?

apply Dependency "disable-host-service-checks" to Service {
  disable_checks = true
  ignore_soft_states = false
  assign where true
}

apply Dependency "interconnect" to Host {
  parent_host_name = "interconnect"
  assign where host.vars.location == "main"
}

log1c · January 20, 2021, 2:52pm

you need a ignore where host.name == "interconnect" for the host dependency.
Otherwise the dependency would be a loop on itself, I think.
Or does the host “interconnect” not have the variable location set to “main”?

staypekos · January 20, 2021, 2:55pm

As far as it’s network between both locations, the location variable is not set on this host.

log1c · January 21, 2021, 1:39pm

One thing I missed earlier:

Is the parent service named “ping4-interconnect” or just “ping4”?

Can you post your current dependency you are testing with the correct names, I am confused now

staypekos · January 22, 2021, 11:45am

Host interconnect has basic ping4 service. I named the dependency ping4-interconnect.

Here is the actual running configuration, I comment out other hosts and tried only with these two

object Host "interconnect" {
  import "generic-host"
  address = "192.168.255.2"
  retry_interval = 15s
}

object Host "testing.tacr.local" {
  import "generic-host"
  display_name = "testing"
  address = "10.10.25.223"
  vars.os = "Linux"
}

dependency definition

apply Dependency "disable-host-service-checks" to Service {
  disable_checks = true
  ignore_soft_states = false
  assign where true
}

apply Dependency "interconnect" to Host {
  parent_host_name = "interconnect"
  assign where host.name == "testing.tacr.local"
}

object Dependency "ping4-interconnect" {
  parent_host_name = "interconnect"
  parent_service_name = "ping4"
  child_host_name = "testing.tacr.local"
  child_service_name = "ping4"
  states = [ OK ]
  disable_checks = true
  ignore_soft_states = false
}

/* #alternative with apply rule
apply Dependency "ping4-interconnect" to Service {
  parent_host_name = "interconnect"
  parent_service_name = "ping4"
  states = [ OK ]
  disable_checks = true
  ignore_soft_states = false
  assign where service.check_command == "ping4"
  ignore where host.name == "interconnect"
}
*/

and the service

apply Service "ping4" {
  import "generic-service"
  check_command = "ping4"
  if (host.name == "interconnect") {
    max_check_attempts = 3
  }
  assign where host.address
}

log1c · January 22, 2021, 12:12pm

Apart from missing states in you host dependency apply rule, though I’m not sure if there is a default, all other dependecies are looking good to me and should work.

Just in case you have the commented apply rule like you posted in your config:
be aware that you need * in front of every line for a multi-line comment:

/* #alternative with apply rule
*apply Dependency "ping4-interconnect" to Service {
*  parent_host_name = "interconnect"
*  parent_service_name = "ping4"
*  states = [ OK ]
*  disable_checks = true
*  ignore_soft_states = false
*  assign where service.check_command == "ping4"
*  ignore where host.name == "interconnect"
*}
*/

staypekos · January 22, 2021, 1:34pm

Yes, it has default values, according to documentation
Configuration Attributes: States - Defaults to [ OK, Warning ] for services and [ Up ] for hosts.

I used

/*
_some_
_multiline_
_configuration_
*/

through all config files and there is no problem with validation and other host disapear from web GUI.
Anyvay I tried to remove the commented code from dependencies configuration but without change.

staypekos · January 22, 2021, 2:22pm

Ok, I probably found the problem. When I omit disable-host-service-checks dependency, everything works good. I think it’s because the ping4-interconnect dependency fails correctly, but after that disable-host-service-checks dependency check testing host, which is OK and the dependency passes, so notification is sent.

[2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'testing.tacr.local!interconnect' passed: Parent host 'interconnect' matches state filter.
[2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'interconnect!ping4!disable-host-service-checks' failed: Parent host 'interconnect' is in a soft state.
[2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'interconnect!ping4!disable-host-service-checks' passed: Parent host 'interconnect' matches state filter.
-> [2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'testing.tacr.local!ping4!ping4-interconnect' failed. Parent service 'interconnect!ping4' is CRITICAL
-> [2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'testing.tacr.local!ping4!disable-host-service-checks' failed: Parent host 'testing.tacr.local' is in a soft state.
-> [2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'testing.tacr.local!ping4!disable-host-service-checks' passed: Parent host 'testing.tacr.local' matches state filter.
[2021-01-22 14:54:51 +0100] notice/Dependency: Dependency 'testing.tacr.local!interconnect' passed: Parent host 'interconnect' matches state filter.

But I’m afraid, I cannot find solution with both dependencies

log1c · January 22, 2021, 2:33pm

good catch, I missed that.

Out of interest, why did you implement the “disable-host-service-checks” dependency?
There already is a default implicit dependency of hosts to their own services. So if a host is down no notification will be sent for the services of that host. The only difference is that the checks are still running.

Apart from that I don’t really know how to implement both without interefering with each other.
You could try setting the host check command of your “interconnect” host to something else than a ping check, so that the host can be UP, while the ping4 service is CRITICAL.

staypekos · January 22, 2021, 2:50pm

I received notifications when host went back to up state, but it’s services were still critical.
When no checks are running if the host is down, then in the time it’s back up, there is still time at least to one check on services (because they are in soft state) and they get ok.
If the check are done during the host down state, when the host is back up, all services are already in hard state and notifications are sent immediately.

I found this solution here

It’s not the problem. In this situation the host interconnect remains up, even the ping service is critical, because it’s still reachable - host reachability RTA values (C,W) 100.00 ms, 200.00 ms vs ping service 3.00 s, 5.00 s

log1c · January 25, 2021, 7:21am

Could you send the state history of the hosts/services (from the webinterface) at the time of the log messages of your previous post:

Maybe we both overlooked something.

staypekos · January 25, 2021, 3:00pm

log1c · January 26, 2021, 10:24am

I am missing the hosts states. those are more relevant atm, as the host dependency seems to be your problem (if I understood correctly)

Looking at this screenshot I don’t see anything wrong.

ping4 on interconnect is critical and sends a notification.
ping4 on testing is still in a soft state and did not send a notification.

staypekos · January 26, 2021, 2:36pm

I think it’s because the the hosts states didn’t changed.
Or I don’t know where to find them.

Yes, I think, I canceled the test, because the ping4 on testing has been still checked.
When I removed the disable-host-service-checks the ping4 on testing turned gray and was no more checked.

log1c · January 27, 2021, 7:48am

Both host and service have a “History” tab in the webinterface.
Service:

Host:

This would mean:
one screenshot per host history
one screenshot per service history.
four in total:

host interconnect
host testing
service ping4 on interconnect
service ping4 on testing

staypekos · January 27, 2021, 2:16pm

Ah, okey. I made new simulation (with the same result)

Host interconnect - no state change

Host testing - no state change

Service ping4 on interconnect

Service ping4 on testing

I attach debug_filtered.log (163.6 KB), from which lines with ApiListener, IdoMysqlConnection and DbConnection are removed.

staypekos · January 27, 2021, 2:33pm

Here is, how it looks like, when I remove the disable-host-service-checks dependency

The ping4 service on testing goes to Overdue state and no more checks are done (already on soft state of ping4 on interconnect)

Service ping4 on interconnect

Service ping4 on testing

log1c · January 27, 2021, 4:37pm

Thx for the screen shots.
It really looks like the disable-host-service-checks depedency somehow interferes with the whole setup.
But I don’t know why.

I found recent issuse that maybe have something to do with your problem, but I’m not sure.
Maybe check those if they look like they might fit.

Other than that I’m out of ideas.

Maybe someone you/of the team can have a “quick” look at this, @theFeu ^^?