Agent checks on remote system

isaaclw · August 12, 2024, 1:43am

I have the following service check:

apply Service "ssh" {
  import "generic-service"
  check_command = "ssh"
  assign where (host.address || host.address6) && host.vars.os == "Linux"
}

Currently it sits in: /etc/icinga2/zones.d/global-templates/services.conf

But I would like to change the command_endpoint to just run on my master node. When I moved it to:
/etc/icinga2/zones.d/master/services.conf
And added command_endpoint = NodeName I got this error:

[2024-08-11 21:42:43 -0400] critical/config: Error: Validation failed for object 'tesla.lan!ssh' of type 'Service'; Attribute 'command_endpoint': Command endpoint must be in zone 'tesla.lan' or in a direct child zone thereof.
Location: in /etc/icinga2/zones.d/master/services.conf: 90:1-90:19
/etc/icinga2/zones.d/master/services.conf(88):  */
/etc/icinga2/zones.d/master/services.conf(89): 
/etc/icinga2/zones.d/master/services.conf(90): apply Service "ssh" {
                                               ^^^^^^^^^^^^^^^^^^^
/etc/icinga2/zones.d/master/services.conf(91):   import "generic-service"
/etc/icinga2/zones.d/master/services.conf(92):

My question is:

How do I perform most of my checks from the agent, but have a few checks that the master runs?

I would prefer to have ‘ping’, ‘ssh’, ‘http’ etc checks to come from outside where as ‘service’ checks and ‘disk’ checks should come from inside the host.

rsx · August 12, 2024, 6:10am

Storing object definitions in different directories does not change the where a check is executed. This is done within service objects. With command_endpoint configured it will be executed there, if not it will be executed on its parent.

jeanm · August 12, 2024, 7:49am

NB: We are using Director, so I will speak only for the case when Director is used.

In Service Apply Rules, you can’t set the command_endpoint. You have to create different Service Templates, one for each command_endpoint, and then create one Service Apply Rule per Service Template.

You can use the same name for all Service Apply Rules, BUT if you do so, you have to make sure you do not apply more than one service with the same name to a host. In other words, your “Assign where” clauses must be mutually exclusive for Apply Rules with the same name.

Hope this helps,

Jean

isaaclw · August 12, 2024, 2:30pm

I moved the check back to global-templates and left the command_endpoint = NodeName (which is the master) and it crashed with a similar error:

[2024-08-12 10:29:10 -0400] critical/config: Error: Validation failed for object 'tesla.lan!ssh' of type 'Service'; Attribute 'command_endpoint': Command endpoint must be in zone 'tesla.lan' or in a direct child zone thereof.
Location: in /etc/icinga2/zones.d/global-templates/services.conf: 59:1-59:19
/etc/icinga2/zones.d/global-templates/services.conf(57):  */
/etc/icinga2/zones.d/global-templates/services.conf(58): 
/etc/icinga2/zones.d/global-templates/services.conf(59): apply Service "ssh" {
                                                         ^^^^^^^^^^^^^^^^^^^
/etc/icinga2/zones.d/global-templates/services.conf(60):   import "generic-service"
/etc/icinga2/zones.d/global-templates/services.conf(61):

isaaclw · August 12, 2024, 2:32pm

Here is my ‘ssh’ service:

apply Service "ssh" {
  import "generic-service"

  check_command = "ssh"
  command_endpoint = NodeName                                        
                                                                     
  assign where (host.address || host.address6) && host.vars.os == "Linux"
}

rsx · August 13, 2024, 6:15am

Please remove this and be happy.

isaaclw · August 13, 2024, 1:45pm

I’m starting to worry people aren’t reading my question.

I removed the entry you mention, but it says it’s checking from itself.

I do not want Host1 to be checking Host1, I want master to be checking Host1:

Note the check source, it says ‘tesla.lan’ on the host: ‘tesla.lan’. it should have the path of my “master”

isaaclw · August 13, 2024, 2:07pm

I created a new host, I put it in the master zone. This host I don’t want to install an agent on it, so putting it in the master zone makes sense.

I was able to perform ssh and ping checks from the master.

Note: this is exactly what I want for the other hosts, but I want to also run the agent and do additional checks locally.

It seems to me that the answer to my initial question is: “It’s not possible” but I wanted to confirm with someone that actually knows if this is the case.

jeanm · August 13, 2024, 2:17pm

Hi,

I am sure this is possible

You should have
command_endpoint = null
in the “apply Service”.

Hope this will work,

Jean

isaaclw · August 13, 2024, 2:21pm

That makes sense to me, but as you can see here, it is still coming from the local agent:

 76 apply Service "ping4" {                        
 77   import "generic-service"                     
 78                                                
 79   check_command = "ping4"                      
 80   command_endpoint = null                      
 81                                                
 82   assign where host.address                    
 83 }                                              
/etc/icinga2/zones.d/global-templates/services.conf

isaaclw · August 13, 2024, 2:24pm

isaaclw:

[2024-08-12 10:29:10 -0400] critical/config: Error: Validation failed for object 'tesla.lan!ssh' of type 'Service'; Attribute 'command_endpoint': Command endpoint must be in zone 'tesla.lan' or in a direct child zone thereof.

This is the error I don’t understand, how it could be possible.

Just in case the wrapping is preventing other people from reading it:

Validation failed for object ‘tesla.lan!ssh’ of type ‘Service’; Attribute ‘command_endpoint’: Command endpoint must be in zone ‘tesla.lan’ or in a direct child zone thereof.

Master isn’t the direct child zone of tesla.lan, it’s the parent. which makes me think it’s not possible to have a parent host check a child host?

rsx · August 13, 2024, 2:34pm

Ok, this is my last answer to you.

Means, this host is in the wrong zone.

No, this is the correct setting by default. No matter if you are using agent or not.

Then you need to define services with

command_endpoint = host.name

Bye

isaaclw · August 13, 2024, 2:50pm

Good, then maybe I’ll get some responses from other people.

I want the host to check itself for disk checks. so no it must be in the right zone. It wouldn’t work otherwise. Unless I misunderstand.

and we’re back to the beginning, because that’s not what I want.

Here I explicitly set the zone I want my check to happen from:

apply Service "ssh" {                                                    
  import "generic-service"                                               
                                                                         
  check_command = "ssh"                                                  
  command_endpoint = "ohm.lan"                                         
                                                                         
  assign where (host.address || host.address6) && host.vars.os == "Linux"
}

And it gives me this error that I referenced before:

isaaclw · August 13, 2024, 4:40pm

Thank you for this comment. I’ve accepted this as the answer.

I think I have it working, if not I’ll post a new question on a fresh foot with my question.

I moved the host to the master zone, and now my other zone folders are empty (which confuses me a bit).

The endpoints are configured in the global template folder, so they don’t need to be moved…

I generally I really don’t understand why zone folders are used if I want everything on the master and global templates… I wonder if there’s any place I can read more about this.

rivad · August 14, 2024, 9:09am

Maybe this threads about satellites could help: Howto tell the Director to use a specific host for expensive checks? - #10 by rivad