Ubuntu monitoring via SSH

doboscs · June 23, 2023, 12:20pm

Hi,

I’m new in the icinga operation

I started with a simple check from icinga documentation to check if it’s working, but I’m facing a problem.

My dashboard tells:

Plugin Output

check_by_ssh: You must provide a host name

what can be the misstype or the problem?

Thanks in advance!

host.conf
template Host “ssh-agent” {
check_command = “hostalive”

vars.agent_type = “ssh”
vars.os_type = “linux”
}

object Host “zabbix” {
import “ssh-agent”

address = “10.102.36.118”
}

services.conf
apply Service “Disk Usage” {
import “generic-service”
check_command = “ssh-disk-usage”
vars.address = “10.102.36.118”
vars.user = “icinga”
vars.ssh_key = “/home/icinga/.ssh/id_rsa”
vars.disk_usage_w = “80%”
vars.disk_usage_c = “90%”
assign where host.vars.os_type == “linux” && host.vars.agent_type == “ssh”
}

commands.conf
object CheckCommand “ssh-disk-usage” {
import “plugin-check-command”
command = [“/usr/lib/nagios/plugins/check_by_ssh”]
arguments = {
“-H” = {
value = “$address$”
description = “Host address”
required = true
skip_key = true
}
“-C” = {
value = “‘/usr/lib/nagios/plugins/check_disk -w $disk_usage_w$ -c $disk_usage_c$ -p /’”
description = “Check command”
required = true
skip_key = true
}
“-l” = {
value = “$user$”
description = “SSH username”
required = true
skip_key = true
}
“-i” = {
value = “$ssh_key$”
description = “SSH private key path”
required = true
skip_key = true
}
}
}

rsx · June 23, 2023, 1:08pm

You need to remove skip_key = true.

BTW: Why don’t you use by_ssh directly?

doboscs · June 23, 2023, 8:27pm

Hi,

with by_ssh I got this message on icinga dashboard:
" Plugin Output

Remote command execution failed: Warning: Identity file /home/icinga/.ssh/id_rsa not accessible: Permission denied."

I checked the permissions on .ssh and id_rsa, id_rsa_pub and looks ok.
Capture

If I try the command from terminal:
/usr/lib/nagios/plugins/check_by_ssh -H 10.102.36.118 -l icinga -i /home/icinga/.ssh/id_rsa -C “/usr/lib/nagios/plugins/check_disk -w 80% -c 90%”

it works.

rsx · June 23, 2023, 8:36pm

Because you then run is as user root most properly. Switch to user icinga via sudo -u icinga bash and try it again.

doboscs · June 26, 2023, 11:48am

Hi,

I changed my technical user for icinga because something went wrong on the icinga user profile (when i wan’t to make a normal ssh connection the cursor still blinks and nothing happened).

Right now I set the user “icnmon” to make the check on the remote client (ubuntu 22.04 server), I made the necessary ssh key generation, and ssh-copy also but I’m facing this problem:

If I try the ssh connection with icnmon user from icinga server, the ssh login process works well, without password prompt.

I checked the .ssh/ permissions but I think that is also ok, I can’t find where is the problem right now…
Permissions on the client side:

Permissions on icinga server:

rsx · June 26, 2023, 11:53am

I’d assume your icinga services at the server are still running as user icinga. Hence, the plugin will be executed as user icinga as well. And this user has obviously not enough permissions to read the identity file.

doboscs · June 26, 2023, 1:15pm

I made a VM restore when the user icinga was ok, and I tryed now with the original settings, but:
" Plugin Output
Remote command execution failed: Host key verification failed."

I checked and from console I can ssh with my icinga user key, and if I run a check manually is also ok.

I checked the authorized_keys file and there is additional character…

Shortly:
manually from icinga user session I can ssh to remote host with my icinga user and the associated ssh key, the command also runs fine from icinga user, the only one error I got on icinga webview.

rsx · June 26, 2023, 1:27pm

What is the output of ps -ef | grep icinga2 at you icinga server?

doboscs · June 26, 2023, 1:34pm

here is the output:

for the “systemctl list-units --type=service --all | grep icinga2” command:

rsx · June 26, 2023, 1:54pm

Your icinga services are running as user nagios (which is default for Debian and Ubuntu) and you have to configure the ssh connection between nagios on server and any user on target.

doboscs · June 26, 2023, 2:05pm

Thanks for your remark I hope this will the last issue
I will try with nagios user, and I will come back with a feedback.

doboscs · June 27, 2023, 9:03am

Today I created the ssh key for nagios user, and I tested now on the client side I can see the success ssh key login, so this topic is ok

But on the icinga web dashboard I got this plugin output for check_disk (right now this is the only check configured)

commands2.conf

services2.conf

hosts2.conf

The other checks (cpu, mem, load, proc, ssh, uptime) are ok.

rsx · June 27, 2023, 10:42am

You define by_ssh_arguments for the service object which are not used in your command object. Hence, check_disk is executed without any arguments and this result in the output.

doboscs · June 27, 2023, 10:46am

In the time I solved the problem
now my services.conf looks in this way:

Thanks for your help!

doboscs · June 28, 2023, 9:55am

Regarding an ESXI 8 customized version, here I am using the root account for the ssh from the icinga server what is working great, but the checks no, I got this message for every check:

Plugin Output

Remote command execution failed: sh: /usr/lib/nagios/plugins/check_memory: Operation not permitted

Plugin Output

Remote command execution failed: sh: /usr/lib/nagios/plugins/check_uptime: not found

If i run the command from icinga server it’s ok:

The content from icinga server is copyed with scp to the client to the same folder structure, the user is root but the on the folder the permissions are 755, what can be the problem?

My hosts.conf where I set to use the root user for checks:
object Host “esxi-1” {
import “generic-host”
address = “10.102.36.11”
vars.ssh_username = “root”
vars.ssh_private_key = “/root/.ssh/id_rsa”
check_command = “hostalive”
vars.os = “Linux”
groups = [ “esxi” ]
vars.notification[“mail”] = {
groups = [ “icingaadmins” ]
}

}

doboscs · June 28, 2023, 12:06pm

finally I can get in to the esxi server with nagios user passwordless, but the service checks status on the icinga dashboard not changed…

Here is the aut.log from the esxi server I can see the success login and the launch of the scripts (the scripts are there at the usual location)

rsx · June 28, 2023, 12:18pm

Running check plugins on an ESXi host locally does not make much sense. Better use something like esxi_hardware.

doboscs · June 28, 2023, 1:54pm

I want to see only the cpu, disk, memory, uptime, load values and I want to use for this the most simpliest solution, for this purpose also the esxi_hardware is the best?

Thanks!

doboscs · June 29, 2023, 9:51am

I comfigured the esxi_hardware check and it’s working only with root user, if I use the nagios user on the esxi server where this user have Read-Only permission I received the permission denied status, do you have mor einformation about the necessary user permission for this check?

I don’t want to grant a root privilege for a checking user…