Vspheredb - using the data for checks

Hello,

I am pretty new to Monitoring and i am a bit lost, so sorry in adavance if this has an obvious answer that i just didnt stumble uppon yet.
Following situation:
We are using VMware with 13 Hosts and over 400 Virtual Machines.
We have set up Icinga2 with the director and a few other things. We have also sucesfully implemented the vspheredb-Module.
We have sucesfully imported the Source and can see a lot of information under the “Virtualization (VMware)” Tab.
However, we have not set up any notifications/alarms yet. Is there a way to set up alarms for the data provided by the module itself?
Sure, the VMs are now available as hosts and we can assign different checks on them, but it is a bit complicated, given how we have a lot of different Operating systems, some checks require an installed agent, etc.
Which is of course fine when it comes to more complicated or specific checks, but since i can for example see the Memory Usage of a Virtual Machine under the “Virtualization (VMware)” tab. is there a simple way to turn this into a Check that would alarm us / send out a notification once it reaches a certain level?
Are there checks available for this sort of thing? can we create our own custom checks for this purpose ?(if so, can someone point me in the right direction?)
or will we have to go through the trouble of setting everything up manually form another source?
thank you in advance, i am sorry if this sounds confusing

You may want to checkout the Notification Apply Rules and Notification Templates in Icinga Director.

With that you can apply notifcation rules en masse to all of your vmware servers, or perhaps different notification rules for different servers based on a given property (such as host name)

The vspheredb module has it’s own (undocumented) check which can be run from the cli with icingacli vspheredb ...

# icingacli vspheredb
USAGE: icingacli vspheredb <command> [<action>] [options]

Available commands:

  check
  daemon
  health
  perf
  task
  vcenter

Show help on a specific command: icingacli help vspheredb <command>

# icingacli vspheredb check
vSphereDB Check Command
=======================

Available actions:

  datastore       Check Datastore Health
  datastores      Check all Datastores
  host            Check Host Health
  hosts           Check all Hosts
  vm              Check Virtual Machine Health
  vms             Check all Virtual Machines

Show help on a specific action: icingacli help vspheredb check <action>

check example for a specific host
'/usr/bin/icingacli' 'vspheredb' 'check' 'host' '--name' 'vmware-host-name' '--perfdata'
The --perfdata parameter should be available with the current master branch of the module.
It can only be used with the modes host, vm or datastore for specific objects, not the overall checks.

Here is the command definition

object CheckCommand "icingacli-vspheredb" {
    import "plugin-check-command"
    command = [ "/usr/bin/icingacli", "vspheredb", "check" ]
    timeout = 1m
    arguments += {
        "(no key)" = {
            order = 1
            required = true
            skip_key = true
            value = "$vspheredb_check_mode$"
        }
        "--name" = {
            order = 2
            required = false
            value = "$vspheredb_object_name$"
        }
        "--perfdata" = {
            order = 3
            required = false
            set_if = "$vspheredb_perfdata$"
        }
    }
}

With this you then can create services to apply to your host objects and then configure notifications for these checks.

thanks, that sounds like a step in the right direction, will definitly check it out

Hi @BananaWaffle ,

Did you manage to make it work?

I followed @log1c hint but without success, I get
vsphere
:expressionless:

Welcome @Fespinha

I use the director to automate the monitoring of the ESX hosts and this results in the following, as reportet by the inspect action on the service: '/usr/bin/icingacli' 'vspheredb' 'check' 'host' '--name' 'HOST-FQDN'

Hi @rivad, I managed to get it work, by adding the nagios user to the icingaweb2 group.

Btw is there any way to create a service check for used space on datastores?
Thanks

1 Like

@Fespinha There is a health check for datastores - https://github.com/Icinga/icingaweb2-module-vspheredb/blob/master/doc/31-Check_Commands.md#check-datastore-health.
I don’t think the used space is exposed yet but you can request this via issue on github.
If it’s urgent, the data is in the DB of the vSphereDB module so one could cobble together a little PHP/Python + SQL. This is a great task for a trainee if you have any :wink: of you can also buy the work from external programmers like the Linuxfabrik https://github.com/Linuxfabrik/monitoring-plugins

Hello together,

I just wanted to implement a check based on the icingacli commando but got a little confused.
The icingacli is only available on the icingaweb2 Server. But the Service check should be placed under the host in icinga2.

Question:
How do I manage to put the service underneath the Host but execute it only on the icingaweb2 server?
On a config file I would add command_endpoint to the service definition, how do I do this with the director?

My thoughts:
Do I need to create a zone for the icingaweb2 Server and set the zone of the service check to exact this? I may thinking about this in a way to complex way…

image
This will execute on the Icinga server not on the host object the service is attached to.
As far as I know there is no free choice only the icinga server and the host.
If the icinga server isn’t the icingaweb2 server then the service needs to be attached to a host object that corresponds to the icingaweb2 server and “Run on agent” needs to be set to “yes”.

@rivad Thank you for the replay.

My Setup have an separate icinga2-Master and an icingaweb2 Server.
Attaching the services of ~300 VM to one Host (icingaweb2) it’s maybe possible but not really an option.

Did someone try to run the icingacli on a satellite? Is this possible?

@pbirokas I run the icingacli on the 2 master nodes.
this way I can use the director to define the hosts but state to not run this specific check on the agent but on the master - as intended by the director.

Alternatively you could write a manual config for this service and use a apply rule triggered by custom vars managed by the director. This way you could use your icingaweb2 as command_endpoint and still use the director to do most of the host config minus this one service definition.

I’m not sure if you could fixate the command to your icingaweb2 server by zone. I guess it’s worth a try.
Until now I only used zones to not leak credentials all over the place.

Did someone try to run the icingacli on a satellite? Is this possible?

If you didn’t install icingaweb2 on the master, you already have it on a satellite/agent.

It seems to me that the information which hosts are in a system can be imported automatically with Director. Or the hosts have to be named each one by name in the configuration files.

What I am looking for - and the first post on this thread seemed to ask in that direction - is an automated way to import a new host in the vCenter which could be achieved by a restart or reload of the config every night. A piece of code which returns a list of host names to which services can be applied for people not using Director.

Is there such a thing?

By this you mean that you want your hosts in your vCenter to be represented as host objects in Icinga?
This is doable via the provided Import Source of the vspheredb module. With this you can import stuff* straight from the module into the Icinga configuration.

stuff:
image

The docs about the possible check commands have also been extended:
https://github.com/Icinga/icingaweb2-module-vspheredb/blob/master/doc/31-Check_Commands.md

Check if they are useful for you.

Yes, I do this but only for hosts where custom attributes are set in the vCenter for the host.
It makes not much sense for us, to import a host if we don’t have the host template and the team to be alerted :wink:

Also this way I don’t open the floodgates and get swamped by hosts that aren’t actionable and important in the monitoring context.

The last peace of the puzzle still missing is the post install and configuration of the agent on new virtual machines.