Applying commands/command template to hosts, e.g., check_https or check_http --ssl

Perhaps I am misunderstanding the differences. I want to simply:

  • send an email alert when disk space on various partitions over a few servers reach a certain threshold e.g. ,<+ 8%
  • send an email when server load exceeds a threshold, e.g., >-5 over a 10 minute time span
  • send an email when sites we have running on ports 443 and another on 8443.

Is there an existing template I can use for this?

Ok, so you seem to be fairly new to the icinga2 (or monitoring) world and currently have little understanding of the system. No disrespect/blaming here, just an observation :slight_smile:

I’ll try to give you a small run-down of how this would be configured, but I would suggest you read the docs chapter about the monitoring basics to get better understanding of the different config objects:

So, to have a check you would have to do the following:
(I will just list the steps, not what exactly to configure)

  1. Configure a Plugin check command.
    – This is the script (maybe with some arguments) that is executed when the check is run.
  2. Create a Service template that uses the command.
    – Service templates can define check execution parameters that will be given down to the service objects
  3. Define a Service (via Apply Rule, a Service Set or as a single object) which imports the template
    – this defines the check that will, be executed. It will inherit options set in the template and here you would set other information for the check parameters into fields (like the http_port, or some thresholds)

With this you than have a check (or maybe multiple checks) that monitor what you configured (e.g disk space)

The notifications basically have the same structure.
You start with a Notification Command, create a Notification template to define some options (interval, states, types, user…) and then create a Notification apply rule to hosts or services.

Well kind of. I was doing OK with Icinga2 but adding Director added a level of complexity that I didn’t expect.

And do appreciate you and others that respond here as I go through the leaning process…

Hm, yeah. The Director has it’s pros and cons/limitations. And starting to use it has its challenges when coming from the config files. But the other way round has its challenges as well :smiley:

But I like to think that it enforces some best practices (like the need to create a service template first before creating a service) which could (but imo shouldn’t) be bypassed when using the config files directly.

I created this command template and used check_http with --ssl but I still see this error:

HTTP WARNING: HTTP/1.1 400 Bad Request - 452 bytes in 0.012 second response time

'/usr/lib64/nagios/plugins/check_http' '-f' 'ok' '--sni' '--ssl' '-H' '%hostname%' '-I' '' '-S' '-p' '443'

What am I doing wrong?

Edit: I see that the -H option was missing after adding that, this started working.

So the last item is Notifications. I had them working fine before adding Director. Is my Assign where correct?

warning/ApplyRule: Apply rule 'Icinga notification of possible issue' (in /var/lib/icinga2/api/packages/director/5878157f-a0b9-4b64-982b-988cd6e25878/zones.d/master/notification_apply.conf: 1:0-1:65) for type 'Notification' does not match anywhere!

cat /var/lib/icinga2/api/packages/director/xx/zones.d/master/notification_apply.conf
apply Notification "Icinga notification of possible issue" to Host {
    import "Icinga notification on a possible server issue"

    interval = 0s
    assign where host.enable_notifications == "true"
    users = [ "icinga-director" ]

host.enable_notifications is a boolean, can you try this syntax?

Ugh I added a host by using icinga2 node wizard and now have a mess. When trying to deploy I get:
* Unable to detect your Icinga 2 Core version (DeployFormsBug7530.php:72)

icingaweb2[3189]: Icinga\Exception\NotFoundError in /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php:642 with message: Failed to load icinga_host ""    
#0 /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php(1173): Icinga\Module\Director\Data\Db\DbObject->loadFromDb()    
#1 /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObject.php(2595): Icinga\Module\Director\Data\Db\DbObject::load()    
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(462): Icinga\Module\Director\Objects\IcingaObject::loadByType()    
#3 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(445): Icinga\Module\Director\Web\Controller\ObjectController->loadObject()    
#4 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(78): Icinga\Module\Director\Web\Controller\ObjectController->eventuallyLoadObject()    
#5 /usr/share/php/Icinga/Web/Controller/ActionController.php(165): Icinga\Module\Director\Web\Controller\ObjectController->init()    
#6 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()    
#7 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()    
#8 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()    
#9 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()    
#10 /usr/share/icingaweb2/public/index.php(4): require_once(String)    
#11 {main}


Any idea what I should do? Reinstall Director?

Edit: well I uninstalled and reinstalled Director. Now I can’t kickstart it: * CURL ERROR: Failed to connect to port 5665: Connection refused (RestApiClient.php:143)

firewalld is not running. Just to be sure I started firewalld and added an exception for port 5665. I even rebooted the server. netstat -plnt|grep 5665 brings back nothing. I see on all of the hosts I’m monitoring that nestat returns a value:

netstat -plnt|grep 5665
tcp6       0      0 :::5665                 :::*                    LISTEN      1139109/icinga2

It’s as if the icinga2 service on the master server is not opening a port and I’ve restarted both the director and icinga2 services.

I do see these warnings:

[2021-10-28 16:31:29 -0400] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/director-global' for unknown zone 'director-global'.
[2021-10-28 16:31:29 -0400] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/master' for unknown zone 'master'.

So there is something misconfigured with the API. I did run icinga2 api setup to no avail only the afore-mentioned warnings…

Update: Running icinga2 node wizard and answering ‘yes’ for the master fixed issues allowing me to import Director.

The icinga2 node wizard command is used to setup the configuration for a satellite/agent OR the master. It creates the certificates(+CA if master), enables the API, and asks for connection details to the parent (if satellite/agent).
It is not for adding/configuring host objects for the monitoring.
That you do via the config files or the Director.

In Service Problems I see this error, any idea what it might mean?

#0 /usr/share/php/Icinga/Exception/IcingaException.php(41): ReflectionClass->newInstanceArgs()
#1 /usr/share/php/Icinga/Web/Controller.php(87): Icinga\Exception\IcingaException::create()
#2 /usr/share/icingaweb2/modules/monitoring/application/controllers/ServiceController.php(38): Icinga\Web\Controller->httpNotFound()
#3 /usr/share/php/Icinga/Web/Controller/ActionController.php(165): Icinga\Module\Monitoring\Controllers\ServiceController->init()
#4 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()
#5 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#6 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()
#7 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()
#8 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#9 {main}

Edit: OK error seems to go away. I believe I was using a cached URL so perhaps this could be an enhancement or better warning/error message?

Looks like I found a bug. I followed this tutorial,

[2021-10-29 15:14:39 -0400] notice/ApiListener: Connected endpoints: (1)
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2021-10-29 15:14:39 -0400] notice/ApiListener: Updating object authority for objects at endpoint ''.
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '3')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '3')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '1')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '1')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: COMMIT
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: BEGIN
[2021-10-29 15:14:39 -0400] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity '': Error: End of file

 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# 0x00000000005A278D in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x000000000085B138 in /usr/lib64/icinga2/sbin/icinga2
 5# 0x000000000088DC8C in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/
[2021-10-29 15:14:39 -0400] warning/JsonRpcConnection: API client disconnected for identity ''
[2021-10-29 15:14:39 -0400] warning/ApiListener: Removing API client for endpoint ''. 0 API clients left.
[2021-10-29 15:14:39 -0400] debug/EndpointDbObject: update is_connected=0 for endpoint ''
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: UPDATE icinga_endpointstatus SET is_connected = '0',  status_update_time = FROM_UNIXTIME(1635534879) WHERE endpoint_object_id = 294 AND instance_id = 1
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: COMMIT
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: BEGIN

From the client logs:

[2021-10-29 16:31:22 -0400] warning/JsonRpcConnection: API client disconnected for identity ''
[2021-10-29 16:31:32 -0400] information/ApiListener: New client connection for identity '' from [::ffff:x.x.x.x]:51464 (certificate validation failed: code 18: self signed certificate)
[2021-10-29 16:31:42 -0400] information/JsonRpcConnection: Closing anonymous connection [::ffff:]:51464 after 10 seconds.
[2021-10-29 16:31:42 -0400] warning/JsonRpcConnection: API client disconnected for identity ''

Update: Hosts are at least showing as Up now how to get the Services going…


These errors remain on all nodes:
warning/ApiListener: Certificate validation failed for endpoint '': code 18: self signed certificate

OK well the self signed certificate message was a clue.

First icinga2 ca list brought back nothing:

icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject


So I ran pki sign-csr --csr --cert /var/lib/icinga2/certs/ca.crt and scp’d the files to the agent’s /var/lib/icinga2/certs directory and restarted icinga2. Works now.

One (hopefully) last question, from the tutorial, I used, it does not have a Command or Command Template. So how do I use check_http and add arguments like --ssl? Everything is based on a Service Set and Service Template.

The tutorial does not show the command creation/configuration step because they are using a already defined command from the ITL. Those are mostly already good to go, as they often have some default values set.
If you want to fit them to your needs I suggest the following:
Go to the (in your case) http command, switch to the fields tab and add all the fields you need for your check. You will get a drop down menu there that will give you the fields of that command under argument macros

The description what each field is for can be found in the Preview tab.
You will find that, for example, the field for -S/–ssl is not present in the drop-down list(as are some others). This is because this field is not assigned to value but to set_if, meaning it is a boolean field not a field that expects a string.
For those fields you need to create the data field with the name http_ssl and type boolean by hand and can then add it via the drop-down list in the fields tab of the command (it will be listed further down the list in alphabetical order and not under the argument macros


Is this under Director and then Commands?


What I’m trying to also achieve is to use the -u option to check a URL that runs a web based email login. So check_http -H ourhost -u I added it as a Service

But this is what Inspect shows:
'/usr/lib64/nagios/plugins/check_http' '-H' 'ourhost' '-u' '' '-I' 'x.x.x.x' '-p' '443'

How do I simply remove the -l and -p options?

Looks like the photo cuts off, but is it underneath “Arguments” as seen in the photo?

I would also double check fields.

Here’s the Service Template, Fields:

But the Executed Command has the extra arguments/options that cause this to show an error: HTTP WARNING: HTTP/1.1 400 Bad Request. Where can I find how to omit -i and -p?

yes, of the already existing http command in the External commands section.
I recommend you use that. There is no need to create a new command.

This is not what this parameter is for. Check check_http -h to get to know the plugin and its parameters.

I suggest you start over, delete the service template and your command. Add your required fields to the http command, then create a service template (e.g “service-template-http-generic”) using this command. Then you can create another service template (e.g “service-template-https”) and set the SSL option to true there and then apply a service from it to a host.

From Director → Commands → http → Fields → (fields dropdown) I do not see a “Add a new Data Field”

This you will find in the Directors main menu
forgot that screenshot in my previous post

Does this mean I can’t combine these 2 checks into one service template, i.e., one for a full URL with the -H and -u options, and another check for --ssl or -S True and an optional port number?

So this works from CLI:

sudo  -u icinga /usr/lib64/nagios/plugins/check_http -H ourhost  -u
HTTP OK: HTTP/1.1 302 Found - 461 bytes in 0.002 second response time |time=0.001751s;;;0.000000 size=461B;;;0

I thought between Arguments and Fields, as well as making them optional, could be covered in one service template?

I would have run the check like
check_http -H -u /webmail -S

For the director this would mean you have a host object for and add a service to this were you set http_uri to /webmail and http_ssl to yes (from the drop down after having create http_ssl as a boolean type data field)

1 Like