Applying commands/command template to hosts, e.g., check_https or check_http --ssl

Just installed Director and I just can’t find how to do some basic things such as a check for a URL with a custom port, e.g., 8443, and SSL is reachable. This was pretty easy outside of Director. I also want to exclude this error: DISK CRITICAL - /run/user/xxx/doc is not accessible: Permission denied (xxx happens to be my Unix user ID in /etc/passwd)

I’ve tried adding commands in Director → Commands → Command Templates and Director → Commands → Commands but then don’t know what to do with them.

And where would the check_https option be? Why doesn’t “Add new Icinga Service template” have check_https as an option?

  • Director version (System - About): 1.8.1
  • Icinga Web 2 version and modules (System - About): 2.9.3
  • Icinga 2 version (icinga2 --version):
icinga2 - The Icinga 2 network monitoring daemon (version: 2.13.1-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Fedora
  Platform version: 34 (Server Edition)
  Kernel: Linux
  Kernel version: 5.13.19-200.fc34.x86_64
  Architecture: x86_64

Build information:
  Compiler: GNU 11.2.1
  Build host: unknown
  OpenSSL version: OpenSSL 1.1.1l  FIPS 24 Aug 2021

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
  • Operating System and version: Fedora 34
  • Webserver, PHP versions: Apache, PHP 7.4.24

Hi Robbie
I’m new in ICINGA too.
But I found a method to add arguments in the command then you need make a template with the command and finaly make the service with this temaplte.
I hope this will help

Can you specify the click path, e.g., Director → Commands → Template

I see you use -f, are you monitoring a redirect? Just curious what the check does.

Hi
I attached some images with the steps, and a link for the help of the check_http plugin too

https://www.monitoring-plugins.org/doc/man/check_http.html





Regards

Thanks so much for the screen shots. On the last step for Services, the version I have is Services then Single Services then Add. However I can’t get the Check command to include the ‘inherited’ portion of Check command. I deleted everything and decided to start from scratch.

[2021-10-20 14:46:36 -0400] critical/config: Error: Validation failed for object 'ourdomain.edu' of type 'Host'; Attribute 'check_command': Attribute must not be empty. Location: in [[stage]/zones.d/master/hosts.conf](https://ourdomain.edu/icingaweb2/director/config/file?config_checksum=7f8c25913008cefd514401b9a4cd684dc51fcd34&deployment_id=76&file_path=zones.d%2Fmaster%2Fhosts.conf&backTo=deployment&highlight=1&highlightSeverity=critical): 1:0-1:34 
[[stage]/zones.d/master/hosts.conf](https://ourdomain.edu/icingaweb2/director/config/file?config_checksum=7f8c25913008cefd514401b9a4cd684dc51fcd34&deployment_id=76&file_path=zones.d%2Fmaster%2Fhosts.conf&backTo=deployment&highlight=1&highlightSeverity=critical)(1): object Host "ourdomain.edu" 
          { ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
[[stage]/zones.d/master/hosts.conf](https://ourdomain.edu/icingaweb2/director/config/file?config_checksum=7f8c25913008cefd514401b9a4cd684dc51fcd34&deployment_id=76&file_path=zones.d%2Fmaster%2Fhosts.conf&backTo=deployment&highlight=2&highlightSeverity=critical)(2): import "Standard Linux Server" 
[[stage]/zones.d/master/hosts.conf](https://ourdomain.edu/icingaweb2/director/config/file?config_checksum=7f8c25913008cefd514401b9a4cd684dc51fcd34&deployment_id=76&file_path=zones.d%2Fmaster%2Fhosts.conf&backTo=deployment&highlight=3&highlightSeverity=critical)(3): 
[2021-10-20 14:46:36 -0400] critical/config: 1 error 
[2021-10-20 14:46:36 -0400] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.

Edit: it appears I need a value in the Check command field within Add new Icgina Host template. What should I put?

The Director already has all the commands from the Icinga Template Library (ITL) added to the “external commands” section. You just need to run the kickstart wizard under Infrastructure->Kickstart wizard (/icingaweb2/director/dashboard?name=infrastructure#!/icingaweb2/director/kickstart).

This will give you, for example, a fully configured http command, where you only have to add your desired fields and can then create a service template using the command.

Small drawback: After adding your desired fields, check what parameter has the variable/field configured as set_if. Like the --sni parameter in the screenshot.
For those parameters you need to change the field type from String to Boolean.
You can do this via the Director under “Define Datafields”
image

1 Like

OK I see the external commands. What do I select in the Command type drop down? Plugin check command?



What are you trying to achieve with the command template?
I have never used them tbh.

Just configure a service template using the command you want and then apply some services (via apply rules, service sets or just as single service objects directly to hosts (or templates)

Perhaps I am misunderstanding the differences. I want to simply:

  • send an email alert when disk space on various partitions over a few servers reach a certain threshold e.g. ,<+ 8%
  • send an email when server load exceeds a threshold, e.g., >-5 over a 10 minute time span
  • send an email when sites we have running on ports 443 and another on 8443.

Is there an existing template I can use for this?

Ok, so you seem to be fairly new to the icinga2 (or monitoring) world and currently have little understanding of the system. No disrespect/blaming here, just an observation :slight_smile:

I’ll try to give you a small run-down of how this would be configured, but I would suggest you read the docs chapter about the monitoring basics to get better understanding of the different config objects:
https://icinga.com/docs/icinga-2/latest/doc/03-monitoring-basics/

So, to have a check you would have to do the following:
(I will just list the steps, not what exactly to configure)

  1. Configure a Plugin check command.
    – This is the script (maybe with some arguments) that is executed when the check is run.
  2. Create a Service template that uses the command.
    – Service templates can define check execution parameters that will be given down to the service objects
  3. Define a Service (via Apply Rule, a Service Set or as a single object) which imports the template
    – this defines the check that will, be executed. It will inherit options set in the template and here you would set other information for the check parameters into fields (like the http_port, or some thresholds)

With this you than have a check (or maybe multiple checks) that monitor what you configured (e.g disk space)

The notifications basically have the same structure.
You start with a Notification Command, create a Notification template to define some options (interval, states, types, user…) and then create a Notification apply rule to hosts or services.

Well kind of. I was doing OK with Icinga2 but adding Director added a level of complexity that I didn’t expect.

And do appreciate you and others that respond here as I go through the leaning process…

Hm, yeah. The Director has it’s pros and cons/limitations. And starting to use it has its challenges when coming from the config files. But the other way round has its challenges as well :smiley:

But I like to think that it enforces some best practices (like the need to create a service template first before creating a service) which could (but imo shouldn’t) be bypassed when using the config files directly.

I created this command template and used check_http with --ssl but I still see this error:

HTTP WARNING: HTTP/1.1 400 Bad Request - 452 bytes in 0.012 second response time

'/usr/lib64/nagios/plugins/check_http' '-f' 'ok' '--sni' '--ssl' '-H' '%hostname%' '-I' '150.108.64.65' '-S' '-p' '443'


What am I doing wrong?

Edit: I see that the -H option was missing after adding that, this started working.

So the last item is Notifications. I had them working fine before adding Director. Is my Assign where correct?

warning/ApplyRule: Apply rule 'Icinga notification of possible issue' (in /var/lib/icinga2/api/packages/director/5878157f-a0b9-4b64-982b-988cd6e25878/zones.d/master/notification_apply.conf: 1:0-1:65) for type 'Notification' does not match anywhere!

cat /var/lib/icinga2/api/packages/director/xx/zones.d/master/notification_apply.conf
apply Notification "Icinga notification of possible issue" to Host {
    import "Icinga notification on a possible server issue"

    interval = 0s
    assign where host.enable_notifications == "true"
    users = [ "icinga-director" ]
}

host.enable_notifications is a boolean, can you try this syntax?

Ugh I added a host by using icinga2 node wizard and now have a mess. When trying to deploy I get:
* Unable to detect your Icinga 2 Core version (DeployFormsBug7530.php:72)

icingaweb2[3189]: Icinga\Exception\NotFoundError in /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php:642 with message: Failed to load icinga_host "ourdomain.edu"    
#0 /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php(1173): Icinga\Module\Director\Data\Db\DbObject->loadFromDb()    
#1 /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObject.php(2595): Icinga\Module\Director\Data\Db\DbObject::load()    
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(462): Icinga\Module\Director\Objects\IcingaObject::loadByType()    
#3 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(445): Icinga\Module\Director\Web\Controller\ObjectController->loadObject()    
#4 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(78): Icinga\Module\Director\Web\Controller\ObjectController->eventuallyLoadObject()    
#5 /usr/share/php/Icinga/Web/Controller/ActionController.php(165): Icinga\Module\Director\Web\Controller\ObjectController->init()    
#6 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()    
#7 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()    
#8 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()    
#9 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()    
#10 /usr/share/icingaweb2/public/index.php(4): require_once(String)    
#11 {main}

icingaerrors

Any idea what I should do? Reinstall Director?

Edit: well I uninstalled and reinstalled Director. Now I can’t kickstart it: * CURL ERROR: Failed to connect to 127.0.0.1 port 5665: Connection refused (RestApiClient.php:143)

firewalld is not running. Just to be sure I started firewalld and added an exception for port 5665. I even rebooted the server. netstat -plnt|grep 5665 brings back nothing. I see on all of the hosts I’m monitoring that nestat returns a value:

netstat -plnt|grep 5665
tcp6       0      0 :::5665                 :::*                    LISTEN      1139109/icinga2

It’s as if the icinga2 service on the master server is not opening a port and I’ve restarted both the director and icinga2 services.

I do see these warnings:

[2021-10-28 16:31:29 -0400] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/director-global' for unknown zone 'director-global'.
[2021-10-28 16:31:29 -0400] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/master' for unknown zone 'master'.

So there is something misconfigured with the API. I did run icinga2 api setup to no avail only the afore-mentioned warnings…

Update: Running icinga2 node wizard and answering ‘yes’ for the master fixed issues allowing me to import Director.

The icinga2 node wizard command is used to setup the configuration for a satellite/agent OR the master. It creates the certificates(+CA if master), enables the API, and asks for connection details to the parent (if satellite/agent).
It is not for adding/configuring host objects for the monitoring.
That you do via the config files or the Director.

In Service Problems I see this error, any idea what it might mean?

#0 /usr/share/php/Icinga/Exception/IcingaException.php(41): ReflectionClass->newInstanceArgs()
#1 /usr/share/php/Icinga/Web/Controller.php(87): Icinga\Exception\IcingaException::create()
#2 /usr/share/icingaweb2/modules/monitoring/application/controllers/ServiceController.php(38): Icinga\Web\Controller->httpNotFound()
#3 /usr/share/php/Icinga/Web/Controller/ActionController.php(165): Icinga\Module\Monitoring\Controllers\ServiceController->init()
#4 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()
#5 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#6 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()
#7 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()
#8 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#9 {main}

Edit: OK error seems to go away. I believe I was using a cached URL so perhaps this could be an enhancement or better warning/error message?

Looks like I found a bug. I followed this tutorial,

[2021-10-29 15:14:39 -0400] notice/ApiListener: Connected endpoints: ourdomain.edu (1)
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2021-10-29 15:14:39 -0400] notice/ApiListener: Updating object authority for objects at endpoint 'ourdomain.edu'.
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '3')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '3')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '1')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '1')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: COMMIT
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: BEGIN
[2021-10-29 15:14:39 -0400] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'ourdomain.edu': Error: End of file

Stacktrace:
 0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
 1# 0x00000000005A278D in /usr/lib64/icinga2/sbin/icinga2
 2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
 3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
 4# 0x000000000085B138 in /usr/lib64/icinga2/sbin/icinga2
 5# 0x000000000088DC8C in /usr/lib64/icinga2/sbin/icinga2
 6# make_fcontext in /lib64/libboost_context.so.1.75.0
[2021-10-29 15:14:39 -0400] warning/JsonRpcConnection: API client disconnected for identity 'ourdomain.edu'
[2021-10-29 15:14:39 -0400] warning/ApiListener: Removing API client for endpoint 'ourdomain.edu'. 0 API clients left.
[2021-10-29 15:14:39 -0400] debug/EndpointDbObject: update is_connected=0 for endpoint 'ourdomain.edu'
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: UPDATE icinga_endpointstatus SET is_connected = '0',  status_update_time = FROM_UNIXTIME(1635534879) WHERE endpoint_object_id = 294 AND instance_id = 1
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: COMMIT
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: BEGIN

From the client logs:

[2021-10-29 16:31:22 -0400] warning/JsonRpcConnection: API client disconnected for identity 'oudomain.edu'
[2021-10-29 16:31:32 -0400] information/ApiListener: New client connection for identity 'oudomain.edu' from [::ffff:x.x.x.x]:51464 (certificate validation failed: code 18: self signed certificate)
[2021-10-29 16:31:42 -0400] information/JsonRpcConnection: Closing anonymous connection [::ffff:150.108.64.65]:51464 after 10 seconds.
[2021-10-29 16:31:42 -0400] warning/JsonRpcConnection: API client disconnected for identity 'oudomain.edu'

Update: Hosts are at least showing as Up now how to get the Services going…


hostsareup

These errors remain on all nodes:
warning/ApiListener: Certificate validation failed for endpoint 'ourdomain.edu': code 18: self signed certificate

OK well the self signed certificate message was a clue.

First icinga2 ca list brought back nothing:

icinga2 ca list
Fingerprint                                                      | Timestamp                | Signed | Subject


-----------------------------------------------------------------|--------------------------|--------|--------

So I ran pki sign-csr --csr master.ourdomain.edu.csr --cert /var/lib/icinga2/certs/ca.crt and scp’d the files to the agent’s /var/lib/icinga2/certs directory and restarted icinga2. Works now.

One (hopefully) last question, from the tutorial, I used, it does not have a Command or Command Template. So how do I use check_http and add arguments like --ssl? Everything is based on a Service Set and Service Template.
serviceset