Perhaps I am misunderstanding the differences. I want to simply:
Is there an existing template I can use for this?
Ok, so you seem to be fairly new to the icinga2 (or monitoring) world and currently have little understanding of the system. No disrespect/blaming here, just an observation 
I’ll try to give you a small run-down of how this would be configured, but I would suggest you read the docs chapter about the monitoring basics to get better understanding of the different config objects:
https://icinga.com/docs/icinga-2/latest/doc/03-monitoring-basics/
So, to have a check you would have to do the following:
(I will just list the steps, not what exactly to configure)
Plugin check command.Service template that uses the command.Service (via Apply Rule, a Service Set or as a single object) which imports the templateWith this you than have a check (or maybe multiple checks) that monitor what you configured (e.g disk space)
The notifications basically have the same structure.
You start with a Notification Command, create a Notification template to define some options (interval, states, types, user…) and then create a Notification apply rule to hosts or services.
Well kind of. I was doing OK with Icinga2 but adding Director added a level of complexity that I didn’t expect.
And do appreciate you and others that respond here as I go through the leaning process…
Hm, yeah. The Director has it’s pros and cons/limitations. And starting to use it has its challenges when coming from the config files. But the other way round has its challenges as well 
But I like to think that it enforces some best practices (like the need to create a service template first before creating a service) which could (but imo shouldn’t) be bypassed when using the config files directly.
I created this command template and used check_http with --ssl but I still see this error:
HTTP WARNING: HTTP/1.1 400 Bad Request - 452 bytes in 0.012 second response time
'/usr/lib64/nagios/plugins/check_http' '-f' 'ok' '--sni' '--ssl' '-H' '%hostname%' '-I' '150.108.64.65' '-S' '-p' '443'
What am I doing wrong?
Edit: I see that the -H option was missing after adding that, this started working.
So the last item is Notifications. I had them working fine before adding Director. Is my Assign where correct?
warning/ApplyRule: Apply rule 'Icinga notification of possible issue' (in /var/lib/icinga2/api/packages/director/5878157f-a0b9-4b64-982b-988cd6e25878/zones.d/master/notification_apply.conf: 1:0-1:65) for type 'Notification' does not match anywhere!
cat /var/lib/icinga2/api/packages/director/xx/zones.d/master/notification_apply.conf
apply Notification "Icinga notification of possible issue" to Host {
import "Icinga notification on a possible server issue"
interval = 0s
assign where host.enable_notifications == "true"
users = [ "icinga-director" ]
}
host.enable_notifications is a boolean, can you try this syntax?
Ugh I added a host by using icinga2 node wizard and now have a mess. When trying to deploy I get:
* Unable to detect your Icinga 2 Core version (DeployFormsBug7530.php:72)
icingaweb2[3189]: Icinga\Exception\NotFoundError in /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php:642 with message: Failed to load icinga_host "ourdomain.edu"
#0 /usr/share/icingaweb2/modules/director/library/Director/Data/Db/DbObject.php(1173): Icinga\Module\Director\Data\Db\DbObject->loadFromDb()
#1 /usr/share/icingaweb2/modules/director/library/Director/Objects/IcingaObject.php(2595): Icinga\Module\Director\Data\Db\DbObject::load()
#2 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(462): Icinga\Module\Director\Objects\IcingaObject::loadByType()
#3 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(445): Icinga\Module\Director\Web\Controller\ObjectController->loadObject()
#4 /usr/share/icingaweb2/modules/director/library/Director/Web/Controller/ObjectController.php(78): Icinga\Module\Director\Web\Controller\ObjectController->eventuallyLoadObject()
#5 /usr/share/php/Icinga/Web/Controller/ActionController.php(165): Icinga\Module\Director\Web\Controller\ObjectController->init()
#6 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()
#7 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#8 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()
#9 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()
#10 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#11 {main}
Any idea what I should do? Reinstall Director?
Edit: well I uninstalled and reinstalled Director. Now I can’t kickstart it: * CURL ERROR: Failed to connect to 127.0.0.1 port 5665: Connection refused (RestApiClient.php:143)
firewalld is not running. Just to be sure I started firewalld and added an exception for port 5665. I even rebooted the server. netstat -plnt|grep 5665 brings back nothing. I see on all of the hosts I’m monitoring that nestat returns a value:
netstat -plnt|grep 5665
tcp6 0 0 :::5665 :::* LISTEN 1139109/icinga2
It’s as if the icinga2 service on the master server is not opening a port and I’ve restarted both the director and icinga2 services.
I do see these warnings:
[2021-10-28 16:31:29 -0400] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/director-global' for unknown zone 'director-global'.
[2021-10-28 16:31:29 -0400] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/master' for unknown zone 'master'.
So there is something misconfigured with the API. I did run icinga2 api setup to no avail only the afore-mentioned warnings…
Update: Running icinga2 node wizard and answering ‘yes’ for the master fixed issues allowing me to import Director.
The icinga2 node wizard command is used to setup the configuration for a satellite/agent OR the master. It creates the certificates(+CA if master), enables the API, and asks for connection details to the parent (if satellite/agent).
It is not for adding/configuring host objects for the monitoring.
That you do via the config files or the Director.
In Service Problems I see this error, any idea what it might mean?
#0 /usr/share/php/Icinga/Exception/IcingaException.php(41): ReflectionClass->newInstanceArgs()
#1 /usr/share/php/Icinga/Web/Controller.php(87): Icinga\Exception\IcingaException::create()
#2 /usr/share/icingaweb2/modules/monitoring/application/controllers/ServiceController.php(38): Icinga\Web\Controller->httpNotFound()
#3 /usr/share/php/Icinga/Web/Controller/ActionController.php(165): Icinga\Module\Monitoring\Controllers\ServiceController->init()
#4 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(59): Icinga\Web\Controller\ActionController->__construct()
#5 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#6 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()
#7 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()
#8 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#9 {main}
Edit: OK error seems to go away. I believe I was using a cached URL so perhaps this could be an enhancement or better warning/error message?
Looks like I found a bug. I followed this tutorial,
[2021-10-29 15:14:39 -0400] notice/ApiListener: Connected endpoints: ourdomain.edu (1)
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: DELETE FROM icinga_runtimevariables WHERE instance_id = 1
[2021-10-29 15:14:39 -0400] notice/ApiListener: Updating object authority for objects at endpoint 'ourdomain.edu'.
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_services', '3')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_services', '3')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_hosts', '1')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: INSERT INTO icinga_runtimevariables (instance_id, varname, varvalue) VALUES (1, 'total_scheduled_hosts', '1')
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: COMMIT
[2021-10-29 15:14:39 -0400] debug/IdoMysqlConnection: Query: BEGIN
[2021-10-29 15:14:39 -0400] notice/JsonRpcConnection: Error while reading JSON-RPC message for identity 'ourdomain.edu': Error: End of file
Stacktrace:
0# __cxa_throw in /usr/lib64/icinga2/sbin/icinga2
1# 0x00000000005A278D in /usr/lib64/icinga2/sbin/icinga2
2# icinga::JsonRpc::ReadMessage(boost::intrusive_ptr<icinga::Shared<icinga::AsioTlsStream> > const&, boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >, long) in /usr/lib64/icinga2/sbin/icinga2
3# icinga::JsonRpcConnection::HandleIncomingMessages(boost::asio::basic_yield_context<boost::asio::executor_binder<void (*)(), boost::asio::executor> >) in /usr/lib64/icinga2/sbin/icinga2
4# 0x000000000085B138 in /usr/lib64/icinga2/sbin/icinga2
5# 0x000000000088DC8C in /usr/lib64/icinga2/sbin/icinga2
6# make_fcontext in /lib64/libboost_context.so.1.75.0
[2021-10-29 15:14:39 -0400] warning/JsonRpcConnection: API client disconnected for identity 'ourdomain.edu'
[2021-10-29 15:14:39 -0400] warning/ApiListener: Removing API client for endpoint 'ourdomain.edu'. 0 API clients left.
[2021-10-29 15:14:39 -0400] debug/EndpointDbObject: update is_connected=0 for endpoint 'ourdomain.edu'
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: UPDATE icinga_endpointstatus SET is_connected = '0', status_update_time = FROM_UNIXTIME(1635534879) WHERE endpoint_object_id = 294 AND instance_id = 1
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: COMMIT
[2021-10-29 15:14:40 -0400] debug/IdoMysqlConnection: Query: BEGIN
From the client logs:
[2021-10-29 16:31:22 -0400] warning/JsonRpcConnection: API client disconnected for identity 'oudomain.edu'
[2021-10-29 16:31:32 -0400] information/ApiListener: New client connection for identity 'oudomain.edu' from [::ffff:x.x.x.x]:51464 (certificate validation failed: code 18: self signed certificate)
[2021-10-29 16:31:42 -0400] information/JsonRpcConnection: Closing anonymous connection [::ffff:150.108.64.65]:51464 after 10 seconds.
[2021-10-29 16:31:42 -0400] warning/JsonRpcConnection: API client disconnected for identity 'oudomain.edu'
Update: Hosts are at least showing as Up now how to get the Services going…
These errors remain on all nodes:
warning/ApiListener: Certificate validation failed for endpoint 'ourdomain.edu': code 18: self signed certificate
OK well the self signed certificate message was a clue.
First icinga2 ca list brought back nothing:
icinga2 ca list
Fingerprint | Timestamp | Signed | Subject
-----------------------------------------------------------------|--------------------------|--------|--------
So I ran pki sign-csr --csr master.ourdomain.edu.csr --cert /var/lib/icinga2/certs/ca.crt and scp’d the files to the agent’s /var/lib/icinga2/certs directory and restarted icinga2. Works now.
One (hopefully) last question, from the tutorial, I used, it does not have a Command or Command Template. So how do I use check_http and add arguments like --ssl? Everything is based on a Service Set and Service Template.
The tutorial does not show the command creation/configuration step because they are using a already defined command from the ITL. Those are mostly already good to go, as they often have some default values set.
If you want to fit them to your needs I suggest the following:
Go to the (in your case) http command, switch to the fields tab and add all the fields you need for your check. You will get a drop down menu there that will give you the fields of that command under argument macros
The description what each field is for can be found in the Preview tab.
You will find that, for example, the field for -S/–ssl is not present in the drop-down list(as are some others). This is because this field is not assigned to value but to set_if, meaning it is a boolean field not a field that expects a string.
For those fields you need to create the data field with the name http_ssl and type boolean by hand and can then add it via the drop-down list in the fields tab of the command (it will be listed further down the list in alphabetical order and not under the argument macros
Is this under Director and then Commands?
What I’m trying to also achieve is to use the -u option to check a URL that runs a web based email login. So check_http -H ourhost -u https://ourhost.ourdomain.edu/webmail I added it as a Service
But this is what Inspect shows:
'/usr/lib64/nagios/plugins/check_http' '-H' 'ourhost' '-u' 'https://ourhost.ourdomain.edu/webmail' '-I' 'x.x.x.x' '-p' '443'
How do I simply remove the -l and -p options?
Looks like the photo cuts off, but is it underneath “Arguments” as seen in the photo?
I would also double check fields.
Here’s the Service Template, Fields:
But the Executed Command has the extra arguments/options that cause this to show an error: HTTP WARNING: HTTP/1.1 400 Bad Request. Where can I find how to omit -i and -p?
yes, of the already existing http command in the External commands section.
I recommend you use that. There is no need to create a new command.
This is not what this parameter is for. Check check_http -h to get to know the plugin and its parameters.
I suggest you start over, delete the service template and your command. Add your required fields to the http command, then create a service template (e.g “service-template-http-generic”) using this command. Then you can create another service template (e.g “service-template-https”) and set the SSL option to true there and then apply a service from it to a host.
This you will find in the Directors main menu
forgot that screenshot in my previous post
Does this mean I can’t combine these 2 checks into one service template, i.e., one for a full URL with the -H and -u options, and another check for --ssl or -S True and an optional port number?
So this works from CLI:
sudo -u icinga /usr/lib64/nagios/plugins/check_http -H ourhost -u https://ourdomain.edu/webmail
HTTP OK: HTTP/1.1 302 Found - 461 bytes in 0.002 second response time |time=0.001751s;;;0.000000 size=461B;;;0
I thought between Arguments and Fields, as well as making them optional, could be covered in one service template?
Interesting.
I would have run the check like
check_http -H ourdomain.edu -u /webmail -S
For the director this would mean you have a host object for ourdomain.edu and add a service to this were you set http_uri to /webmail and http_ssl to yes (from the drop down after having create http_ssl as a boolean type data field)