This you will find in the Directors main menu
forgot that screenshot in my previous post
Does this mean I can’t combine these 2 checks into one service template, i.e., one for a full URL with the -H and -u options, and another check for --ssl or -S True and an optional port number?
So this works from CLI:
sudo -u icinga /usr/lib64/nagios/plugins/check_http -H ourhost -u https://ourdomain.edu/webmail
HTTP OK: HTTP/1.1 302 Found - 461 bytes in 0.002 second response time |time=0.001751s;;;0.000000 size=461B;;;0
I thought between Arguments and Fields, as well as making them optional, could be covered in one service template?
Interesting.
I would have run the check like
check_http -H ourdomain.edu -u /webmail -S
For the director this would mean you have a host object for ourdomain.edu and add a service to this were you set http_uri to /webmail and http_ssl to yes (from the drop down after having create http_ssl as a boolean type data field)
1 Like
Wow so you “paramaterize” all the elements into objects. Guess there’s more than one way to skin a cat here. This now works, where I created a service just for this host:
1 Like
I generally prefer combining into one service as well. You can think of it similar to checking tcp port 443, checking the SSL handshake and then checking a GET, but all in one command. If it fails the TCP handshake, it will be CRITICAL and inform why - so there is generally no reason to deconstruct each test into individual services.
Does this look right? Still no email notifications. I have an Always Time Period. Email from CLI works. What does ‘corp’ refer to here? icinga2 object list --type=notification does not return any values/output.
“corp” was relevant in the other thread but may not be here. The reason I used it as an example was with regards to the fact that it is a boolean and to use that same syntax (i.e. do not compare it to a string).
Ok thanks. Any idea why notifications aren’t working? When I didn’t have Director everything just worked. What else can I provide?
If icinga2 object list --type=notification is not showing anything, can you share your configs for the Notification Apply Rule?
zones.d/master/notification_apply.conf
apply Notification "Services Notification" to Service {
import "Template for Services to be monitored"
period = "Always"
assign where host.vars.os in [ "Linux" ]
states = [ Critical ]
types = [ Problem ]
users = [ "sysadmin" ]
}
zones.d/master/notification_apply.conf
apply Notification "Host Notifications" to Host {
import "Template for Hosts to be monitored"
period = "Always"
assign where host.vars.os in [ "Linux" ]
states = [ Down ]
users = [ "sysadmin" ]
}
First guess is assign where host.vars.os in [ "Linux" ]. You may not be getting that value (there are some other threads that are similar, iirc). I would test it with something more standard/universally accessible first. Personally I like using host.groups since can be used somewhat like a tag.
Yep I tried that after a few others.
I don’t have any groups ATM.
Any other suggestions?
I added a group dies this look right?
zones.d/master/notification_apply.conf
apply Notification "Services Notification" to Service {
import "Template for Services to be monitored"
period = "Always"
assign where "admin" in host.groups
states = [ Critical ]
types = [ Problem ]
users = [ "sysadmin" ]
}
I tried for Host Notifications to have host.groups = admin and got a stacktrace:
#0 [internal function]: Icinga\Application\ApplicationBootstrap->Icinga\Application\{closure}()
#1 /usr/share/php/Icinga/Web/View.php(107): htmlspecialchars()
#2 /usr/share/icingaweb2/library/vendor/Zend/View/Helper/FormText.php(70): Icinga\Web\View->escape()
#3 /usr/share/icingaweb2/library/vendor/Zend/View/Abstract.php(345): Zend_View_Helper_FormText->formText()
#4 /usr/share/php/Icinga/Web/View.php(251): Zend_View_Abstract->__call()
#5 /usr/share/icingaweb2/modules/director/application/views/helpers/FormDataFilter.php(239): Icinga\Web\View->__call()
#6 /usr/share/icingaweb2/modules/director/application/views/helpers/FormDataFilter.php(216): Zend_View_Helper_FormDataFilter->selectGroup()
#7 /usr/share/icingaweb2/modules/director/application/views/helpers/FormDataFilter.php(164): Zend_View_Helper_FormDataFilter->element()
#8 /usr/share/icingaweb2/modules/director/application/views/helpers/FormDataFilter.php(178): Zend_View_Helper_FormDataFilter->filterExpressionHtml()
#9 /usr/share/icingaweb2/modules/director/application/views/helpers/FormDataFilter.php(83): Zend_View_Helper_FormDataFilter->renderFilterExpression()
#10 /usr/share/icingaweb2/modules/director/application/views/helpers/FormDataFilter.php(69): Zend_View_Helper_FormDataFilter->renderFilter()
#11 /usr/share/icingaweb2/library/vendor/Zend/View/Abstract.php(345): Zend_View_Helper_FormDataFilter->formDataFilter()
#12 /usr/share/php/Icinga/Web/View.php(251): Zend_View_Abstract->__call()
#13 /usr/share/icingaweb2/library/vendor/Zend/Form/Decorator/ViewHelper.php(254): Icinga\Web\View->__call()
#14 /usr/share/icingaweb2/library/vendor/Zend/Form/Element.php(2050): Zend_Form_Decorator_ViewHelper->render()
#15 /usr/share/icingaweb2/library/vendor/Zend/Form/Decorator/FormElements.php(115): Zend_Form_Element->render()
#16 /usr/share/icingaweb2/library/vendor/Zend/Form/DisplayGroup.php(914): Zend_Form_Decorator_FormElements->render()
#17 /usr/share/icingaweb2/library/vendor/Zend/Form/Decorator/FormElements.php(115): Zend_Form_DisplayGroup->render()
#18 /usr/share/icingaweb2/library/vendor/Zend/Form.php(2974): Zend_Form_Decorator_FormElements->render()
#19 /usr/share/icinga-php/ipl/vendor/ipl/html/src/HtmlDocument.php(339): Zend_Form->render()
#20 /usr/share/icinga-php/ipl/vendor/ipl/html/src/BaseHtmlElement.php(255): ipl\Html\HtmlDocument->renderUnwrapped()
#21 /usr/share/icinga-php/ipl/vendor/ipl/html/src/BaseHtmlElement.php(324): ipl\Html\BaseHtmlElement->renderContent()
#22 /usr/share/icinga-php/ipl/vendor/ipl/html/src/HtmlDocument.php(498): ipl\Html\BaseHtmlElement->renderUnwrapped()
#23 /usr/share/icinga-php/ipl/vendor/ipl/html/src/HtmlDocument.php(369): ipl\Html\HtmlDocument->render()
#24 /usr/share/icingaweb2/modules/incubator/vendor/gipfl/icingaweb2/src/Zf1/SimpleViewRenderer.php(65): ipl\Html\HtmlDocument->__toString()
#25 /usr/share/icingaweb2/modules/incubator/vendor/gipfl/icingaweb2/src/Zf1/SimpleViewRenderer.php(104): gipfl\IcingaWeb2\Zf1\SimpleViewRenderer->render()
#26 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action/HelperBroker.php(272): gipfl\IcingaWeb2\Zf1\SimpleViewRenderer->postDispatch()
#27 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(518): Zend_Controller_Action_HelperBroker->notifyPostDispatch()
#28 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch()
#29 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#30 /usr/share/php/Icinga/Application/Web.php(304): Zend_Controller_Front->dispatch()
#31 /usr/share/php/Icinga/Application/webrouter.php(107): Icinga\Application\Web->dispatch()
#32 /usr/share/icingaweb2/public/index.php(4): require_once('/usr/share/php/...')
#33 {main}
And this warning might help:
[2021-11-06 22:55:06 -0400] warning/ApplyRule: Apply rule 'Services Notification'
(in [stage]/zones.d/master/notification_apply.conf: 1:0-1:52) for type 'Notification' does not match anywhere!
[2021-11-06 22:55:06 -0400] warning/ApplyRule: Apply rule 'Host Notifications'
(in [stage]/zones.d/master/notification_apply.conf: 11:1-11:47) for type 'Notification' does not match anywhere!
host.groups is an array, so you’ll want to use contains. Also make sure the host group exists and the host is a member of it.
1 Like
OK now I get a test notification and
## zones.d/master/notification_apply.conf
apply Notification "Services Notification" to Service
{
import "[Template for Services to be monitored]"
period = "Always"
assign where service.enable_notifications
states = [ Critical, Unknown ] users = [ "sysadmin" ]
}
icinga2 object list --type=notification|more
Object 'ourschool.edu!Service Template for Server Monitoring Disk Usage!Services Notification' of type 'Notification':
% declared in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 1:0-1:52
* __name = "ourschool.edu!Service Template for Server Monitoring Disk Usage!Services Notification"
* command = "mail-service-notification"
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_templates.conf', lines 2:5-2:41
* command_endpoint = ""
* host_name = "ourschool.edu"
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 1:0-1:52
* interval = 1800
* name = "Services Notification"
* package = "director"
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 1:0-1:52
* period = "Always"
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_templates.conf', lines 3:5-3:21
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 4:5-4:21
* service_name = "Service Template for Server Monitoring Disk Usage"
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 1:0-1:52
* source_location
* first_column = 0
* first_line = 1
* last_column = 52
* last_line = 1
* path = "/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf"
* states = [ "Critical", "Unknown" ]
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_templates.conf', lines 4:5-4:31
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 6:5-6:34
* templates = [ "Services Notification", "Template for Services to be monitored" ]
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 1:0-1:52
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_templates.conf', lines 1:0-1:60
* times = null
* type = "Notification"
* types = null
* user_groups = [ "admin" ]
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_templates.conf', lines 5:5-5:29
* users = [ "sysadmin" ]
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_templates.conf', lines 6:5-6:26
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 7:5-7:26
* vars = null
* zone = "master"
% = modified in '/var/lib/icinga2/api/packages/director/1df86275-35fe-403b-b734-0df845305c68/zones.d/master/notification_apply.conf', lines 1:0-1:52
But I never get an email when a service is showing down, e.g., systemctl stop icinga2 on an agent. Why aren’t the email alerts going out other than via a test in the GUI?
Here is a log snip from icinga2.log:
[2021-11-08 15:40:56 -0500] information/Checkable: Checkable 'ourschool.edu!Service Template for Server Monitoring Web https' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-11-08 15:41:31 -0500] information/ApiListener: New client connection from [::ffff:150.108.64.65]:54024 (no client certificate)
[2021-11-08 15:41:31 -0500] information/HttpServerConnection: Request: GET /v1/ (from [::ffff:150.108.64.65]:54024), user: root, agent: , status: OK).
[2021-11-08 15:41:31 -0500] information/HttpServerConnection: Request: GET /v1/ (from [::ffff:150.108.64.65]:54024), user: root, agent: , status: OK).
[2021-11-08 15:41:31 -0500] information/HttpServerConnection: HTTP client disconnected (from [::ffff:150.108.64.65]:54024)
[2021-11-08 15:41:33 -0500] information/ApiListener: New client connection from [::ffff:150.108.64.65]:54026 (no client certificate)
[2021-11-08 15:41:33 -0500] information/HttpServerConnection: Request: GET /v1/ (from [::ffff:x.x.x.x]:54026), user: root, agent: , status: OK).
[2021-11-08 15:41:33 -0500] information/HttpServerConnection: Request: GET /v1/ (from [::ffff:x.x.x.x]:54026), user: root, agent: , status: OK).
[2021-11-08 15:41:33 -0500] information/HttpServerConnection: HTTP client disconnected (from [::ffff:x.x.x.x]:54026)
I’m trying suggestions from [this tutorial], (NSM2021/Agenda/AlertsIcinga2Email – Learn Workshops)
When I try to send a notification with the force option no email is sent and these are the logs:
warning/PluginNotificationTask: Notification command for object 'host.myschool.edu!Service Template for Server Monitoring System Load' (PID: 2040128, arguments: '/etc/icinga2/scripts/mail-service-notification.sh' '-4' 'x.x.x.x' '-6' '' '-b' 'root' '-c' '$host.display_name$' '-d' '2021-11-08 16:48:08 -0500' '-e' 'Service Template for Server Monitoring System Load' 'myschool.edu' '-n' 'host' '-o' 'CRITICAL - load average: 7.50, 6.83, 5.31' '-r' 'me@mychool.edu' '-s' 'CRITICAL' '-t' 'CUSTOM' '-u' 'Service Template for Server Monitoring System Load') terminated with exit code 1, output: Required parameter 'HOSTDISPLAYNAME' is missing.
Required parameters:
-d LONGDATETIME ($icinga.long_date_time$)
-e SERVICENAME ($service.name$)
-l HOSTNAME ($host.name$)
-n HOSTDISPLAYNAME ($host.display_name$)
-o SERVICEOUTPUT ($service.output$)
-r USEREMAIL ($user.email$)
-s SERVICESTATE ($service.state$)
-t NOTIFICATIONTYPE ($notification.type$)
-u SERVICEDISPLAYNAME ($service.display_name$)
Optional parameters:
-4 HOSTADDRESS ($address$)
-6 HOSTADDRESS6 ($address6$)
-b NOTIFICATIONAUTHORNAME ($notification.author$)
-c NOTIFICATIONCOMMENT ($notification.comment$)
-i ICINGAWEB2URL ($notification_icingaweb2url$, Default: unset)
-f MAILFROM ($notification_mailfrom$, requires GNU mailutils (Debian/Ubuntu) or mailx (RHEL/SUSE))
-v ($notification_sendtosyslog$, Default: false)
But HOSTDISPLAYNAME is clearly there. I can’t find in any place where the argument for the '-c' '$host.display_name$' part of the log entry being set.
Are there two values after -e?
Good eye yes, there appears to be 2 values: '-e' 'Service Template for Server Monitoring System Load' 'myschool.edu' '-n' 'host' '-o'
Where else could the -e option being set?
I think you might be mixing host notifications and service notifications here.
DOWN is the state of a host.
For this you would need to configure a host notification.
The whole command creation part from the tutorial is not needed, because you already have both host and service commands present in the director and the work without problems.
Just use those in your notification templates.
Currently you are extending/overwriting the pre-existing commands with additional parameters from you newly created command and that leads to problems (my guess)
My bad, I meant service is showing CRITICAL.
Yes I can send the notification via “Send Notifications” but the email never goes out all I see in the logs are:
[2021-11-09 09:28:03 -0500] information/HttpServerConnection: HTTP client disconnected (from [::ffff:150.108.64.65]:54762)
[2021-11-09 09:28:03 -0500] information/Checkable: Checkable 'ourschool.edu!Service Template for Server Monitoring Web https' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.
[2021-11-09 09:28:42 -0500] information/ConfigObject: Dumping program state to file '/var/lib/icinga2/icinga2.state'
[2021-11-09 09:28:52 -0500] information/WorkQueue: #6 (ApiListener, SyncQueue) items: 0, rate: 0/s (0/min 0/5min 0/15min);
[2021-11-09 09:28:52 -0500] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 1.23333/s (74/min 158/5min 159/15min);
[2021-11-09 09:28:52 -0500] information/IdoMysqlConnection: Pending queries: 5 (Input: 3/s; Output: 3/s)