Regular notification scripts are explained in the official documentation here and further here
To Do’s
If you like those guides or feel like they are outdated and you want to contribute to those guides with your own. Please feel free to add your own written guide on how to handle notifications with the director in written form or even as a visual guide screencast.
Are there new scripts no included in the Director installation that includes a standardized version of mail-host-notification.sh and mail-service-notification.sh?
No, but they do need to exist in your commands directory (ie, /usr/lib64/nagios/plugins/ in most cases). In the director, you will then define the command (unless they are externally defined in the Director, which they could be – we don’t use the default mail notification commands)
The two commands, that Icinga2 defines by default, are store in /etc/icinga2/conf.d/commandsconf and can be imported into the Director via the kickstart wizard. After that they will be shown in the external commands section.
In case you ran the Icinga 2 Node Wizard previously the inclusion of the conf.d directory will most likely be turned off. Then you will have to include the files via the icinga2.conf file by adding lines like
include "conf.d/api-users.conf"
include "conf.d/commands.conf"
The default script location is /etc/icinga2/scripts and that is where the pre-defined command expects them.
When I did that and restarted icinga2 I get this error:
critical/config: Error: Object 'root' of type 'ApiUser' re-defined: in /etc/icinga2/conf.d/api-users.conf: 4:1-4:21; previous definition: in /etc/icinga2/conf.d/api-users.conf: 4:1-4:21
Location: in /etc/icinga2/conf.d/api-users.conf: 4:1-4:21
/etc/icinga2/conf.d/api-users.conf(2): * The ApiUser objects are used for authentication against the API.
/etc/icinga2/conf.d/api-users.conf(3): */
/etc/icinga2/conf.d/api-users.conf(4): object ApiUser "root" {
^^^^^^^^^^^^^^^^^^^^^
as well as:
critical/config: Error: Object 'mail-host-notification' of type 'NotificationCommand' re-defined: in /etc/icinga2/conf.d/commands.conf: 9:1-9:51; previous definition: in /etc/icinga2/conf.d/commands.conf: 9:1-9:51
Location: in /etc/icinga2/conf.d/commands.conf: 9:1-9:51
/etc/icinga2/conf.d/commands.conf(7): */
/etc/icinga2/conf.d/commands.conf(8):
/etc/icinga2/conf.d/commands.conf(9): object NotificationCommand "mail-host-notification" {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/etc/icinga2/conf.d/commands.conf(10): command = [ ConfigDir + "/scripts/mail-host-notification.sh" ]
/etc/icinga2/conf.d/commands.conf(11):
[2021-10-19 12:17:30 -0400] critical/cli: Config validation failed. Re-run with 'icinga2 daemon -C' after fixing the config.
Since you are using the director to create these commands, you need to remove the duplicate object from the existing config file that is noted in the logs.
Did you run the icinga2 node wizard beforehand to setup the master instance?
Check if your icinga2.conf has some lines like
// Disabled by the node setup CLI command on 2021-02-15 14:00:37 +0000
// include_recursive "conf.d"
// Added by the node setup CLI command on 2021-02-15 14:00:37 +0000
include "conf.d/api-users.conf"
The error message sounds to me like you haven’t and Icinga is trying to import the conf-files multiple times (because of include_recursive "conf.d" and the single includes)
When you run the node wizard to set up your master instance it will ask you to stop including the conf.d directory altogether and will only include the api-users.conf by default.
The conf.d dir only holds sample configuration after the initial setup but the commands.conf file is one that I always import to save me the hassle of creating the notification commands in the director manually.
cat /etc/icinga2/icinga2.conf
/**
* Icinga 2 configuration file
* - this is where you define settings for the Icinga application including
* which hosts/services to check.
*
* For an overview of all available configuration options please refer
* to the documentation that is distributed as part of Icinga 2.
*/
/**
* The constants.conf defines global constants.
*/
include "constants.conf"
* The zones.conf defines zones for a cluster setup.
* Not required for single instance setups.
*/
include "zones.conf"
/**
* The Icinga Template Library (ITL) provides a number of useful templates
* and command definitions.
* Common monitoring plugin command definitions are included separately.
*/
include <itl>
include <plugins>
include <plugins-contrib>
include <manubulon>
type or paste code here
* This includes the Icinga 2 Windows plugins. These command definitions
* are required on a master node when a client is used as command endpoint.
*/
include <windows-plugins>
/**
* This includes the NSClient++ check commands. These command definitions
* are required on a master node when a client is used as command endpoint.
*/
include <nscp>
/**
* The features-available directory contains a number of configuration
* files for features which can be enabled and disabled using the
* icinga2 feature enable / icinga2 feature disable CLI commands.
* These commands work by creating and removing symbolic links in
* the features-enabled directory.
*/
include "features-enabled/*.conf"
/**
* Although in theory you could define all your objects in this file
* the preferred way is to create separate directories and files in the conf.d
* directory. Each of these files must have the file extension ".conf".
*/
include_recursive "conf.d"
So I first had Icinga2 working and then installed and attempted to configure Director. Do I run the wizard again?
You can run the wizard again.
Here is the part about it in the docs: Distributed Monitoring - Icinga 2
The you also find the information about what the wizard actually does.
It shouldn’t have any impact on your setup other than that the conf.d directory will not be included any more.
Do a snapshot of the VM before just to have a backup in case you have problems afterwards.
OK I ran the wizard and the couple hosts I had set up in Icinga2 before installing Director, have disappeared. I suppose I can add them back as Hosts under Director.
Yes. You should add the host templates, hosts, service templates and services via the director.
You can also take a look the the example config to get some inspiration
written form or 
even as a visual guide screencast.
