Need Help Setting Up Email Notification

Icinga 2
1

I have a new Icinga 2 server with Director running on a new Ubuntu 24.04 VM.
Agents are running on servers, things are working normally there. (except one Windows Server I can’t get to talk to the Icinga 2 server)

To finalize the setup I need to get the alerts going but I can’t find simple enough instructions for how to set up the alerts themselves (high CPU utilization alerts, low disk space, etc.) and I can’t find anything that tells me how to set up the email notification if you already have your own SMTP server (which we do). All the instructions I can find only tell you how to set up Postfix on the Icinga server, which I don’t need to do. I just need to point it to our working SMTP server.

I’m a Windows guy just learning Linux, but I’m a fast learner. I can follow directions very well so long as they don’t skip steps or expect me to know Linux commands I don’t know. I figure this should all be configurable in the Icinga 2 Director, but its’ not exactly intuitive for beginners there.

2

Did you follow the example host and service notification commands and the documentation?
Important is to understand, how the objects relate to each other in Icinga2.
Each of the objects has it’s own documentation to show you what you can set and what is a runtime property (you can inspect them via API or icinga2 object list see trouble shooting in the docs).

Host:

  - name, address, vars.\*, groups = \[HostGroup...\]



Service:

  - host_name -> Host

  - check_command -> CheckCommand

  - vars.\*, groups = \[ServiceGroup...\]



CheckCommand:

  - command = plugin/script

  - arguments, env, timeout



Notification:

  - host_name/service_name -> Host/Service

  - users = \[User...\] or user_groups = \[UserGroup...\]

  - command = NotificationCommand

  - types = \[Problem, Recovery, Acknowledgement, etc.\]

  - states = \[OK, Warning, Critical, Unknown\]

  - times = TimePeriod (optional), interval, apply rules



NotificationCommand:

  - command = script/mail/slack/webhook

  - arguments (use notification, host/service vars)



User:

  - email, pager, vars.\*, groups = \[UserGroup...\]



UserGroup:

  - members = \[User...\]

or in a more graphical representation:

+------------------+
|       Host       |
+---------+--------+
          |
          | 1..N Services per Host
          v
+---------+--------+
|      Service     |
+---------+--------+
          |
          | 1 CheckCommand per Service
          v
+---------+--------+
|   CheckCommand   |
+---------+--------+
          |
          | emits status -> Check Result (runtime)
          v
+---------+--------+
|   Notification   |<-----------------------------+
+---------+--------+                              |
          |                                       |
          | 1 NotificationCommand per Notification|
          v                                       |
+---------+--------+                              |
|NotificationCommand|                             |
+---------+--------+                              |
          |                                       |
          | targets                               |
          v                                       |
+---------+--------+       +------------------+   |
|        User      |<------+    UserGroup     +---+
+------------------+       +------------------+
(memberships: User ∈ UserGroup)
3

Where do I put in the basic SMTP server information that will cover all those service notifications?

4

I did put it into /etc/exim4 and the notification scripts call the mail command.
I guess exim, postfix or any local mail transport agent that provides the mail command will do.

5

Ok, now I’m back to being confused.
I was going to (and kinda did) install Postfix because I was told I needed it. But Postfix just creates an SMTP server for Icinga to use for sending out the email notificiations. We already have our own SMTP server.
Isn’t there a way to configure Icinga 2 to just use that existing SMTP server for all email notifications without having to install Postfix or Exim4?

6

You can configure postfix to act as a satellite, so it forwards anything to another smtp server

7

But that still involves installing Postfix on the Icinga 2 server.
Is there any way to tell Icinga 2 to forward the SMTP calls without installing PostFix or Exim4 on the Icinga 2 server?

8

Your notification command needs to send the constructed message and subject to the contacts. The examples use the mail command of the local MTAs (exim4 or postfix) but nothing is stopping you from using a an other command that directly talks to your SMTP server or convert the bash example into an python script and use a library that can talk to your SMTP, gmail, whatsapp, teams, slack, webex…

The usage of the mail command of the local MTA is the traditional UNIX and Linux way as every user on every system has his own mailbox and a lot of programs can or do use messages besides logs to inform about problems and notices (MAILTO in crontabs). Administrators normally use aliases to gather messages and forward them to there team or personal mailboxes.

9

i mean, you can ofc write your own notification script.

For my personal setup, i didnt want to have postfix on my icinga machine, so i just created my own notification script which just sends mails using my smtp server hosted externally. If you are interested, i can show you my script. Ofc, its pretty much tailored to my needs, but maybe it gives you an idea.

10

A little script to replace the mail cli command would be enough as MVP in combination with the example host and service mail notification commands :wink:

11

Thank you for all the help.
It’s starting to sound like it might be the best idea just to put Postfix back on if nothing else than to get the email commands built into the OS.

If i did that, would I be able to use the Postfix commands to talk to our SMTP server?

12

If i did that, would I be able to use the Postfix commands to talk to our SMTP server?

Yes, if you configure it as a satellite. This results is a Postfix installation that only accepts local mail and forwards it to your SMTP.

13

Are you looking just for a script to generate the mail messages towards your smtp server? You will find this for hosts and services under /etc/icinga2/scripts/mail-*.sh (or from the icinga source icinga2/etc/icinga2/scripts at master · Icinga/icinga2 · GitHub). You can just call this scripts from your icinga notification configuration when ever icinga has to send mails. It uses /usr/bin/mail as a dependency (provided by mailutils/mailx), but you can also use any other mail client by changing MAILBIN variable in the scripts.

1 Like
14

This is the central issue in the entire discussion :wink:

15

Ok. I got the Postfix server installed, setup and tested, it’s working with our SMTP server.
Now where/how in Icigina 2 with Director so I put in the SMTP server or do I not need to do that now?

16

Within the notification script, i.e., a bash script for which a default is provided, but that you can (and should) customise.

NB: there are 2 scripts, one for the Host Notifications, and the other for the Service Notification. They use different sets of variables.

17

PS: these notification scripts should also work when run from the command line (except of course for the values coming from Icinga when invoked from Icinga, including the target mail address)