Icinga won't send emails

This seems to be recurring theme in the forums and I’ve followed many threads but none have resolved my issue.

sudo -u nagios ./mail-service-notification.sh -d test -l test -n test -o test <blah, blah blah> works, mail is received OK.

Enable Notifications is turned on everywhere I can find a setting for it.
Notification templates and Apply are all configured to enable all services & hosts to notify under all conditions.

Contacts & groups are configured. Contacts have email addresses.

Sending a notification with the web interface even with forced enabled does nothing. There is nothing in my postfix mail log, no errors in icinga logs just nothing.

Where do I even start to diagnose this, is there a diagnostic command I can run from icinga that will produce some meaningful diagnostics information?

Ironically this is the exact opposite of my main monitoring system that just won’t shut up sending emails!

Hi

To start run the validation and see if you are told that the notification logic is applied appropriately.
Then enable the debug log and see that when a notification logic should be triggered, what you see in the debug log.

Those are the first steps, and they will help us help you.

Good hunting

To start run the validation and see if you are told that the notification logic is applied appropriately.

I don’t know what you mean. Do you mean this:

icinga2 daemon -C
[2020-08-09 13:33:58 +0100] information/cli: Icinga application loader (version: r2.11.4-1)
[2020-08-09 13:33:58 +0100] information/cli: Loading configuration file(s).
[2020-08-09 13:33:58 +0100] information/ConfigItem: Committing config item(s).
[2020-08-09 13:33:58 +0100] information/ApiListener: My API identity: monitoring.tsdhosting.co.uk
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 HostGroup.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 FileLogger.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 NotificationComponent.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 IcingaApplication.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 3 Hosts.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 ApiListener.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 CheckerComponent.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 4 Zones.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 ExternalCommandListener.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 2 Endpoints.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 ApiUser.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 236 CheckCommands.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 2 TimePeriods.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 UserGroup.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 1 User.
[2020-08-09 13:33:58 +0100] information/ConfigItem: Instantiated 3 Services.
[2020-08-09 13:33:58 +0100] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2020-08-09 13:33:58 +0100] information/cli: Finished validating the configuration file(s).

I ran the debug option and generated a huge log which contained nothing of any value about email or notification.

Fiddling about with director notification settings I am getting this error

Location: in [stage]/zones.d/master/notification_templates.conf: 5:5-5:41 [stage]/zones.d/master/notification_templates.conf(3): begin = 30s [stage]/zones.d/master/notification_templates.conf(4): } [stage]/zones.d/master/notification_templates.conf(5): command = “mail-service-notification” ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [stage]/zones.d/master/notification_templates.conf(6): interval = 5m [stage]/zones.d/master/notification_templates.conf(7): period = “7x24”

But the command it says doesn’t exist is right there in director:

image

I can successfully run those scripts as the nagios user on the command line.

I got a bit further, it seems that because I’m using Director mail-service-notification.sh won’t work because it’s incompatible. So out of the box, following the docs the system is broken.

I followed the instructions here but that doesn’t work either. icinga log shows “terminated with exit code 128, output: execvpe(/usr/lib/nagios/plugins/object) failed: No such file or directory”

The default notification commands are stored in /etc/icinga2/conf.d/commands.conf

If you have this directory included in your icinga2.conf file then the commands can be used after using the Icinga Director kickstart.
As you have the commands inside the Director (under Commands->External Commands I guess) you have successfully imported them.
Did you run the icinga2 node wizard after the first kickstart in the Director? This could have disabled the inclusion of the conf.d directory and therefore the Director “doesn’t know” the command anymore.

I’ve done the Director Kickstart icinga2 node wizard include / don’t include conf.d dance along with appropriate restarts of icinga many times to no avail.

Can you show the current content of your icinga2.conf file?

What does the icinga2.log say around the time your are expecting a notification to be sent?

Hi - re “I got a bit further, it seems that because I’m using Director mail-service-notification.sh won’t work because it’s incompatible”… That sounds unlikely.

First you should check that your script is actually run by Icinga. Easily done; insert a line like this near begin of your “.sh” script:
[ "$DBG" ] && echo -e "At start ($(date)) of $0, call-up arg's were: $*\n\n" >> /tmp/_m-s-n.debuglog

($DBG here is an imaginary variable, that you would have set non-empty beforehand, perhaps in a preceding definition, e.g. DBG=yes.)

In parallel, you may also have to check that your notification rule in Director was set up okay by you; Director differentiates between external & internal commands. Perhaps you can let us see the configuration of your notification rule and any imports you use in it (and their imports, recursively).

P.S.: I work for a firm that has 15+ customers using Icinga professionally (installed by us) for partly over a decade. Of course there’s room for improvement. You learn which components to use and which to avoid. We’re pretty happy with Icinga, esp. improvements of Icinga2 over the essentially straight old-time Nagios fork Icinga1.

According to this article the script isn’t compatible.

I followed the instructions and eventually ended up with a system that could send mail.

I have other problems now in that a simple memory check won’t run. It’s just one problem after another. I’ve no doubt it works once it’s all setup but the learning curve even for someone like me who used Nagios by editing the config files directly, the curve is enormous.

I was hoping Director would make this easier, it just seems to add yet more inconsistency and strangeness.

That’s the documentation you linked earlier; I had a look at that then. The scripts described there are *-by-mail.sh; the mail-*-notification.sh scripts are other scripts and need to be defined accordingly in Director (easiest: as internal commands), or per fixed external definition. You have a lot of leeway on your notification scripts with Icinga.

“One problem after another” is a good description of life as an i.t. admin.*.

Describe your problem in a new thread, documenting how your memory script was set up as command in Director, and how the service template looks that you set up for it, etc., and I promise to have a look at it (you may have to give me a nudge - I pass by here perhaps once a week as a rule).

(* Paul Simon sang on his Graceland album: “The thought that life could be better is written indelibly onto our hearts” - that’s my admin. motto :upside_down_face:)

P.S. (late edit): I see you already opened a thread on the memory problem. I think “log1c” is doing a good job of advising you there… Be careful what you call what, i.e. usual Icinga jargon; a command definition is not a “service object”. In general it’s a good idea in my experience to always have a service template for any command definition, and then import this template in any services you define, either as single services allocated to a host, or as services within a service set.

1 Like