Interesting mail notification behavior

Hello,

We encountered a weird behavior of the mail service notification.

We have passive Services defined with exactly the same simple configuration. Only their names are different:

apply Service "MailTestX" {
  import "passive-service"
  assign where host.name == "host.xxx.local"
}

apply Service "MailTestY" {
  import "passive-service"
  assign where host.name == "host.xxx.local"
}

apply Service "MailTestZ" {
  import "passive-service"
  assign where host.name == "host.xxx.local"
}

apply Service "MailTestW" {
  import "passive-service"
  assign where host.name == "host.xxx.local"
}

We also have the most simple Host, Notification, User and UserGroup definitions:

object Host "host.xxx.local" {
	check_command = "dummy"
	vars.mail = true
}

apply Notification "NOC mail-test" to Service {
	import "mail-service-notification"
	user_groups = [ "noc_test" ]
	interval = 0
	assign where host.vars.mail
}

object UserGroup "noc_test" {
	display_name = "NOC Test Group"
}

object User "Xxxxx" {
	import "generic-user"
	display_name = "Yyyyy Xxxxx"
	groups = [ "noc_test" ]
	email = "yyyyy.xxxx@xxx.hu"
}

When we test the mail notification setting the service states, we received the notification emails only for two Services from the four, i.e emails are sent for services “MailTestX” and “MailTestW”, but no mail for “MailTestY” and “MailTestZ”. We tested it on several ways with different configurations and with different numbers of services but we always had services without sending emails.

What could be the reason?
Please, help!

Regards,
Endre

Do you have multiple masters? I have seen such behaviour when the second one had notification feature disabled or mailx not installed.

If it is not so simple, I would start with the maillog and have a look into to verify mails are created when Icinga 2 or the Webinterface tells you a notification was send. If a log message is shown you have to investigate where email delivery fails, if no message is logged then enable the debuglog to find the problem in Icinga 2.

Hi Dirk,

Thanks for your reply!

Yes, we have two masters in a pacemaker cluster, but at the moment the Icinga2 is not load balanced, so we are using only a dedicated one of the two.

The Webinterface and the Icinga log also shows that no notification was send so I don’t think we could find any relevant information in maillog.

Investigation with debugged Icinga2 log could be an option, we’ll try it.

Thanks,
Endre

Unfortunately, the Icinga debug log shows the same: Icinga attempts to send mail to one service but no sending for the other service :frowning:

MailTestX (no mail sent)

[2021-01-15 10:28:16 +0100] information/Checkable: Checking for configured notifications for object 'Mailtest.mvmh.local!MailTestX'
[2021-01-15 10:28:16 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:28:16 +0100] debug/Checkable: Checkable 'Mailtest.mvmh.local!MailTestX' has 1 notification(s).
[2021-01-15 10:28:16 +0100] notice/ApiListener: Relaying 'event::SetForceNextNotification' message
[2021-01-15 10:28:16 +0100] notice/ApiListener: Sending message 'event::SetForceNextNotification' to 'mgmt-icinga2-master-nodeb.mvmh.local'

MailTestY (mail sent)

[2021-01-15 10:28:21 +0100] information/Checkable: Checking for configured notifications for object 'Mailtest.mvmh.local!MailTestY'
[2021-01-15 10:28:21 +0100] notice/ApiListener: Relaying 'event::SendNotifications' message
[2021-01-15 10:28:21 +0100] debug/Checkable: Checkable 'Mailtest.mvmh.local!MailTestY' has 1 notification(s).
[2021-01-15 10:28:21 +0100] notice/ApiListener: Relaying 'event::SetForceNextNotification' message
[2021-01-15 10:28:21 +0100] notice/Notification: Attempting to send  notifications for notification object 'Mailtest.mvmh.local!MailTestY!NOC mail-test'.
[2021-01-15 10:28:21 +0100] notice/ApiListener: Sending message 'event::SetForceNextNotification' to 'mgmt-icinga2-master-nodeb.mvmh.local'

Relaying 'event::SendNotifications' message means it wants to execute the notification on the other node if I am correct, so are you sure icinga 2 really runs only on one master and does not know of two endpoints? Perhaps you can post your zones.conf.

I am asking because this would be the most simple to solve problem. If there are two endpoints configured in one zone and both are up the checks and notifications would be be split evenly between both and you would need to watch the log of both.

You can be right. The Icinga2 is up and running on both nodea and nodeb master servers. The notification relaying always goes to the nodeb according to the debug log:

[2021-01-15 10:27:11 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:27:18 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:27:56 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:28:16 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:28:20 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:28:24 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:28:26 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:28:30 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:29:09 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
[2021-01-15 10:29:23 +0100] notice/ApiListener: Sending message 'event::SendNotifications' to 'mgmt-icinga2-master-nodeb.mvmh.local'
...

But we can’t see any issue with the mailing in nodeb.

The master part of the zones.conf:

/*
 * Generated by Icinga 2 node setup commands
 * on 2018-07-27 10:43:36 +0200
 */

object Endpoint "mgmt-icinga2-master-nodea.mvmh.local" {
}

object Endpoint "mgmt-icinga2-master-nodeb.mvmh.local" {
	host = "10.110.221.15"
}

object Zone "master" {
        endpoints = [ "mgmt-icinga2-master-nodea.mvmh.local", "mgmt-icinga2-master-nodeb.mvmh.local" ]
}

Endre

Ok, this helps in narrowing down the issue because if it only fails on nodeb, investigation why it fails here is the next step to do.

My typical list of things to check:

  • Is the notification objects synced to second node? icinga2 object list --type notification
  • Is the notification feature enabled? icinga2 feature list
  • Is the notification command there / excutable / in the right context? ls -lZ /etc/icinga2/scripts
  • Is the mail command installed? which mailx
  • Is the mail being send / forwarded via the local mta? view /var/log/mail.log
  • Is there an error message for the forwarding like “not accepted”? (I have seen environments where only one node was allowed to send mail, so from the other node relaying was not accepted.)
  • Is there an error on the mail relay? (I have seen a relay misconfigured so mail would be bounced but the local mta did not get the bounce and no mailbox for the user existed.)