Not all notifications are sent out for problems

Hello,

I encountered that when I assign more notification opjects to a service. Not all notifications are sent out on relevant state changes.
Here is my configuration:

apply Notification "mail-icingaadmin-24x7" to Service {
  import "mail-service-notification"
  user_groups = [ "ALERT-DG-MAIL" ]

  interval = 2h

  //vars.notification_logtosyslog = true

  assign where service.enable_notifications && host.vars.sla == "BO2105_SRV_24x7"

}

Above any problem for services with host.var.sla = “BO2105_SRV_24x7” and default notifications enabled should send an email.
Below is an extra object for sending sms in extra cases:

apply Notification "sms-windows" to Service {
  import "sms-service-notification"
  user_groups = [ "ALERT-DG-SMS-WIN" ]

  interval = 1h

  states = [ OK, Critical, Unknown ]
  types = [ Problem, Recovery, FlappingStart, FlappingEnd ]

  period = "non-workhours"

  assign where service.enable_notifications && host.vars.os == "Windows" && host.vars.sla == "BO2105_SRV_24x7"
}

Now looking at the services history I see that only the sms was sent out. I still miss mails from the first mail object.
The service itself in icingaweb2 tells me all related contacts (mail contacts + sms contacts) but notifications only were sent to sms contacts when hard state changes to critical.
Is there something that I missunderstood?
I need notifications generally via mail and only specific states via sms. But always both at the same time.
Any hints are kindly apreciated
Thanks and kind regards

Ralf

Hi Ralf,

are we talking about a single node or is this a clustered setup with 2 nodes?

Only thing that I see is that you have different intervals configured - although this should imho just affect the re-notifications.

I would do two tests and see if the behaviour remains the same:

  • set the interval to 1h for both notifications
  • add times.begin = 1 to the notifications

This is just a shot in the dark now but what I would test next.

For additional tests trying to reproduce it, it would be helpful to know which icinga2 version you’re running.

BR,
Marcel

1 Like

Hi Marcel,

Icinga version is r2.10.5-1
Its a clustered setup with two nodes in the master zone using galera cluster as backend and one single satellite in a subzone.
It is not only the subzone affected.

I changed the parameters as you mentioned and will now wait what happens. :slight_smile:
Thanks and best regards

Ralf

Hi Ralf,

triggered notifications should also be mentioned in the logs. Now, as you have 2 nodes in the same zone it can happen that nodeA sends the SMS while nodeB sends the Mail. Did you check the logs as well?

Icingaweb2 grabs all the information from the database of icinga2. I would just double-check that both nodes can write to the database without issues. (This does not apply if you have enable_ha = True in the ido_mysql.conf. In that case all writes will be performed by the same node).

Let us know if you have any news. :slight_smile:

Hi Marcel,

enable_ha is set to 1 on both nodes.
Unfortunately, there are no mails sent out. I can check this also in the icingaweb2 service history. Only SMS receipients are mentioned in the notification mark. At least all recipients should be notified that are shown in the contact(groups) of the notification section for the service itself. Which is not the case
That is why I guess its not a configuration mistake.

However, changing the interval to the same value and adding the time.begin parameter had no further effect.
Is there any other option, that I can check?
Thank you and best regards

Ralf

Hi Ralf,

it’s a bit hard and I can just fire some more shoots into the dark. :sweat_smile:

So, what I would do:

  • ensure that the notifications feature is enabled on both nodes icinga2 feature list
  • check that the expected users, usergroups and notifications exist via the icinga2 cli tool (--type user & --type usergroup and --type notification)

The object list for the notifications will also show you the enabled notifications per service check.

I hope this is helpful.

Check the icinga2 bash completion tool to make your life on the cmd line a bit easier. https://icinga.com/docs/icinga2/latest/doc/11-cli-commands/#icinga-2-cli-bash-autocompletion

1 Like

Hey,

there was indeed notifications disabled on the second node that caused mails not being send from this host. I wonder since i never enabled notifications on the first node…

Now at leased come more messages from where I was still missing.
The SMS / Mail difference I could not yet verify but will get back soon if the problem has now solved also.
Thanks for your hints and kind regards

Ralf

1 Like