Icinga2 Director - automatic email Notifications

My problem is that I don’t get automatic email notifications from Icinga.

A MANUALLY invoked notification from the Dashboard with a checked box works for me: “Forced” (The notification is sent out regardless od tim restrictions and whether or not notifications enabled.)

From the above, I think that the error is only in the Director settings. Mail sender (postfix) is functional.

I’m missing the other pair of eyes though. I am not able to find the error myself, even with the passage of time.
I went through everything not once, twice, but 20 times.

Some configuration details although I suspect the problem is in the Director settings:
I run Icinga on a Docker container in OpenShift:

  • Director version: 1.9.1
  • Icinga Web 2 version and modules: 2.11.1 (director, doc, grafana, incubator, ipl, monitoring, reactbundle)
  • Icinga 2 version: IMAGE: jordan/icinga2 ver. r2.13.5-1
  • Operating System and version: ContOS 7
  • I use postfix as a mail server

I’ve already gone through this entire forum, as well as the official and unofficial guides. But he didn’t find a solution.
For example:
https://icinga.com/docs/icinga-2/latest/doc/03-monitoring-basics/#notifications
https://ws.learn.ac.lk/wiki/NSM2021/Agenda/AlertsIcinga2Email#no1

Something I can clarify, or someone who encountered the same error?
I believe that the error will be somewhere between the hosts-notification service setting, I just don’t know where.

If forced works, then maybe a filter on the notification attached to the server or service or on the user called by the notification isn’t letting the email getting send.

Yes Yes, there will be a mistake somewhere.
The user has set:
Send notifications: Yes
Disabled: No

However, I am not sure if it is necessary to have some “Custom properties” set for the user to connect with Hosts and Services…

what are the state and transition filters applied to the user?

I have the following set via template user:

I have it set up like this:

You don’t care for host UP & Down messages?!

Since there is currently no messages at all… It probably doesn’t matter :wink:
However, I monitor the ping on the service - this is a more important parameter for me than up & down :wink:

If the server goes down and the host check detects this before one of the services fails, you will not get the notifications of the services because the host is automatically the parent of the services. This is to stop a flood of notifications.

1 Like

Ah, I understand. Useful. I will add to the user template.
The question is still why the notification does not go. Can I add some documents for research?

Can you enable the debug log, restart icinga, provoke a notification (one normal and one forced), disable debug log, restart icinga?

This way you should see in the log, what happens and what is the difference.

1 Like

Good point, thank you. In about 60 seconds, I got a huge amount of “spilled tea” type data.
I tried searching for email addresses in it using fce grep, but I only found those emails that actually arrived in the mailbox. (Manual recalled.)
The assumption that the error will be in the notification settings itself is confirmed.
Postfix is ​​functional, both automatic notifications from the POD and triggered notifications from the Icinga Dashboard work.
The only thing that doesn’t work are automatic notifications on service and hosts.

I think you will not find the email if it is filtered out. First find the one that got triggered with force to find a pattern to locate the one without force and compare.

Also try to post filters of the notification and user objects. You can find them via command line:

icinga2 object list --type notification --name 'HOST!SERVICE' | grep 'states\|types\|users\|user_goups\|period'
icinga2 object list --type user --name 'USERNAME' | grep 'states\|types\|groupss\|period'

https://icinga.com/docs/icinga-2/latest/doc/15-troubleshooting/#notifications-troubleshooting

2 Likes

As @rivad said, the log will not contain the mail address. But it will contain the username. Or you grep for “notification”, that should yield more results.

The CLI commands will give specific objects for service or user, showing if they contain the desired configuration.

Please also share your config of the Notification Template, the notification apply rule and the user object via the Directors preview tab.
Click on “Show resolved” to display the full configuration with fields from any imported templates as well:
image

Small correction

not if filtered with --name

1 Like

Great. Thank you very much. This moved us.
It looks like the configuration set via Director won’t save for some reason.
Because the configuration obtained via CLI contains different data :frowning:
That could be the reason why I can’t find the error.
Because the configuration entered via Director corresponds to the instructions.

CLI:
root@icinga-204-vqg6n:/# icinga2 object list --type user --name ‘Vojtech’ | grep ‘states|types|groupss|period’

  • period = “TimePeriod 24/7/365”
  • states = [ “Critical”, “Unknown”, “Warning” ]
  • types = [ “Custom”, “DowntimeStart”, “FlappingStart”, “Problem”, “Recovery” ]

root@icinga-204-vqg6n:/# icinga2 object list --type notification --name ‘host!ping4!Service mail notification’ | grep ‘states|types|users|user_groups|period’

  • period = “TimePeriod 24/7/365”
  • states = [ “Critical”, “OK”, “Unknown”, “Warning” ]
  • types = [ “Acknowledgement”, “Custom”, “DowntimeEnd”, “DowntimeRemoved”, “DowntimeStart”, “FlappingEnd”, “FlappingStart”, “Problem”, “Recovery” ]
  • user_groups = null
  • users = [ “Vojtech” ]

UI - Director:

It seems that what I enter in the Director interface does not update at all.
I do see the newly entered data in the Director.
But the background doesn’t really change at all.
Newly entered data into DIector is not visible either. :frowning:

Did you deploy the pending changes in the Director?
Did the deployment run without errors?

The deployment is pending for several days… :disappointed_relieved:

Check that the background daemon is running. It is responsible for the config deployment:
https://icinga.com/docs/icinga-director/latest/doc/75-Background-Daemon/

Thanks thanks. I also worked on this link in the meantime - when I googled why Deploy is still in pending status. :wink:

Of course, Background-Daemon is not running. But I don’t understand why.

If I read correctly, it’s “new” since version 1.7 Background Daemon - Icinga Director And it’s already quite old.

I am using this latest image
Docker Hub - that there would be a bug in it?

I decided to be radical yesterday. I threw away weeks of work.
I created a new docker image, ran it in a container.

The result is that now I don’t want notifications at all. So it looks like I can start debugging again from the beginning. :frowning_face:
So I’ll pause here for a while before I get somewhere further through the settings. Because now I’m weeks back and nothing works at all :frowning:

Ah, docker… :smiley:
I have little experience there (just created some images/containers myself), and none with the linked image.

Might be a “mean” suggestion, but I’d say setup the system “by hand”/step-by-step at first to understand it and then add docker as another layer to make things more complex^^

I personally see no real benefit of running the icinga2 core and the database in a container.
An agent and the webinterface maybe, but it complicates things, especially when you (like me) have not much experience with containers.

1 Like