Only forced mail notifications are received for HOST/SERVICE

sarojamarraj · September 2, 2021, 7:39pm

Hello Friends,
I have been using Icinga for quite a long but last month we decided to build it from scratch on a new physical machine. I’m facing an issue with the mail notification, I don’t get any email, in fact, it doesn’t show anything in the logs also. When i sent notification forcefully then i receive an email and the same is logged very well in the logs as well.
I am very bad at formatting, please forgive me.

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.12.3-1)

Copyright (c) 2012-2021 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Ubuntu
  Platform version: 20.04.2 LTS (Focal Fossa)
  Kernel: Linux
  Kernel version: 5.4.0-74-generic
  Architecture: x86_64

Build information:
  Compiler: GNU 9.3.0
  Build host: runner-hh8q3bz2-project-298-concurrent-0
  OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

Operating System and version
Ubuntu 20.04.2 LTS \n \l

Enabled features: api checker debuglog ido-mysql mainlog notification

Icinga Web 2 Version
2.8.2
Git commit
8a89839af94a247ee2149b2336c73b8251b477c0
PHP Version
7.4.18
Git commit date
2020-08-17
Copyright
© 2013-2021 The Icinga Project

icinga2 daemon -C
[2021-09-02 21:34:24 +0200] information/cli: Icinga application loader (version: r2.12.3-1)
[2021-09-02 21:34:24 +0200] information/cli: Loading configuration file(s).
[2021-09-02 21:34:24 +0200] warning/config: Ignoring directory '/var/lib/icinga2/api/zones/gattaca.idainfront.se' for unknown zone 'gatta.domain.se'.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Committing config item(s).
[2021-09-02 21:34:24 +0200] information/ApiListener: My API identity: gatta.domain.se
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 NotificationComponent.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 49 Hosts.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 2 NotificationCommands.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 2 FileLoggers.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 109 Notifications.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 IcingaApplication.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 4 HostGroups.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 CheckerComponent.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 3 Zones.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 Endpoint.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 2 ApiUsers.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 ApiListener.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 143 CheckCommands.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 TimePeriod.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 1 UserGroup.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 3 Users.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 142 Services.
[2021-09-02 21:34:24 +0200] information/ConfigItem: Instantiated 4 ServiceGroups.
[2021-09-02 21:34:24 +0200] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2021-09-02 21:34:24 +0200] information/cli: Finished validating the configuration file(s)

dgoetz · September 3, 2021, 6:54am

Force ignores the filters, so every user with an assigned notification will get it.
Does the icinga.log tell you that no notification exists for a problem?
If yes, there is probably some error in the filters on notification and/or user.
You can also post an example of this objects (including templates if necessary) and perhaps we can find an error.

sarojamarraj · September 3, 2021, 9:01am

Thanks for the reply, below are the Icinga logs. Here first I initiated “| grep notification” on the server cmd then in Icinga GUI clicked on check now for critical memory check still nothing was in the log. Then clicked on send custom notification without forced still nothing in the logs. Then resent custom notification but this time I checked the mark FORCED and I received the mail alert while the same was been logged.

root@gattaca:~# tail -f /var/log/icinga2/icinga2.log | grep Notification
[2021-09-03 10:43:32 +0200] information/Notification: Sending 'Custom' notification 'gold.domain.se!linux_memory_check!service_notify_apply' for user 'AMAR'
[2021-09-03 10:43:32 +0200] information/Notification: Sending 'Custom' notification 'gold.domain.se!linux_memory_check!service_notify_apply' for user 'johh'
[2021-09-03 10:43:32 +0200] information/Notification: Sending 'Custom' notification 'gold.domain.se!linux_memory_check!service_notify_apply' for user 'Ken'
[2021-09-03 10:43:32 +0200] information/Notification: Completed sending 'Custom' notification 'gold.domain.se!linux_memory_check!service_notify_apply' for checkable 'gold.domain.se!linux_memory_check' and user 'AMAR' using command 'mail-service-notification'.
[2021-09-03 10:43:32 +0200] information/Notification: Completed sending 'Custom' notification 'gold.domain.se!linux_memory_check!service_notify_apply' for checkable 'gold.domain.se!linux_memory_check' and user 'Ken' using command 'mail-service-notification'.
[2021-09-03 10:43:32 +0200] information/Notification: Completed sending 'Custom' notification 'gold.domain.se!linux_memory_check!service_notify_apply' for checkable 'gold.domain.se!linux_memory_check' and user 'johh' using command 'mail-service-notification'.

Time period template and time period config

zones.d/director-global/timeperiod_templates.conf
template TimePeriod "9to5" {
    import "legacy-timeperiod"
    ranges = {
        "friday"	= "09:00-17:00"
        "monday"	= "09:00-17:00"
        "thursday"	= "09:00-17:00"
        "tuesday"	= "09:00-17:00"
        "wednesday"	= "09:00-17:00"
    }
}

------------------------------------------------------------------

zones.d/director-global/timeperiods.conf
object TimePeriod "office_hours" {
    import "legacy-timeperiod"
    import "9to5"

    ranges = {
    }
}

Users group, template and config

zones.d/master/usergroups.conf
object UserGroup "icinga2admin" {
}

---------------------------------------------------------
zones.d/director-global/user_templates.conf
template User "icinga2_admin" {
    enable_notifications = true
    period = "office_hours"
    states = [ Critical, Down, OK, Unknown, Up, Warning ]
    types = [ Custom, Problem, Recovery ]
    groups = [ "icinga2admin" ]
}
-------------------------------------------------------------
zones.d/master/users.conf
object User "AMAR" {
    import "icinga2_admin"

    email = "amar@domain.se"
    enable_notifications = true
    period = "office_hours"
    states = [ Critical, Down, OK, Up, Warning ]
    types = [ Custom, Problem, Recovery ]
    groups = [ "icinga2admin" ]
}

Notification template for host & service and config file

zones.d/master/notification_templates.conf
template Notification "host_notify" {
    command = "mail-host-notification"
    period = "office_hours"
    states = [ Down, Up ]
    types = [ Custom, Problem, Recovery ]
    user_groups = [ "icinga2admin" ]
    users = [ "AMAR" ]
}

zones.d/master/notification_templates.conf
template Notification "service_notify" {
    command = "mail-service-notification"
    period = "office_hours"
    states = [ Critical, OK, Warning ]
    types = [ Custom, Problem, Recovery ]
    user_groups = [ "icinga2admin" ]
    users = [ "AMAR" ]
}

Notification apply:-

## zones.d/master/notification_apply.conf

apply Notification "host_notify_apply" to Host { import "[host_notify](https://gattaca.domain.se/icingaweb2/director/notification?name=host_notify)" assign where "critical_servers" in host.groups states = [ Down, Up ] types = [ Custom, Problem, Recovery ] user_groups = [ "icinga2admin" ] users = [ "AMAR" ] }

## zones.d/master/notification_apply.conf

apply Notification "service_notify_apply" to Service { import "[service_notify](https://gattaca.domain.se/icingaweb2/director/notification?name=service_notify)" assign where "critical_service" in service.groups states = [ Critical, OK, Warning ] types = [ Custom, Problem, Recovery ] user_groups = [ "icinga2admin" ] users = [ "AMAR" ] }

I’m more interested only to get alert for my critical host group and critical service group

dgoetz · September 3, 2021, 9:27am

The filters look good, so I think the timeperiod is the problem.

zones.d/director-global/timeperiods.conf
object TimePeriod "office_hours" {
    import "legacy-timeperiod"
    import "9to5"

    ranges = {
    }
}

If I am correct this imports the ranges from “9to5”, but then overrides all ranges. So changing it to

zones.d/director-global/timeperiods.conf
object TimePeriod "office_hours" {
    import "legacy-timeperiod"
    import "9to5"
}

or

zones.d/director-global/timeperiods.conf
object TimePeriod "office_hours" {
    import "legacy-timeperiod"
    import "9to5"

    ranges += {
    }
}

should work.

If the first one is what you need you can also forgo the template. The second one would allow to overwrite or add ranges if needed in the future.

sarojamarraj · September 3, 2021, 10:34am

Thanks for the quick reply, you really made my day, the issue is resolved. Many thanks to you

I added the range as per your suggestion and it worked!

zones.d/director-global/timeperiods.conf
object TimePeriod "office_hours" {
    import "legacy-timeperiod"
    import "9to5"

    ranges = {
        "friday"	= "09:00-17:00"
        "monday"	= "09:00-17:00"
        "thursday"	= "09:00-17:00"
        "tuesday"	= "09:00-17:00"
        "wednesday"	= "09:00-17:00"
    }
}

NithyaR · December 27, 2021, 10:19am

Hi all, i have an issue.
i have created a contact, contact group and i have set a notification to send mail alerts whenever the server is down . In the UI, it says notification sent, but i never receive the mail.
any help would be appreciated.

Thanks,
R.NITHYA

NithyaR · December 27, 2021, 10:52am

i have configured contacts, contactgroup and notification for mail alerts . In the UI it says notification sent, but i dont receive any mail.

Can you please let me know if i missed out anything else.

hthuemin · February 3, 2022, 3:29pm

Thank you so much, that was the solution for me too, cool