We have begun receiving unexplained DOWNTIMESTART and DOWNTIMEEND email notifications for one particular host, despite not having this configured anywhere we can find. It happens any time the host enters or exits any scheduled downtime (either regularly configured ones, or ad-hoc temporary ones added from the web ui)
I have read the docs and am aware you can configure the notifications you want on the notification or user object, and there is a notification type filter for DowntimeStart or DowntimeEnd, but we are not using those anywhere. I have even done a recursive grep to ensure this, eg grep -ir downtimestart /etc/icinga2/, and nowhere in our config is that found.
We have around 10 hosts of the same type who’s configuration files are basically identical copies. However this one particular host, which we will refer to as host07 is receiving these notifications. The only known difference in config is that this host does have regularly scheduled downtime and the others don’t.
We have even used the following command on 2 hosts to see how their configuration files were built, (host07 which does trigger the notifications and host08 which does not), and after even a lot of parsing and sorting of the output, we did not see any difference between the notification config of these two.
icinga2 object list --type Notification --name 'host07*'
While we have multiple notifications setup, the mail-host is the one configured to send to the specified email address and thus seems to be the notification in question
We tried a direct comparison of both hosts like this, for that specific notification, and the results are identical for both (so I will just share the one for the host that triggering the notices)
~ root@icinga01 34> icinga2 object list --type Notification --name 'host07.abc!mail-host'
Object 'host07.abc!mail-host' of type 'Notification':
% declared in '/etc/icinga2/conf.d/notifications.conf', lines 98:1-98:38
* __name = "host07.abc!mail-host"
* command = "mail-host-notification"
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 240:3-240:36
* command_endpoint = ""
* host_name = "host07.abc"
% = modified in '/etc/icinga2/conf.d/notifications.conf', lines 98:1-98:38
* interval = 1800
* name = "mail-host"
* package = "_etc"
% = modified in '/etc/icinga2/conf.d/notifications.conf', lines 98:1-98:38
* period = "24x7"
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 247:3-247:17
* service_name = ""
* states = [ 16, 32 ]
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 242:3-242:23
* templates = [ "mail-host", "mail-host-notification" ]
% = modified in '/etc/icinga2/conf.d/notifications.conf', lines 98:1-98:38
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 239:1-239:46
* times = null
* type = "Notification"
* types = [ 32, 16, 64, 8, 128, 256, 4 ]
% = modified in '/etc/icinga2/conf.d/templates.conf', lines 243:3-245:29
* user_groups = [ "techops" ]
% = modified in '/etc/icinga2/conf.d/notifications.conf', lines 101:3-101:29
* users = null
* vars = null
* zone = ""
Note - while I couldn’t find documentation of the integer values for “types”, having poked around in /var/cache/icinga2/icinga2.vars, I have reason to believe downtimestart is 1, and downtimeend is 2, which we don’t see in this output, as I would expect if they are not enabled.
Here are the relevant bits of the config
conf.d/notifications.conf
apply Notification "mail-host" to Host {
import "mail-host-notification"
user_groups = [ "techops" ]
assign where host.name
}
templates.conf
template Notification "mail-host-notification" {
command = "mail-host-notification"
states = [ Up, Down ]
types = [ Problem, Acknowledgement, Recovery, Custom,
FlappingStart, FlappingEnd,
DowntimeRemoved ]
period = "24x7"
}
users.conf
object User "techops-user" {
import "generic-user"
display_name = "TechOps User"
groups = [ "emergency", "techops" ]
email = "xxx@xxxxxx.abc"
}
We also tried enabling debug mode at one point, and while the logs were about 100MB per minute, nothing useful was found
icinga2 feature enable debuglog && /etc/init.d/icinga2 restart
After lots of parsing of the logs we have never found an explanation why it is sending these notifications.
The only difference seen in the logs is on a host which does not have the issue, we see a message like this, which isn’t followed by anything relvant
[2023-06-01 19:50:01 -0700] information/Checkable: Checking for configured notifications for object 'host08.abc'
Whereas on the host that does have the problem, it shows the same message, and immediately shows messages that it is sending various notifications (though this seems false since it doesn’t send all of these)
[2023-06-01 18:50:15 -0700] information/Checkable: Checking for configured notifications for object 'host07.abc'
[2023-06-01 18:50:15 -0700] information/Notification: Sending notification 'host07.abc!pager-host-office-hours' for user 'techops-user'
[2023-06-01 18:50:15 -0700] information/Notification: Sending notification 'host07.abc!pager-host-office-hours' for user 'john'
[2023-06-01 18:50:15 -0700] information/Notification: Sending notification 'host07.abc!pager-host-office-hours' for user 'smith'
[2023-06-01 18:50:15 -0700] information/Notification: Sending notification 'host07.abc!mail-host' for user 'techops-user'
A couple important notes.
This did begin recently after having temporarily implemented a different notification, and later on reverting it. However that change did not touch any of the relevant config for this host or these notification.
Also our icinga version is very out of date, so it might be a bug. There are plans to upgrade or rebuild completely soon, but in the mean time are seeing if there is a solution.
~ root@icinga01 1> icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.4.4-1)
...
Platform: Gentoo
~ root@icinga01 2> icinga2 feature list
Disabled features: compatlog debuglog gelf graphite icingastatus livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command ido-mysql mainlog notification
~ root@icinga01 3> icinga2 daemon -C
...
information/cli: Finished validating the configuration file(s).
Note - some warnings were seen in config validation about ignored config objects, though none appear relevant to this host, downtimestart, or these notifications in question.
Thank you in advance for any assistance. We are running low on troubleshooting ideas at this point.