Hmmm, I should bring up tar the next time this happens (we are using notifications to automatically create or update tickets now).
2 things to note for anyone who joins the party:
parenting is amazing if you have it setup correctly. Unfortunately in my environment (working for an ISP), parenting doesn’t help since OSPF or BGP can pick a new route at any moment and destroy the parenting.
as of the time of this post, there is a known bug with scheduled downtimes in which they are not persistent across reloads/restarts of Icinga2 (we use director to dynamically import hosts/services, with ~20 reloads a day)
Downtimes not reapplied after a reload/restart of Icinga2 · Issue #8968 · Icinga/icinga2 · GitHub
Downtime bug aside, downtimes are the “built-in” way that Icinga would handle whether or not to send out notifications. You can mass disable notifications using the method in the above posts, or disabling a notification rule (you might do these things in a large maintenance that may impact monitoring’s ability to monitor, or like a POP router that may affect multiple customers)
You can rate limit emails using Postfix or other mail protocols, but it varies by service.
Other notification channels you might use (Slack, Pager Duty, etc…) may require a different solution, or have no solution built in.
There are some things that you can do limit the amount of notifications; first notification delays, interval between notifications, disabling re-notifications.
Tar and feathers seems like a good idea though.