I have a case where a node A is parent of 20 nodes. Now what happened was node A went down. It was shown as red in Icinga and descendents of node A were not reachable (shown as purple If I remember correctly). I got email notification only for node A which is I want.
But what happened was when node A came up, in a span of 10-15 seconds I got a email notification for each of 20 descendants that it down.
Why did I get notification for 20 descendents? They were powered up. Could it be that A came up but it was not forwarding traffic to its 20 descendents for 10-15 seconds? So in such case is it possible to delay the sending of notification for descendents? Something like retry_interval and max_check_attempts. Do these parameters work in this reverse case when a parent node comes up and we have to check its unreachable descendents?