I monitor a couple of small clusters from a single host, “CA”. CA is a master and the cluster machines are satellites. Notification is via email, and CA is behind a single router, R. So if R dies, notification can’t happen.
Fortunately I have another machine, “NY”, outside the router. So I set up a separate Icinga instance there and have it watching R. Its notification path doesn’t use R. Problem solved.
Except I’m watching some other things on NY, and I’d really like those things to be controlled from CA and on CA’s dashboard. So what I really want is for NY to be a satellite for most checks, but for it to run a few other checks independently (like R’s liveness) and send notifications for those ones directly on its own initiative, rather than feeding them back through the master on CA.
Is that possible? I’m still an Icinga beginner, but I’ve gotten the impression that it could be done if I set up the configurations right.
I can give more details if necessary, but I’m trying to keep this short.