Service notifications sent while host in downtime

Hi,
I have some cloud servers running that start and stop on demand. This is triggered by a script. This script now also adds or removes the downtime in Icinga for these hosts. And this does work as far as I can see. But when the host is still in downtime Icinga sends problem notifications via email.

My script works as follows:

  • Server shutdown:
    • Add downtime in Icinga
    • Shutdown the server
  • Server startup:
    • Start server
    • Remove downtime
    • Add new downtime with duration=2min (give it some time to start up)

See the screenshots down below:

  • 9:25:08 host downtime scheduled
  • 9:25:32 service soft down
  • 9:26:12 host is soft down in Icinga
  • 9:27:51 service hard down -> notification sent
  • 9:28:11 host is hard down in Icinga

How to prevent those notifications?

History of the host:

Service:
service

Hi @poncho,

Perhaps, you should use the API to check the host state first in icinga? then if the host is OK remove the DT ?
That way you dont have to add a 2 min downtime and keep it simple for you and the icinga host.

-> DT shutdown
-> shutdown
-> startup
-> Check Icinga host status (ping ok?)
-> remove DT

How are you scheduling the DT for this host? It seems you are missing the ping service in that command?

see the topic here for a example:

The question here is: If a host has a downtime active, do the services respect it or do they need their own downtime too?

Services need their own I am afraid :slight_smile: That way you should no longer get the notifications

Wondering why. Makes no sense to me. But OK, I will add this too and check again.

I still get notifications from services. I think I have read about notifications that will be send if the downtime started after the problem state. But what now? I can add and remove downtimes but I can not edit them. When I start the downtime I do not know how long it has to be.

service

Hi there,

You can try to start a down time in a flexible state instead of a fixed state before you shutdown the server. That way the downtime starts the moment the problem starts being there. and lasts for the amount of time you specify. see:

Yes to edit you need to just remove and add!

OK, I added downtimes for host and all services and adjusted the downtime duration and now it works. I don’t get notifications anymore. Thanks!