Service notifications sent while host in downtime

[poncho]

Hi,
I have some cloud servers running that start and stop on demand. This is triggered by a script. This script now also adds or removes the downtime in Icinga for these hosts. And this does work as far as I can see. But when the host is still in downtime Icinga sends problem notifications via email.

My script works as follows:

• Server shutdown:
  • Add downtime in Icinga
  • Shutdown the server
• Server startup:
  • Start server
  • Remove downtime
  • Add new downtime with duration=2min (give it some time to start up)

See the screenshots down below:

• 9:25:08 host downtime scheduled
• 9:25:32 service soft down
• 9:26:12 host is soft down in Icinga
• 9:27:51 service hard down -> notification sent
• 9:28:11 host is hard down in Icinga

How to prevent those notifications?

History of the host:

Service:

[belastingvormulier (William van Beek)]

Hi @poncho,

Perhaps, you should use the API to check the host state first in icinga? then if the host is OK remove the DT ?
That way you dont have to add a 2 min downtime and keep it simple for you and the icinga host.

-> DT shutdown
-> shutdown
-> startup
-> Check Icinga host status (ping ok?)
-> remove DT

How are you scheduling the DT for this host? It seems you are missing the ping service in that command?

see the topic here for a example:

Set downtime for all services of one host via API

Hi guys, I’m trying hard to set a downtime for all services of one host via API. In the webfrontend there is a seperate checkbox “all_services” which is working fine, but how can I use it through the API? I tried for example: curl -k -s -u 'actionuser' -H 'Accept: application/json' -X POST 'https://SERVER:5665/v1/actions/schedule-downtime?type=Host&host='HOST'&all_services=1' -d '{ "author": "ME", "comment": "TEST", "notify": true, "pretty": true, "start_time": 1561012086, "end_time": 1561033…

[poncho]

The question here is: If a host has a downtime active, do the services respect it or do they need their own downtime too?

[belastingvormulier (William van Beek)]

@poncho

The question here is: If a host has a downtime active, do the services respect it or do they need their own downtime too?

Services need their own I am afraid That way you should no longer get the notifications

[poncho]

Wondering why. Makes no sense to me. But OK, I will add this too and check again.

[poncho]

I still get notifications from services. I think I have read about notifications that will be send if the downtime started after the problem state. But what now? I can add and remove downtimes but I can not edit them. When I start the downtime I do not know how long it has to be.

[belastingvormulier (William van Beek)]

Hi there,

You can try to start a down time in a flexible state instead of a fixed state before you shutdown the server. That way the downtime starts the moment the problem starts being there. and lasts for the amount of time you specify. see:

https://icinga.com/docs/icinga2/latest/doc/08-advanced-topics/#flexible-downtime

@poncho

I can add and remove downtimes but I can not edit them.

Yes to edit you need to just remove and add!

[poncho]

OK, I added downtimes for host and all services and adjusted the downtime duration and now it works. I don’t get notifications anymore. Thanks!