Automated downtime for new hosts and services

Anyone doing this so new hosts/services don’t alert if there is a problem to work out with new additions? Today, I push out new checks via the Director then go find the host/service and put in sched downtime quickly before any false alerts happen.

I suppose a log file could be monitored with swatch to trigger a script that does this via the API but I bet there is a better way to do this.

What do others do about new hosts/services to prevent false alerts? Different host/service groups or contacts?

I’m also looking for that feature…

I made a script to set automatic downtime via eventhandler. Its not for new hosts only but you can give it a try. After hosts are working just remove the eventhandler from the host object.

1 Like

Thanks man! Looks good! Any ideas how to get a list of a recently added hosts? :slight_smile:

The script looks amazing. Nice work. I am a little confused as to how it would be used with an eventhandler like this. Maybe I don’t have enough experience with eventhandlers but I think what you are suggesting is to create all new hosts with this eventhandler and then when it goes green after a “bake in” period, remove the eventhandler. Unless I am missing something, automating this still seems like some key pieces are missing.

I was thinking about some kind of deploy-hook feature that could allow a script to launch. The script could check the deploy details from the database and apply a specified fixed downtime to the new hosts and services.

I guess I could poll the director db for deployments but that’s not very efficient. I have had a lot of success using swatch to execute scripts based on log files so maybe that would be my approach for now and use Python with the icinga2api.client module which makes using the API very simple.

Still seems like this would be a common feature that others would find useful to set downtime on new hosts/services to give time to fix any networking/firewall/nrpe/nsclient++/SNMP/certificate/etc issues before alerting.

Thanks,
Dave

1 Like

I would probably setup something like this:

downtimes.conf:
apply ScheduledDowntime “NewHost-downtime” to Host {
author = “YourName”
comment = “Scheduled downtime for new hosts”

ranges = {
monday = “00:00-23:59”
tuesday = “00:00-23:59”
wednesday = “00:00-23:59”
thursday = “00:00-23:59”
friday = “00:00-23:59”
saturday = “00:00-23:59”
sunday = “00:00-23:59”
}

assign where host.vars.ignoreDowntime == true
}

Then set the ignoreDowntime field to true when creating the object (or on the template, if using director). And once your want the downtime to actually be taken serious, set it to false.

1 Like

I would like this to be 100% automated as I don’t manually add/remove hosts in the Director. We have various Anisble playbooks and import/syncs in the Director that are managed by other teams outside of the Director.

Would we change the default template to have host.vars.ignoreDowntime == true, then use the API later to set the value false on the host/service later after it’s “ready”?

That is probably what I would do. But then again, I am fairly new to Icinga and I have limited servers I manage.

If there is a downside to my idea, I am sure someone will point it out :slight_smile: