I’m configuring our Icinga2 to send SMS notifications and automatically create tickets on certain bug trackers when necessary.
I’ve got my bash script that calls an API to send the SMS, and another script that calls the bug tracker’s API. I’ve configured the commands/templates and notifications accordingly, everything works perfectly.
Now, whenever there’s a problem during the API calls, my scripts retry a few times before quitting and exiting with a code different than 0.
My questions are :
How does Icinga2 behaves when that happens ?
Does it retry to send the notification ?
Can it be configured to behave differently depending on the scripts’ exit codes ?
Also from our knowledge icinga trigers only the script. That’s it. What the script is doing, is your responsibility (catch errors etc.). As Carsten wrote you will see it in logs
But what can you do in the script if’s not correctly?
You create a loop in your script which tries it as long as should to notify you, connect to bug trackers/sms etc.
You could expand your code in that way, if the notification doesn’t work, you send a message to a passive service in icinga.
The retry loop is already in place in our scripts, we’ll add a “last resort” procedure that will try to call another API to send an SMS and send an email somewhere else.