Sending Icinga2 notifications to Slack

Meta Icinga Tutorials and Guides
, ,
1

Autor: @watermelon
Original: https://monitoring-portal.org/t/sending-icinga2-notifications-to-slack/2901

Integrating Icinga2 with Slack for notifications

Source:
GitHub - nisabek/icinga2-slack-notifications

Introduction

It’s good to have more than one form of notification with Icinga alerts, whether that’s through email, SMS, on-screen (dashboard), and through messaging applications like WhatsApp, Viber, and Slack. I wasn’t able to integrate WhatsApp with Icinga notifications because a recent update (as of 6/15/18) supposedly broke the library (GitHub - Yowsup) that allows for transmission of automated messages to WhatsApp. As for Viber, they have been making some changes as well and are removing the previous method of “chatbotting” (with Public Accounts) and moving forward with a new feature they call Viber Communities. This feature has not come out yet, so I decided to try to work with Slack. Perhaps in the future, I can get to writing guides for Viber and WhatsApp integrations.

I found that somebody had already integrated Slack into their workflow and shared it on GitHub, so I decided to write up a guide outlining the specific steps for how to get everything to work.

Requirements

  • Assuming you have Icinga2 up and running, all you need is curl!

Installation

Setting Slack up

  1. Create a Slack workspace
    • Within the workspace, create a new channel for Icinga notifications
  2. Go to https://api.slack.com/ to create an “App” for your workspace
    • This will be access the Slack API
    • Make sure it’s created under the newly created Slack workspace
  3. Add a web hook to the new Slack App
    • Look for the Features tab in the left-hand column and go to Incoming Webhooks
    • Switch the Active Incoming Webhooks slider to “On”
    • Click Add New Webhook to Workspace at the bottom of the page
      • Authorize the webhook to post to the previously created Icinga notifications channel within Slack
  4. Test out the web hook to make sure it works by using the sample curl request provided after the web hook is created
    • The command will look something like this:
curl -X POST -H 'Content-type: application/json' --data '{"text":"Hello, World!"}' https://hooks.slack.com/services/BLAHBLAH/BLAHWATERMELONBLAH/BLAHBLAHBLAH
  • Then, you should see a message in the Icinga notifications channel saying “Hello, World!”

Adding the config files

  1. Clone the repository containing the config files for the Slack notifications to your Icinga server
    • git clone https://github.com/nisabek/icinga2-slack-notifications.git
    • Note: there is an alternative way to install the icinga2-slack-notifications package with Debian, details located on nisabek’s repo
  2. Copy the slack-notifications-user-configuration.conf.template
    • cp slack-notifications/slack-notifications-user-configuration.conf.template slack-notifications/slack-notifications-user-configuration.conf
  3. In slack-notifications-user-configuration.conf, change the lines that require your Slack web hook URL as well as your Icinga2 base URL
template Notification "slack-notifications-user-configuration" {
    import "slack-notifications-default-configuration"

    vars.slack_notifications_webhook_url = "<YOUR SLACK WEBHOOK URL>, e.g. https://hooks.slack.com/services/TOKEN1/TOKEN2"
    vars.slack_notifications_icinga2_base_url = "<YOUR ICINGA2 BASE URL>, e.g. http://icinga-web.yourcompany.com/icingaweb2"
}
  • Note: You’ll find the web hook URL in that sample curl command from before
  1. Ensure the permissions are correct for the config files
chmod -R root:root slack-notifications/*
chmod 0640 slack-notifications/*
  • Note: This may vary depending on your setup
  1. Ensure the checker, command, and notification features are enabled through icinga2 feature list
    • If you need to enable a feature, do icinga2 feature enable FEATURE_NAME

Customizing to your environment

  1. To enable Slack notifications on a Host or Service, simply add the slack_notifications variable set to enabled to that object
    • vars.slack_notifications = “enabled”
    • You could do this at the template level as well to have your Host and Services inherit this property
  2. If you have a distributed environment, you’ll probably want to put this config (all three files in the slack-notifications directory) within the global-templates zone. Otherwise, you can put it in the conf.d directory.

Conclusion

And that’s it! You should now be able to test (using a custom notification for example) the sending of notifications directly to Slack via curl and the Slack API.

FAQ

There is more information about how everything works at nisabek’s icinga2-slack-notifications repository if you were curious. Lemme know if you have any questions and I can try to answer them.

Notification use wrong options / variables

If you can’t override options or variables to your script by setting macros in Icinga 2, make sure that you don’t have any defaults or overrides in your notification script. While this is rather obvious we see it a lot that people tend to forget about the contents of the script itself.

2

Hi, splendid write up! I’m curious however how one gets this one to work with director. I’ve played around a quite a bit today. Im however unable to override the variables:

vars.slack_notifications_webhook_url
vars.slack_notifications_channel

I’m trying to replace those variables using fields from director (in order to alert services/hosts to different channels), but to no avail. Any idea on how to dynamically assign those values with fields from director?

3

Could you show us how you tried changing them? Especially what the preview tab of your objects look like?

It should be no problem setting or overriding them in service or host objects.

4 
object Host "**************" { 
	display_name = "rabbitmq" 
	address = "*********" 
	check_command = "[hostalive4](*************)"
	max_check_attempts = "10" 
	check_period = "always" 
	check_interval = 5m
	retry_interval = 1m
	enable_notifications = true
	enable_active_checks = true
	enable_passive_checks = false
	enable_flapping = true
	enable_perfdata = true
	vars.environment = "Production"
	vars.server_role = "unspecified"
	vars.slack_notifications = "enabled"
	vars.slack_notifications_channel = "*********"
	vars.slack_notifications_webhook_url = "https://hooks.slack.com/services/***********" 
}

I censored out some bits with ****

The slack variables are indeed correct, however when I test a notification the default slack configuration vars shows up instead. So it does not get overridden.

These variables are set using custom properties for each host / service.

5

That should work, indeed.

You should enable the debuglog, run some notifications and then use grep to search for the exact command being used (unless you have a full blown logmanagement solution like Elastic Stack)

6

I managed to solve it, and I’m rather embarrassed to tell how. I forgot that I’d put some default values in slack-notifications-user-configuration.conf. Once I commented out those vars, everything worked as it should. :blush:

2 Likes
7

Please use the “solution” checkbox on the post that answered your question.

You’ll achieve two things:

  • Give kudos to the person who solved your problem
  • Show others that there is no more issue and they don’t have to read through your whole thread just to find that there’s no more they can do
8

There is no solution box. Maybe we should put this in the FAQ part of this HOWTO instead? :slight_smile:

9

Sorry, my bad. I didn’t realize that you replied to a wiki article.

Yes, maybe, we should add this to the article. I’ll take care of that.

edit: I just added it. Be assured, we had some support tickets about this exact matter, too. So, you’re not the only one. :wink:

1 Like
10

Hi, I we have several servers belonging to different projects in our Icinga solution. Can anyone tell if it would be possible to send icinga notifications to different slack channels depending on which project the server belongs to?