How to call a webhook when snmp_storage triggers a warning

Icinga 2
1

I have my master icigna2 with configuration at /etc/icinga2/zones.d/example_zone with the below file example_hostname.conf

object Host "example_hostname" {
  import "generic-host"
  vars.os = "Linux"
  address = "x.x.x.x"
  vars.location = "Zone"
  vars.snmp_oid = ".1.1.1.1.1.1.1.1.0"
  vars.snmp_storage["Storage"] = {
    snmp_warn = "90"
    snmp_crit = "95"
  }
}

The above works fine.
Now what I would like to set up is an event handler that calls a webhook once a warning is triggered for only this specific host.
How can this be done, please? I tried following the documentation but couldn’t figure it out.

2

Hi & welcome,

Common way is to create a new NotificationCommand.

BTW: For better readability we always love to see posts formatted according to this guideline.

1 Like
3

+1 for notification commands;

on our old Icinga 1.x (just for use of internal monitoring), we have a notification command that calls a webhook into Slack for slack notifications. I scrape and store an email to user ID in a redis db – the notification send the email to the command, looks up the user ID in redis, and sends a slack message.

Not a webhook, but we also use a custom notification to send alerts into a rabbitmq queue to be processed to create or update tickets as well.

Notification commands don’t have to “notify” at all per se – it can call a script that runs reboot now for all it cares.

You might also try event handlers:
https://icinga.com/docs/icinga-2/latest/doc/09-object-types/#objecttype-eventcommand

4

I could not understand how notification commands can be called and triggered when a warning happens.
I opted to work with event handlers but can’t seem to make it work either.
I created a file at /etc/icinga2/zones.d/a_zone/commands.conf

object EventCommand "restart_service" {
  command = [ "/usr/lib64/nagios/restart_service" ]
}

At /usr/lib64/nagios/restart_service, I have a test command:

#!/bin/bash
touch /tmp/testing_script

When I call the script with the icinga user, the file is successfully created.
Could it be I have something wrong/missing here?

apply Service "Storage" {
  import "generic-service"
  check_command = "snmp-storage"
  event_command = "restart_service"
  vars.snmp_community = "fdnfdDJsfdh"
  assign where (host.address || host.address6) && host.vars.os == "Linux" && host.zone == "a_zone" && ! host.vars.snmp_storage["Storage"]
5

For notifications:

Let’s take the webhook out of the equation for a moment and say that you want to send email alerts when a warning happens for a given host and/or service:

  • you would define the email command (Icinga includes this by default iirc)
  • you would set up a notification template. In the template, you would specify that you only care about “Warning” state transitions
  • you would create a notification apply rule, import the template, and apply the notification to the service(s)/host(s) that you care about

Apply rule docs:
https://icinga.com/docs/icinga-2/latest/doc/03-monitoring-basics/#apply-rules

I’m not sure about event commands here, for all I know what you have above is right, what do the logs say when the warning is triggered? You can probably tail -f /var/log/icinga2/icinga2.log | grep -i restart_service and find out; or just grep -i restart_service /var/log/icinga2/icinga2.log to see what has happened before.

1 Like
6

@steaksauce with those steps I managed to find my way with the below configuration.

template NotificationCommand "empty_json_file-template-service" {
    import "plugin-notification-command"
    command = [ "/usr/lib64/nagios/plugins/empty_json_file" ]
}

object NotificationCommand "empty_json_file-service" {
  import "plugin-notification-command"
  import "empty_json_file-template-service"
  command = [ "/usr/lib64/nagios/plugins/empty_json_file" ]

}

apply Notification "Empty-JSON-file" to Service {
  command = "empty_json_file-service"
  users = ["opsgenie"]
  states = [ Warning ]
  times.begin = 1m 
  assign where host.name == "SOME_HOSTNAME" &&  service.name == "Storage"
}

I am not sure if a template is required for such a small task but it works, thanks.

1 Like