Mattermost Notification

PiMA · January 12, 2021, 10:11am

Hello Community

I have an Icinga VM and installed this plugin https://github.com/Reamer/icinga2-mattermost to get notification in Mattermost.

I already used the check command on github to test if its working. So regarding this command it works but I dont get any notificaiton in Mattermost. Seems like its not a plugin problem more an icinga problem.

Because when I use a dummy host check which is down i should get a notfication but when i check on icinga web it shows this :

did is miss a simple part where i have to enable notification in Icinga??
I hope anyone can give me a Hint.

Thank you so much

MarcusCaepio · January 12, 2021, 10:52am

Hey,
It’s always useful to post your config, when you need help. So please show the command config, the notification config, the user config, etc…

PiMA · January 12, 2021, 11:37am

okay here are my config file:
commands.conf :

  object NotificationCommand "mattermost_service" {
  import "plugin-notification-command"

  command = [ PluginContribDir + "/icinga2-mattermost/mattermost.py" ]

  arguments = {
    "--channel" = {
      "description" = "mattermost channel"
      "required" = true
      "value" = "$mattermost_channel$"
    }
    "--hostalias" = {
      "required" = true
      "value" = "$mattermost_hostalias$"
    }
    "--notificationtype" = {
      "required" = true
      "value" = "$mattermost_notificationtype$"
    }
    "--author" = {
      "value" = "$mattermost_author$"
    }
    "--comment" = {
      "value" = "$mattermost_comment$"
    }
    "--oneline" = {
      "set_if" = "$mattermost_oneline$"
    }
    "--servicedesc" = {
      "required" = true
      "value" = "$mattermost_servicedesc$"
    }
    "--serviceoutput" = {
      "required" = true
      "value" = "$mattermost_serviceoutput$"
    }
    "--servicestate" = {
      "required" = true
      "value" = "$mattermost_servicestate$"
    }
    "--url" = {
      "description" = "mattermost incomming webhook url"
      "required" = true
      "value" = "$mattermost_url$"
    }
  }

  vars += {
    "mattermost_channel" = "$user.vars.channel$"
    "mattermost_hostalias" = "$host.display_name$"
    "mattermost_notificationtype" = "$notification.type$"
    "mattermost_author" = "$notification.author$"
    "mattermost_comment" = "$notification.comment$"
    "mattermost_oneline" = "$user.vars.oneline$"
    "mattermost_servicedesc" = "$service.display_name$"
    "mattermost_serviceoutput" = "$service.output$"
    "mattermost_servicestate" = "$service.state$"
    "mattermost_url" = "$user.vars.url$"
  }
}

object NotificationCommand "mattermost_host" {
  import "plugin-notification-command"

  command = [ PluginContribDir + "/icinga2-mattermost/mattermost.py" ]

  arguments = {
    "--channel" = {
      "description" = "mattermost channel"
      "required" = true
      "value" = "$mattermost_channel$"
    }
    "--hostalias" = {
      "required" = true
      "value" = "$mattermost_hostalias$"
    }
    "--hostoutput" = {
      "required" = true
      "value" = "$mattermost_hostoutput$"
    }
    "--hoststate" = {
      "required" = true
      "value" = "$mattermost_hoststate$"
    }
    "--notificationtype" = {
      "required" = true
      "value" = "$mattermost_notificationtype$"
    }
    "--author" = {
      "value" = "$mattermost_author$"
    }
    "--comment" = {
      "value" = "$mattermost_comment$"
    }
    "--oneline" = {
      "set_if" = "$mattermost_oneline$"
    }
    "--url" = {
      "description" = "mattermost incomming webhook url"
      "required" = true
      "value" = "$mattermost_url$"
    }
  }

  vars += {
    "mattermost_channel" = "$user.vars.channel$"
    "mattermost_hostalias" = "$host.display_name$"
    "mattermost_hostoutput" = "$host.output$"
    "mattermost_hoststate" = "$host.state$"
    "mattermost_notificationtype" = "$notification.type$"
    "mattermost_author" = "$notification.author$"
    "mattermost_comment" = "$notification.comment$"
    "mattermost_oneline" = "$user.vars.oneline$"
    "mattermost_url" = "$user.vars.url$"
  }
}

user.conf:

object User "mattermost" {
  import "generic-user"
  display_name = "Mattermost User"
  enable_notifications = true

  vars.channel = "Icinga2" /* Use here the url channel name, ie. without special characters and spaces */
  vars.url = "https://xxxxxxxxxxx.com/hooks/xxxxxxxxxxxxxxx"  /* Copy incoming Webhook URL defined in mattermost */
  vars.oneline = false /* Use true if you prefer having all output squashed on one line */
}

notifications.conf:

apply Notification "mattermost_service" to Service {
  assign where true
  command = "mattermost_service"
  users = [ "mattermost" ]
  period = "24x7"
  types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  states = [ OK, Warning, Critical, Unknown ]
}
apply Notification "mattermost_host" to Host {
  assign where true
  command = "mattermost_host"
  users = [ "mattermost" ]
  period = "24x7"
  types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  states = [ Up, Down ]
}

PiMA · January 13, 2021, 11:32am

I just checked when I try to send the command via web interface manually it still does not send a notification

when i check the log while trying to do so it shows this:

[root@ixxxx icinga2]# tail -n5 icinga2.log
[2021-01-13 12:25:17 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 8.21667/s (493/min 2451/5min 2736/15min);
[2021-01-13 12:25:17 +0100] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 0, rate: 7.16667/s (430/min 2146/5min 2243/15min);
[2021-01-13 12:27:32 +0100] information/ApiListener: New client connection from [xxx.xxx.xxxx.xxx]:xxxxx (no client certificate)
[2021-01-13 12:27:32 +0100] information/HttpServerConnection: Request: POST /v1/actions/send-custom-notification (from [xxx.xxx.xxx.xxx]:xxxxx), user: icingaweb2, agent: ).
[2021-01-13 12:27:32 +0100] information/HttpServerConnection: HTTP client disconnected (from [xxx.xxx.xxx.xxx]:xxxxx)

homerjay · January 13, 2021, 4:00pm

Hi.

Do you get a notification in Mattermost, when you send a test-message via commandline?

E.g.:

/usr/lib/nagios/plugins/icinga2-mattermost/mattermost.py \
--url "https://yourwebhookurl" \
--channel "Icinga2" \
--hostalias "test" \
--oneline \
--notificationtype "some-alarm"

Greetings.

PiMA · January 14, 2021, 7:36am

I checked this command

/usr/lib64/nagios/plugins/icinga2-mattermost/mattermost.py --url https://example.com/hooks/xxxxxxxxxx --channel icinga2 --notificationtype "some-alarm" --hostalias test

And I get a notification

PiMA · January 14, 2021, 7:57am

I think there is a problem with sending notification here, it seems like the icingaweb can’t send notification. If I try it here manually it does not send any notificition and still write “for this problem was not notification send”. I think somewhere are the notification disabled but I can’t find where?

homerjay · January 14, 2021, 9:34am

Hi again.

When you manually sent the notification via icingaweb2, did you enable “force”?
Icingaweb2 -> System -> Monitoring Health -> Feature Commands:
Are notifications enabled?
Does:

icinga2 object list --type notification

Show your dummy-host1 with the host alert?

Greetings.

PiMA · January 14, 2021, 2:10pm

yes I did and nothing happened
enabled
the list is way too long so I did it with " | grep dummy " and I get this

PiMA · January 14, 2021, 2:13pm

[root@xxxx~]# icinga2 object list --type notification | grep -A 30 dummy
Object 'dummy-host2!mattermost_host' of type 'Notification':
  % declared in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * __name = "dummy-host2!mattermost_host"
  * command = "mattermost_host"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 52:3-52:29
  * command_endpoint = ""
  * host_name = "dummy-host2"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * interval = 1800
  * name = "mattermost_host"
  * package = "_etc"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * period = "24x7"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 54:3-54:17
  * service_name = ""
  * source_location
    * first_column = 1
    * first_line = 50
    * last_column = 44
    * last_line = 50
    * path = "/etc/icinga2/conf.d/notifications.conf"
  * states = [ "Up", "Down" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 56:3-56:23
  * templates = [ "mattermost_host" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd", "DowntimeStart", "DowntimeEnd", "DowntimeRemoved" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 55:3-55:129
  * user_groups = null
  * users = [ "mattermost" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 53:3-53:26
  * vars = null
  * zone = ""

MarcusCaepio · January 26, 2021, 9:35am

Did you execute the CLI command as root or icinga/nagios user (depending on your OS) ?

PiMA · January 27, 2021, 11:08am

I am running the command
./mattermost.py --url https://mattermost.xxxxxxxxxx.com/hooks/xxxxxxxxxxxx --notificationtype CUSTOM --hostalias myexample.com
as root
in the folder
/usr/lib64/nagios/plugins/icinga2-mattermost

and my OS is linux

petew · January 28, 2021, 5:42pm

does the command work if it’s the icinga/nagios user? i’ve had problems with check scripts not being executable by nagios user in the past.

PiMA · January 29, 2021, 7:15am

I can’t change the user with su icinga ? I tried but I get an error. This seems not the problem because on my second icinga istance it works fine with the root user. I can’t even send a notification manually when I am on the icinga webinterface. Or how do I run a command as a different user?

petew · February 1, 2021, 1:56pm

without knowing what the error is, here’s some things to look at:

check the permissions on the icinga2-mattermost script
is icinga2 running as icinga user or nagios user (even if running as root you may need to do sudo su - icinga
check the path in the icinga constants.conf
check notifications.conf for path and command

PiMA · February 2, 2021, 1:07pm

I checked

the script has ( -rwxr-xr-x / 0755 ) permission
and the folder has the same ( drwxr-xr-x )
like all other plugins.
```
 su - icinga
```

Last login: Wed Jan 27 12:11:55 CET 2021 on pts/0
This account is currently not available.

but this shouldnt be the problem because i have the same issue on my second icinga and there is the mattermost plugin working

the path in constants.conf is /usr/lib64/nagios/plugins which is the right one, alls plugins are stored here
regarding the notifcation.conf I dont know where I should put a path in here? and the command is also right

homerjay · February 2, 2021, 2:20pm

Hi again.

Since the “icinga” user does not have a login shell, you can not directly login.
As a workaround, you can use sudo here:

sudo -u icinga /path/to/the/script   # Directly executing the script
# alternatively to get a shell: sudo -u icinga bash 
# or whatever shell instead of bash (/bin/sh, ...)

Assuming that you are using RedHat/CentOS (because the user is “icinga”):
Did you check for SELinux related problems?

Greetings

PiMA · February 3, 2021, 10:18am

I just checked I can run the command as icinga user manually and it works, I get a notification in my mattermost channel. I will check for SELinux problems and let you know what I find.

PiMA · February 5, 2021, 12:22pm

Can’t find anything in SELinux. What could be related to this problem.

homerjay · February 8, 2021, 8:50am

Hi.

Maybe this helps:
link

SELinux was often mentioned in this forum, so maybe a search gives you further hints.

Greetings.