Mattermost Notification

Hello Community

I have an Icinga VM and installed this plugin https://github.com/Reamer/icinga2-mattermost to get notification in Mattermost.

I already used the check command on github to test if its working. So regarding this command it works but I dont get any notificaiton in Mattermost. Seems like its not a plugin problem more an icinga problem.

Because when I use a dummy host check which is down i should get a notfication but when i check on icinga web it shows this :

image

did is miss a simple part where i have to enable notification in Icinga??
I hope anyone can give me a Hint.

Thank you so much

Hey,
It’s always useful to post your config, when you need help. So please show the command config, the notification config, the user config, etc…

okay here are my config file:
commands.conf :

  object NotificationCommand "mattermost_service" {
  import "plugin-notification-command"

  command = [ PluginContribDir + "/icinga2-mattermost/mattermost.py" ]

  arguments = {
    "--channel" = {
      "description" = "mattermost channel"
      "required" = true
      "value" = "$mattermost_channel$"
    }
    "--hostalias" = {
      "required" = true
      "value" = "$mattermost_hostalias$"
    }
    "--notificationtype" = {
      "required" = true
      "value" = "$mattermost_notificationtype$"
    }
    "--author" = {
      "value" = "$mattermost_author$"
    }
    "--comment" = {
      "value" = "$mattermost_comment$"
    }
    "--oneline" = {
      "set_if" = "$mattermost_oneline$"
    }
    "--servicedesc" = {
      "required" = true
      "value" = "$mattermost_servicedesc$"
    }
    "--serviceoutput" = {
      "required" = true
      "value" = "$mattermost_serviceoutput$"
    }
    "--servicestate" = {
      "required" = true
      "value" = "$mattermost_servicestate$"
    }
    "--url" = {
      "description" = "mattermost incomming webhook url"
      "required" = true
      "value" = "$mattermost_url$"
    }
  }

  vars += {
    "mattermost_channel" = "$user.vars.channel$"
    "mattermost_hostalias" = "$host.display_name$"
    "mattermost_notificationtype" = "$notification.type$"
    "mattermost_author" = "$notification.author$"
    "mattermost_comment" = "$notification.comment$"
    "mattermost_oneline" = "$user.vars.oneline$"
    "mattermost_servicedesc" = "$service.display_name$"
    "mattermost_serviceoutput" = "$service.output$"
    "mattermost_servicestate" = "$service.state$"
    "mattermost_url" = "$user.vars.url$"
  }
}

object NotificationCommand "mattermost_host" {
  import "plugin-notification-command"

  command = [ PluginContribDir + "/icinga2-mattermost/mattermost.py" ]

  arguments = {
    "--channel" = {
      "description" = "mattermost channel"
      "required" = true
      "value" = "$mattermost_channel$"
    }
    "--hostalias" = {
      "required" = true
      "value" = "$mattermost_hostalias$"
    }
    "--hostoutput" = {
      "required" = true
      "value" = "$mattermost_hostoutput$"
    }
    "--hoststate" = {
      "required" = true
      "value" = "$mattermost_hoststate$"
    }
    "--notificationtype" = {
      "required" = true
      "value" = "$mattermost_notificationtype$"
    }
    "--author" = {
      "value" = "$mattermost_author$"
    }
    "--comment" = {
      "value" = "$mattermost_comment$"
    }
    "--oneline" = {
      "set_if" = "$mattermost_oneline$"
    }
    "--url" = {
      "description" = "mattermost incomming webhook url"
      "required" = true
      "value" = "$mattermost_url$"
    }
  }

  vars += {
    "mattermost_channel" = "$user.vars.channel$"
    "mattermost_hostalias" = "$host.display_name$"
    "mattermost_hostoutput" = "$host.output$"
    "mattermost_hoststate" = "$host.state$"
    "mattermost_notificationtype" = "$notification.type$"
    "mattermost_author" = "$notification.author$"
    "mattermost_comment" = "$notification.comment$"
    "mattermost_oneline" = "$user.vars.oneline$"
    "mattermost_url" = "$user.vars.url$"
  }
}

user.conf:

object User "mattermost" {
  import "generic-user"
  display_name = "Mattermost User"
  enable_notifications = true

  vars.channel = "Icinga2" /* Use here the url channel name, ie. without special characters and spaces */
  vars.url = "https://xxxxxxxxxxx.com/hooks/xxxxxxxxxxxxxxx"  /* Copy incoming Webhook URL defined in mattermost */
  vars.oneline = false /* Use true if you prefer having all output squashed on one line */
}

notifications.conf:

apply Notification "mattermost_service" to Service {
  assign where true
  command = "mattermost_service"
  users = [ "mattermost" ]
  period = "24x7"
  types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  states = [ OK, Warning, Critical, Unknown ]
}
apply Notification "mattermost_host" to Host {
  assign where true
  command = "mattermost_host"
  users = [ "mattermost" ]
  period = "24x7"
  types = [ Problem, Acknowledgement, Recovery, Custom, FlappingStart, FlappingEnd, DowntimeStart, DowntimeEnd, DowntimeRemoved ]
  states = [ Up, Down ]
}

I just checked when I try to send the command via web interface manually it still does not send a notification

image

when i check the log while trying to do so it shows this:

[root@ixxxx icinga2]# tail -n5 icinga2.log
[2021-01-13 12:25:17 +0100] information/WorkQueue: #5 (ApiListener, RelayQueue) items: 0, rate: 8.21667/s (493/min 2451/5min 2736/15min);
[2021-01-13 12:25:17 +0100] information/WorkQueue: #7 (IdoPgsqlConnection, ido-pgsql) items: 0, rate: 7.16667/s (430/min 2146/5min 2243/15min);
[2021-01-13 12:27:32 +0100] information/ApiListener: New client connection from [xxx.xxx.xxxx.xxx]:xxxxx (no client certificate)
[2021-01-13 12:27:32 +0100] information/HttpServerConnection: Request: POST /v1/actions/send-custom-notification (from [xxx.xxx.xxx.xxx]:xxxxx), user: icingaweb2, agent: ).
[2021-01-13 12:27:32 +0100] information/HttpServerConnection: HTTP client disconnected (from [xxx.xxx.xxx.xxx]:xxxxx)

Hi.

Do you get a notification in Mattermost, when you send a test-message via commandline?

E.g.:

/usr/lib/nagios/plugins/icinga2-mattermost/mattermost.py \
--url "https://yourwebhookurl" \
--channel "Icinga2" \
--hostalias "test" \
--oneline \
--notificationtype "some-alarm"

Greetings.

I checked this command

/usr/lib64/nagios/plugins/icinga2-mattermost/mattermost.py --url https://example.com/hooks/xxxxxxxxxx --channel icinga2 --notificationtype "some-alarm" --hostalias test

And I get a notification

image

I think there is a problem with sending notification here, it seems like the icingaweb can’t send notification. If I try it here manually it does not send any notificition and still write “for this problem was not notification send”. I think somewhere are the notification disabled but I can’t find where?

Hi again.

  1. When you manually sent the notification via icingaweb2, did you enable “force”?
  2. Icingaweb2 -> System -> Monitoring Health -> Feature Commands:
    Are notifications enabled?
  3. Does:
icinga2 object list --type notification

Show your dummy-host1 with the host alert?


Greetings.

  1. yes I did and nothing happened
  2. enabled
  3. the list is way too long so I did it with " | grep dummy " and I get this

image

[root@xxxx~]# icinga2 object list --type notification | grep -A 30 dummy
Object 'dummy-host2!mattermost_host' of type 'Notification':
  % declared in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * __name = "dummy-host2!mattermost_host"
  * command = "mattermost_host"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 52:3-52:29
  * command_endpoint = ""
  * host_name = "dummy-host2"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * interval = 1800
  * name = "mattermost_host"
  * package = "_etc"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * period = "24x7"
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 54:3-54:17
  * service_name = ""
  * source_location
    * first_column = 1
    * first_line = 50
    * last_column = 44
    * last_line = 50
    * path = "/etc/icinga2/conf.d/notifications.conf"
  * states = [ "Up", "Down" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 56:3-56:23
  * templates = [ "mattermost_host" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 50:1-50:44
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd", "DowntimeStart", "DowntimeEnd", "DowntimeRemoved" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 55:3-55:129
  * user_groups = null
  * users = [ "mattermost" ]
    % = modified in '/etc/icinga2/conf.d/notifications.conf', lines 53:3-53:26
  * vars = null
  * zone = ""

Did you execute the CLI command as root or icinga/nagios user (depending on your OS) ?

I am running the command
./mattermost.py --url https://mattermost.xxxxxxxxxx.com/hooks/xxxxxxxxxxxx --notificationtype CUSTOM --hostalias myexample.com
as root
in the folder
/usr/lib64/nagios/plugins/icinga2-mattermost

and my OS is linux

does the command work if it’s the icinga/nagios user? i’ve had problems with check scripts not being executable by nagios user in the past.

I can’t change the user with su icinga ? I tried but I get an error. This seems not the problem because on my second icinga istance it works fine with the root user. I can’t even send a notification manually when I am on the icinga webinterface. Or how do I run a command as a different user?

without knowing what the error is, here’s some things to look at:

  • check the permissions on the icinga2-mattermost script
  • is icinga2 running as icinga user or nagios user (even if running as root you may need to do sudo su - icinga
  • check the path in the icinga constants.conf
  • check notifications.conf for path and command

I checked

  • the script has ( -rwxr-xr-x / 0755 ) permission
    and the folder has the same ( drwxr-xr-x )
    like all other plugins.

  •  su - icinga
    

Last login: Wed Jan 27 12:11:55 CET 2021 on pts/0
This account is currently not available.

but this shouldnt be the problem because i have the same issue on my second icinga and there is the mattermost plugin working

  • the path in constants.conf is /usr/lib64/nagios/plugins which is the right one, alls plugins are stored here

  • regarding the notifcation.conf I dont know where I should put a path in here? and the command is also right

Hi again.

Since the “icinga” user does not have a login shell, you can not directly login.
As a workaround, you can use sudo here:

sudo -u icinga /path/to/the/script   # Directly executing the script
# alternatively to get a shell: sudo -u icinga bash 
# or whatever shell instead of bash (/bin/sh, ...)

Assuming that you are using RedHat/CentOS (because the user is “icinga”):
Did you check for SELinux related problems?


Greetings

I just checked I can run the command as icinga user manually and it works, I get a notification in my mattermost channel. I will check for SELinux problems and let you know what I find.

Can’t find anything in SELinux. What could be related to this problem.

Hi.

Maybe this helps:
link

SELinux was often mentioned in this forum, so maybe a search gives you further hints.


Greetings.