Best practice Icinga2 and Icingaweb 2 on different nodes with jira integration

Hello Community,

I have a distributed monitoring setup with a master node and about 300 agents and I want to implement the icingaweb2-module-jira.

Icingaweb2 is installed on a different node than the master.

Now I installed and configured the jira module on my icingaweb2 node, can see the jira tickets in icingaweb and can manually create new tickets with icingacli as described in the 10-Notifications.md documentation.

But I want to automatically create jira issues by using a NotificationCommand object on the master node.
And there is the problem: As I know, all NotificationCommands are executed on the master node but the master node doesn’t have icingacli installed nor does itknow that the jira module exist.

What is the best way to configure it?

Give as much information as you can, e.g.

  • I am not using the Icinga Director.
  • Operating System and version: Debian 10
  • Icinga2 version: (r2.13.0-1)
  • Icinga Web 2 version: 2.9.3
    • jira: 1.1.0
    • incubator: 0.7.0

Hello @rafi01010!

Have you tried running icingacli by SSH?

Best,
AK

Hello @Al2Klimov,

yes, I already thought about this solution, but I don’t know if it is the best option for security reasons.
But I think it is the only best solution without installing and configuring Jira module twice.

Is there no out-of-the-box functionality for assigning the execution endpoint for a notification command?

Oh! Notification has a command_endpoint!

Thank you for the research. I will try it these days and keep you up to date if it works! :slight_smile:

I tried to implement it today but it doesn’t work as excepted.

I Added a NotificationCommand, template Notification and applied some services / hosts to this notifiaction.
I also specified the command_endpoint. (this option is validated by icinga, because if the host does not exist it is not working)

After running icinga2 object list I see command_endpoint is set correctly:

Object 'SomeServiceNotifiaction' of type 'Notification':
  % declared in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * __name = "HIDDEN"
  * command = "jira-notification"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 375:3-375:31
  * command_endpoint = "HIDDEN ICINGAWEB2 FRONTEND NODE"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 391:3-391:46
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 305:3-305:46
  * host_name = "HIDDEN"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * interval = 7200
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 303:3-303:15
  * name = "jira_services_prod"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * period = "24x7"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 393:3-393:17
  * service_name = "HIDDEN"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * source_location
    * first_column = 1
    * first_line = 298
    * last_column = 50
    * last_line = 298
    * path = "/etc/icinga2/zones.d/global-templates/notifications.conf"
  * states = [ "OK", "Warning", "Critical", "Unknown" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 377:3-377:45
  * templates = [ "jira_services_prod", "jira-service-notification" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 374:1-374:49
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 378:3-379:40
  * user_groups = null
  * users = [ "jira" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 301:3-301:23
  * vars
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 381:3-389:3
    * jira_ack_author = "$notification.author$"
    * jira_description = "$service.output$"
    * jira_issuetype = "HIDDEN"
    * jira_project = "HIDDEN"
    * jira_state = "$service.state$"
    * jira_summary = "$service.name$ on $host.name$ is $service.state$"
    * jira_template = "HIDDEN"
  * zone = "HIDDEN"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50

Object 'jira-notification' of type 'NotificationCommand':
  % declared in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 265:1-265:46
  * __name = "jira-notification"
  * arguments
    % = modified in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 268:5-311:5
    * --ack-author
      * description = "This author name will be used when acknowledging Icinga problems once a JIRA issue got created"
      * value = "$jira_ack_author$"
    * --command-pipe
      * description = "Legacy Icinga command pipe. Should only be used on Icinga 1.x system without a correctly configured Icinga Web 2 monitoring module"
      * value = "$jira_command_pipe$"
    * --description
      * description = "JIRA issue description"
      * required = true
      * value = "$jira_description$"
    * --host = "$host.name$"
    * --issuetype
      * description = "JIRA issue type (e.g. Incident)"
      * required = true
      * value = "$jira_issuetype$"
    * --no-acknowledge
      * description = "D not acknowledge  Icinga problems once a JIRA issue got created"
      * value = "$jira_no_acknowledge$"
    * --project
      * description = "JIRA project name (e.g. IT)"
      * required = true
      * value = "$jira_project$"
    * --service = "$service.name$"
    * --state
      * description = "Host/Service state (e.g. DOWN, CRITICAL)"
      * value = "$jira_state$"
    * --summary
      * description = "JIRA issue summary"
      * required = true
      * value = "$jira_summary$"
    * --template
      * description = "Issue template name (templates.ini section). This allows to pass custom fields to JIRA"
      * value = "$jira_template$"
  * command = [ "/usr/bin/icingacli", "jira", "send", "problem" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 267:5-267:65
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "jira-notification"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 265
    * last_column = 46
    * last_line = 265
    * path = "/etc/icinga2/zones.d/global-templates/commands.conf"
  * templates = [ "jira-notification", "plugin-notification-command", "plugin-notification-command" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 265:1-265:46
    % = modified in 'methods-itl.conf', lines 22:2-22:122
    % = modified in 'methods-itl.conf', lines 22:2-22:122
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = "global-templates"

But I found this in the icinga log file:

[2021-11-05 12:52:22 +0100] warning/PluginNotificationTask: Notification command for object 'monitor01.agenda.de!Icinga' (PID: 25897, arguments: '/usr/bin/icingacli' 'jira' 'send' 'problem' '--ack-author' '' '--description' 'HIDDEN' '--host' 'HIDDEN' '--issuetype' 'HIDDEN' '--project' 'HIDDEN' '--service' 'Icinga' '--state' 'WARNING' '--summary' 'HIDDEN' '--template' 'HIDDEN') terminated with exit code 128, output: execvpe (/usr/bin/icingacli) failed: No such file or directory

The nagios user has all the required permissions to execute the icingacli command (on the icingaweb2 frontend node). I think the icinga master node performces this notification on it’s own node an not (as intended) on the defined command_endpoint node.

@Al2Klimov I see you already woked on it in the last years:

Maybe you can speed up this PR a little bit :slight_smile:

@jbrost @nhilverling