Best practice Icinga2 and Icingaweb 2 on different nodes with jira integration

rafi01010 · October 29, 2021, 11:50am

Hello Community,

I have a distributed monitoring setup with a master node and about 300 agents and I want to implement the icingaweb2-module-jira.

Icingaweb2 is installed on a different node than the master.

Now I installed and configured the jira module on my icingaweb2 node, can see the jira tickets in icingaweb and can manually create new tickets with icingacli as described in the 10-Notifications.md documentation.

But I want to automatically create jira issues by using a NotificationCommand object on the master node.
And there is the problem: As I know, all NotificationCommands are executed on the master node but the master node doesn’t have icingacli installed nor does itknow that the jira module exist.

What is the best way to configure it?

Give as much information as you can, e.g.

I am not using the Icinga Director.
Operating System and version: Debian 10
Icinga2 version: (r2.13.0-1)
Icinga Web 2 version: 2.9.3
- jira: 1.1.0
- incubator: 0.7.0

Al2Klimov · November 2, 2021, 11:02am

Hello @rafi01010!

Have you tried running icingacli by SSH?

Best,
AK

rafi01010 · November 2, 2021, 12:07pm

Hello @Al2Klimov,

yes, I already thought about this solution, but I don’t know if it is the best option for security reasons.
But I think it is the only best solution without installing and configuring Jira module twice.

Is there no out-of-the-box functionality for assigning the execution endpoint for a notification command?

Al2Klimov · November 2, 2021, 12:48pm

Oh! Notification has a command_endpoint!

github.com

Icinga/icinga2/blob/master/lib/icinga/notification.ti#L93-L97

    
      
          	[config, navigation] name(Endpoint) command_endpoint (CommandEndpointRaw) {
          		navigate {{{
          			return Endpoint::GetByName(GetCommandEndpointRaw());
          		}}}
          	};

rafi01010 · November 2, 2021, 1:25pm

Thank you for the research. I will try it these days and keep you up to date if it works!

rafi01010 · November 5, 2021, 12:36pm

I tried to implement it today but it doesn’t work as excepted.

I Added a NotificationCommand, template Notification and applied some services / hosts to this notifiaction.
I also specified the command_endpoint. (this option is validated by icinga, because if the host does not exist it is not working)

After running icinga2 object list I see command_endpoint is set correctly:

Object 'SomeServiceNotifiaction' of type 'Notification':
  % declared in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * __name = "HIDDEN"
  * command = "jira-notification"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 375:3-375:31
  * command_endpoint = "HIDDEN ICINGAWEB2 FRONTEND NODE"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 391:3-391:46
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 305:3-305:46
  * host_name = "HIDDEN"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * interval = 7200
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 303:3-303:15
  * name = "jira_services_prod"
  * package = "_etc"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * period = "24x7"
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 393:3-393:17
  * service_name = "HIDDEN"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
  * source_location
    * first_column = 1
    * first_line = 298
    * last_column = 50
    * last_line = 298
    * path = "/etc/icinga2/zones.d/global-templates/notifications.conf"
  * states = [ "OK", "Warning", "Critical", "Unknown" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 377:3-377:45
  * templates = [ "jira_services_prod", "jira-service-notification" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 374:1-374:49
  * times = null
  * type = "Notification"
  * types = [ "Problem", "Acknowledgement", "Recovery", "Custom", "FlappingStart", "FlappingEnd" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 378:3-379:40
  * user_groups = null
  * users = [ "jira" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 301:3-301:23
  * vars
    % = modified in '/etc/icinga2/zones.d/global-templates/templates.conf', lines 381:3-389:3
    * jira_ack_author = "$notification.author$"
    * jira_description = "$service.output$"
    * jira_issuetype = "HIDDEN"
    * jira_project = "HIDDEN"
    * jira_state = "$service.state$"
    * jira_summary = "$service.name$ on $host.name$ is $service.state$"
    * jira_template = "HIDDEN"
  * zone = "HIDDEN"
    % = modified in '/etc/icinga2/zones.d/global-templates/notifications.conf', lines 298:1-298:50

Object 'jira-notification' of type 'NotificationCommand':
  % declared in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 265:1-265:46
  * __name = "jira-notification"
  * arguments
    % = modified in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 268:5-311:5
    * --ack-author
      * description = "This author name will be used when acknowledging Icinga problems once a JIRA issue got created"
      * value = "$jira_ack_author$"
    * --command-pipe
      * description = "Legacy Icinga command pipe. Should only be used on Icinga 1.x system without a correctly configured Icinga Web 2 monitoring module"
      * value = "$jira_command_pipe$"
    * --description
      * description = "JIRA issue description"
      * required = true
      * value = "$jira_description$"
    * --host = "$host.name$"
    * --issuetype
      * description = "JIRA issue type (e.g. Incident)"
      * required = true
      * value = "$jira_issuetype$"
    * --no-acknowledge
      * description = "D not acknowledge  Icinga problems once a JIRA issue got created"
      * value = "$jira_no_acknowledge$"
    * --project
      * description = "JIRA project name (e.g. IT)"
      * required = true
      * value = "$jira_project$"
    * --service = "$service.name$"
    * --state
      * description = "Host/Service state (e.g. DOWN, CRITICAL)"
      * value = "$jira_state$"
    * --summary
      * description = "JIRA issue summary"
      * required = true
      * value = "$jira_summary$"
    * --template
      * description = "Issue template name (templates.ini section). This allows to pass custom fields to JIRA"
      * value = "$jira_template$"
  * command = [ "/usr/bin/icingacli", "jira", "send", "problem" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 267:5-267:65
  * env = null
  * execute
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    % = modified in 'methods-itl.conf', lines 23:3-23:30
    * arguments = [ "notification", "user", "cr", "itype", "author", "comment", "resolvedMacros", "useResolvedMacros" ]
    * deprecated = false
    * name = "Internal#PluginNotification"
    * side_effect_free = false
    * type = "Function"
  * name = "jira-notification"
  * package = "_etc"
  * source_location
    * first_column = 1
    * first_line = 265
    * last_column = 46
    * last_line = 265
    * path = "/etc/icinga2/zones.d/global-templates/commands.conf"
  * templates = [ "jira-notification", "plugin-notification-command", "plugin-notification-command" ]
    % = modified in '/etc/icinga2/zones.d/global-templates/commands.conf', lines 265:1-265:46
    % = modified in 'methods-itl.conf', lines 22:2-22:122
    % = modified in 'methods-itl.conf', lines 22:2-22:122
  * timeout = 60
  * type = "NotificationCommand"
  * vars = null
  * zone = "global-templates"

But I found this in the icinga log file:

[2021-11-05 12:52:22 +0100] warning/PluginNotificationTask: Notification command for object 'monitor01.agenda.de!Icinga' (PID: 25897, arguments: '/usr/bin/icingacli' 'jira' 'send' 'problem' '--ack-author' '' '--description' 'HIDDEN' '--host' 'HIDDEN' '--issuetype' 'HIDDEN' '--project' 'HIDDEN' '--service' 'Icinga' '--state' 'WARNING' '--summary' 'HIDDEN' '--template' 'HIDDEN') terminated with exit code 128, output: execvpe (/usr/bin/icingacli) failed: No such file or directory

The nagios user has all the required permissions to execute the icingacli command (on the icingaweb2 frontend node). I think the icinga master node performces this notification on it’s own node an not (as intended) on the defined command_endpoint node.

rafi01010 · November 5, 2021, 12:51pm

@Al2Klimov I see you already woked on it in the last years:

Maybe you can speed up this PR a little bit

github.com/Icinga/icinga2

Add possibility to trigger an execution on an agent

opened 09:37AM - 28 Jul 20 UTC

R-Sommer

enhancement area/notifications

## Is your feature request related to a problem? Please describe. I'd like to… have an option where I can configure a notification object to execute a command on agent instead of the scheduling node. ## Describe the solution you'd like Add attribute commend_endpoint to [Notification](https://icinga.com/docs/icinga2/latest/doc/09-object-types/#notification). ## Describe alternatives you've considered Define a passive check having a custom command which will be executed on an agent and manually trigger its execution via #8034

Al2Klimov · November 5, 2021, 1:24pm

@jbrost @nhilverling

rafi01010 · December 10, 2021, 3:15pm

Hi, is there any news on this topic?

rafi01010 · February 22, 2022, 1:30pm

Hi @Al2Klimov @jbrost @nhilverling,

please can you give me any Informamtion if this will happen in one of the next icinga2 versions?
I would be very pleased