Only getting 1 (Jira) Notification per service

Hi guys. Great forum :slightly_smiling_face:. I’ve learned a lot since my last visit.

I am currently testing notifications using the Jira module.

I have an issue where I am only getting one notification triggered per service, and not one per host per service. So, in the screenshots below only lhr1-prod-svc-icinga2 is getting alerts triggered:

icinga1

icinga2

Here’s the notification command:

object NotificationCommand "jira-service-notification" {
    import "plugin-notification-command"
    command = [ "/usr/bin/icingacli", "jira", "send", "problem", "--no-acknowledge" ]
    arguments += {
        "--ack-author" = {
            description = "This author name will be used when acknowledging Icinga problems once a JIRA issue got created"
            value = "$jira_ack_author$"
        }
        "--command-pipe" = {
            description = "Legacy Icinga command pipe. Should only be used on Icinga 1.x system without a correctly configured Icinga Web 2 monitoring module"
            value = "$jira_command_pipe$"
        }
        "--description" = {
            description = "JIRA issue description"
            required = true
            value = "$jira_description$"
        }
        "--host" = "$host.name$"
        "--issuetype" = {
            description = "JIRA issue type (e.g. Incident)"
            required = true
            value = "$jira_issuetype$"
        }
        "--no-acknowledge" = {
            description = "Do not acknowledge Icinga problems once a JIRA issue got created"
            value = "$jira_no_acknowledge$"
        }
        "--project" = {
            description = "JIRA project name (e.g. ITSM)"
            required = true
            value = "$jira_project$"
        }
        "--service" = "$service.name$"
        "--state" = {
            description = "Service state (e.g. CRITICAL)"
            value = "$service.state$"
        }
        "--summary" = {
            description = "JIRA issue summary"
            required = true
            value = "$jira_summary$"
        }
        "--template" = {
            description = "Issue template name (templates.ini section). This allows to pass custom fields to JIRA"
            value = "$jira_template$"
        }
    }
    vars.jira_description = "$service.output$"
    vars.jira_summary = "$service.name$ on $host.name$ is $service.state$"
}

And the apply Notification:

apply Notification "jira-service-notification" to Service {
  command = "jira-service-notification"

  //state = [ OK, Warning, Critical, Unknown ]
  types = [ Problem, Acknowledgement, Recovery, Custom,
            FlappingStart, FlappingEnd,
            DowntimeStart, DowntimeEnd, DowntimeRemoved ]

  user_groups = [ "icingaadmins" ]

  //assign where host.vars.notification.mail && host.vars.notification.mail.groups != []
  assign where host.vars.os

  vars += {
    jira_project = "ITOPS"
    jira_issuetype = "Incident"
    notification_logtosyslog = true
  }

  period = "24x7"
}

The ‘unique’ custom fields are correctly set in Jira:

icinga3

The logs show that there is a ‘Checkable’ for that host but doesn’t appear to do anything:

[2020-09-30 11:20:02 +0000] information/Checkable: Checkable 'lhr1-prod-svc-icinga1!firewall' has 1 notification(s). Checking filters for type 'Problem', sends will be logged.

The other host triggers fine:

[2020-09-30 11:11:37 +0000] information/Notification: Completed sending 'Problem' notification 'lhr1-prod-svc-icinga2!firewall!jira-service-notification' for checkable 'lhr1-prod-svc-icinga2!firewall' and user 'icingaadmin' using command 'jira-service-notification'.

Could anyone offer any assistance?

OK. I’ve done a little more investigation. The two servers above are my HA masters. I only had the notifications module enabled on my Primary (configuration) master.

I enabled the notifications on the second master and set enable_ha = true in notification.conf.

I can see notifications attempting to be triggered now, but there is a problem in that the command definition is for icingacli:

[2020-09-30 13:23:16 +0000] warning/PluginNotificationTask: Notification command for object 'lhr1-prod-svc-icinga1!firewall' (PID: 570904, arguments: '/usr/bin/icingacli' 'jira' 'send' 'problem' '--no-acknowledge' '--description' 'CRITICAL: ufw is inactive' '--host' 'lhr1-prod-svc-icinga1' '--issuetype' 'Incident' '--project' 'ITOPS' '--service' 'firewall' '--state' 'CRITICAL' '--summary' 'firewall on lhr1-prod-svc-icinga1 is CRITICAL') terminated with exit code 128, output: execvpe(/usr/bin/icingacli) failed: No such file or directory

icingacli is only installed as part of icinga web 2, which only resides on my primary master.

Is there a way to force all notifications to run on my primary master?

As a way forward, I might write a shell wrapper script that calls icingacli via SSH, create a new definition, and force the calls to the primary master, if invoked from the secondary. Definitely not HA, but it might have to do for now - unless anyone’s got a better suggestion?