ACK issues after upgrade Icinga2/Icingaweb2

Icinga 2
#1

Hi all,

have just upgraded to Icinga 2.12.3/icinga web 2.8.2 from 2.11.x/2.6.x and it all went quite well except for ACKs. There were quite a few ACKs that had been in place for quite some time and they have fallen out of the ACK state - the object clearly states that they are ‘not ACK’d’ but when I go to ACK the service or host again I see:

icinga2: Can’t send external icinga command: …service … is already acknowledged.

Important to know: If I make an ACK on a new critical that appears for example, it’s OK.

In this thread: Acknowledgements not syncing between masters

I read of someone having the same issue. So I checked my Modules - monitoring - Backends entries and I have one Command Transport which is a Transport of Icinga2 Api nature. And that is the only entry, I have tested the credentials with an API call and it’s ok. I do not have my 2 masters as entries under ‘Transport’ and that thread says to have them both listed but I am not sure this is exactly the same issue.

Any ideas?

TLDR: old ACK’s existing prior to Icinga upgrade to latest version are no longer ACK’d but Icinga will not re-ack them as they are considered as ACK’d despite now showing as so, with an error message appearing if attempting to ACK.

one edit: notifications are switched off via the global notifcations switch

#2

Bit more info here from the logs - some errors and you can see where we try to ack something that is already deemed as ‘acked’:

2021-03-03T17:59:30+01:00 - DEBUG - Sending Icinga command “actions/acknowledge-problem” to the API “:”
2021-03-03T17:59:30+01:00 - DEBUG - Executing curl -s -X POST -H ‘Accept: application/json’ -k -u
2021-03-03T17:59:30+01:00 - DEBUG - * About to connect() to x.x.x.x port 5665 (#25)

  • Trying …
  • Connected to () port 5665 (#25)
  • skipping SSL peer certificate verification
  • NSS: client certificate not found (nickname not specified)
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • Server certificate:
  •   subject: CN=<host>

  •   start date: Mar 28 09:30:42 2017 GMT

  •   expire date: Mar 24 09:30:42 2032 GMT

  •   common name: <host>

  •   issuer: CN=Icinga CA
  • Server auth using Basic with user ‘icingaweb2’

POST /v1/actions/acknowledge-problem HTTP/1.1
Authorization: Basic
Host: x.x.x.x:5665
Content-Type: application/json
Accept: application/json
Content-Length: 145

  • upload completely sent off: 145 out of 145 bytes
    < HTTP/1.1 500 Internal Server Error
    < Server: Icinga/2.12.3
    < Content-Type: application/json
    < Content-Length: 122
    <
  • Connection #25 to host x.x.x.x left intact

2021-03-03T17:59:30+01:00 - ERROR - Icinga\Module\Monitoring\Exception\CommandTransportException in /usr/share/icingaweb2/modules/monitoring/library/Monitoring/Command/Transport/ApiCommandTransport.php:234 with message: Can’t send external Icinga command: 409 Service !windows_icinga_port is already acknowledged.
2021-03-03T17:59:30+01:00 - ERROR - Icinga\Module\Monitoring\Exception\CommandTransportException in /usr/share/icingaweb2/modules/monitoring/library/Monitoring/Command/Transport/CommandTransport.php:137 with message: icinga2: Can’t send external Icinga command: 409 Service !windows_icinga_port is already acknowledged.
#0 /usr/share/icingaweb2/modules/monitoring/application/forms/Command/Object/AcknowledgeProblemCommandForm.php(163): Icinga\Module\Monitoring\Command\Transport\CommandTransport->send(Object(Icinga\Module\Monitoring\Command\Object\AcknowledgeProblemCommand))
#1 /usr/share/php/Icinga/Web/Form.php(1171): Icinga\Module\Monitoring\Forms\Command\Object\AcknowledgeProblemCommandForm->onSuccess()
#2 /usr/share/icingaweb2/modules/monitoring/library/Monitoring/Web/Controller/MonitoredObjectController.php(157): Icinga\Web\Form->handleRequest()
#3 /usr/share/icingaweb2/modules/monitoring/application/controllers/ServiceController.php(85): Icinga\Module\Monitoring\Web\Controller\MonitoredObjectController->handleCommandForm(Object(Icinga\Module\Monitoring\Forms\Command\Object\AcknowledgeProblemCommandForm))
#4 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Monitoring\Controllers\ServiceController->acknowledgeProblemAction()
#5 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch(String)
#6 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#7 /usr/share/php/Icinga/Application/Web.php(300): Zend_Controller_Front->dispatch(Object(Icinga\Web\Request), Object(Icinga\Web\Response))
#8 /usr/share/php/Icinga/Application/webrouter.php(99): Icinga\Application\Web->dispatch()
#9 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#10 {main}