Can't send external Icinga command: 401 Unauthorized

I have this problem in my web interface, as in the title - both when I click Check now and when I try to open the Source tab, where I see the following stack trace:

Can't send external Icinga command: 401 Unauthorized. Please check your user credentials.

#0 /usr/share/icingaweb2/modules/icingadb/library/Icingadb/Command/Transport/ApiCommandTransport.php(301): Icinga\Module\Icingadb\Command\Transport\ApiCommandTransport->sendCommand()
#1 /usr/share/icingaweb2/modules/icingadb/library/Icingadb/Command/Transport/CommandTransport.php(111): Icinga\Module\Icingadb\Command\Transport\ApiCommandTransport->send()
#2 /usr/share/icingaweb2/modules/icingadb/application/controllers/HostController.php(89): Icinga\Module\Icingadb\Command\Transport\CommandTransport->send()
#3 /usr/share/icingaweb2/modules/icingadb/library/Icingadb/Web/Controller.php(490): Icinga\Module\Icingadb\Controllers\HostController->sourceAction()
#4 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Icinga\Module\Icingadb\Web\Controller->dispatch()
#5 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#6 /usr/share/php/Icinga/Application/Web.php(290): Zend_Controller_Front->dispatch()
#7 /usr/share/php/Icinga/Application/webrouter.php(105): Icinga\Application\Web->dispatch()
#8 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#9 {main}

Unfortunately I don’t know PHP in any detail; I have already searched extensively, and found this among others, but I don’t really understand whether it relates to my problem.

root@vogon:/var/log/apache2# icinga2 -V
icinga2 - The Icinga 2 network monitoring daemon (version: r2.13.7-1)

Copyright (c) 2012-2023 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later <https://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
  Platform: Debian GNU/Linux
  Platform version: 11 (bullseye)
  Kernel: Linux
  Kernel version: 5.10.0-13-amd64
  Architecture: x86_64

Build information:
  Compiler: GNU 10.2.1
  Build host: runner-hh8q3bz2-project-575-concurrent-0
  OpenSSL version: OpenSSL 1.1.1n  15 Mar 2022

Application information:

General paths:
  Config directory: /etc/icinga2
  Data directory: /var/lib/icinga2
  Log directory: /var/log/icinga2
  Cache directory: /var/cache/icinga2
  Spool directory: /var/spool/icinga2
  Run directory: /run/icinga2

Old paths (deprecated):
  Installation root: /usr
  Sysconf directory: /etc
  Run directory (base): /run
  Local state directory: /var

Internal paths:
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Modified attributes path: /var/lib/icinga2/modified-attributes.conf
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid

-----

root@vogon:/var/log/apache2# icinga2 feature list
Disabled features: compatlog debuglog elasticsearch gelf graphite influxdb influxdb2 livestatus opentsdb perfdata statusdata syslog
Enabled features: api checker command icingadb mainlog notification

-----

Modules etc:

Icinga Web 2 Version 	2.11.4
Git commit 	11453bfa92a70a44efbf7f966f5e7f27e9300a28
PHP Version 	7.4.33
Git commit date 	2023-01-26

Loaded Libraries
icinga/icinga-php-library 	0.11.0
icinga/icinga-php-thirdparty 	0.11.0

Loaded Modules
doc 		2.11.4 	Configure
icingadb 		1.0.2 	Configure
Copyright © 2013-2023 Icinga GmbH

-----

Browser: Firefox 102.9.0esr (64bit)

-----

root@vogon:/var/log/apache2# php -v
PHP 7.4.33 (cli) (built: Feb 22 2023 20:07:47) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.33, Copyright (c), by Zend Technologies

you need an icinga2 api user with permissions:
https://icinga.com/docs/icinga-web/latest/modules/monitoring/doc/05-Command-Transports/

this user needs to be added in the icingadb module configuration:

OK - I went there and found the monitoring module disabled, so I enabled it. Now I see this error everywhere - this is from when I click the Create a New Command Transport button, but it broke all of the monitoring:

No backend has been configured

#0 /usr/share/icingaweb2/modules/monitoring/library/Monitoring/Backend/MonitoringBackend.php(76): Icinga\Module\Monitoring\Backend\MonitoringBackend::loadConfig()
#1 /usr/share/icingaweb2/modules/monitoring/application/controllers/ConfigController.php(262): Icinga\Module\Monitoring\Backend\MonitoringBackend::instance()
#2 /usr/share/icingaweb2/library/vendor/Zend/Controller/Action.php(507): Icinga\Module\Monitoring\Controllers\ConfigController->createtransportAction()
#3 /usr/share/php/Icinga/Web/Controller/Dispatcher.php(76): Zend_Controller_Action->dispatch()
#4 /usr/share/icingaweb2/library/vendor/Zend/Controller/Front.php(937): Icinga\Web\Controller\Dispatcher->dispatch()
#5 /usr/share/php/Icinga/Application/Web.php(290): Zend_Controller_Front->dispatch()
#6 /usr/share/php/Icinga/Application/webrouter.php(105): Icinga\Application\Web->dispatch()
#7 /usr/share/icingaweb2/public/index.php(4): require_once(String)
#8 {main}

and the module directory seems rather empty, actually:

# ll /etc/icingaweb2/modules/
total 4
drwxrws--- 2 www-data icingaweb2 4096 Mar 23 14:46 icingadb/

After disabling the monitoring module, things work again, but of course, I can’t run external commands.

Could you share /etc/icingaweb2/modules/monitoring/commandtransports.ini (of course redacted)?

why enable the monitoring module?

look into the module configuration of the icingadb module

Unfortunately, that file doesn’t exist:

root@vogon:~# ll /etc/icingaweb2/modules
total 4
drwxrws--- 2 www-data icingaweb2 4096 Mar 23 14:46 icingadb/

Sorry, I misunderstood - when I look at the icingadb module command transports, I see that a user is already defined there, which I have called root.

check the password and compare with the api-users.conf
check the permissions of the user

after changing the password or permissions in icinga2 you need to restart the icinga2 service

That user isn’t defined in the web interface - should it be?

I copied the username and password from /etc/icinga2/conf.d/api-users.conf, and the passive checks work fine with these:

root@vogon:/var/log# cat /etc/icinga2/conf.d/api-users.conf
/**
 * The ApiUser objects are used for authentication against the API.
 */
object ApiUser "root" {
  password = "abcd1234567890"
  // client_cn = ""

  permissions = [ "*" ]
}

so if you put this into the icingadb module config, does it work?

host can be 127.0.0.1
port should be 5665 except you changed it

Hi - sorry for bumping this topic; my fault for letting it die unresolved, but I’ve been dragged away to other projects.

So, to answer Moreamazingnick - no it still doesn’t work. I copied the info from /etc/icinga2/conf.d/api-users.conf to the icingadb module’s Command Transports:

root@vogon:/var/log/icinga2# cat /etc/icingaweb2/modules/icingadb/commandtransports.ini
[icinga2]
skip_validation = "0"
transport = "api"
host = "localhost"
port = "5665"
username = "root"
password = "abcd1234567890"

I did this through the web interface, and it was accepted there, but I still can’t do things like schedule downtime. Unfortunately there doesn’t seem to be any relevant information in the icinga logs; actually, the message is logged in /var/log/syslog, and it says:

Nov  2 09:48:21 vogon icingaweb2[712880]: Sending Icinga command "actions/schedule-downtime" to the API "localhost:5665"
Nov  2 09:48:21 vogon icingaweb2[712880]: Can't send external Icinga command: 401 Unauthorized. Please check your user credentials.

The odd thing is, these credentials work from all other host reporting via the API.

Hah! That last line in my previous reply inspired me to try something: I changed the host = "localhost" to use the actual hostname; I think, in order to get the passive checks to work, I configured icinga to listen on the real IP.

Now it seems to work, and I can run external commands.

Thanks to everyone for your patience.