Icinga for Windows and centreon_plugins.exe - UNKNOWN: Internal error: execution issue: Access is denied

Aleksey-Maksimov · August 1, 2024, 7:59am

Hello.

We are trying to use the executable file of the plugin centreon_plugins.exe (Release Centreon Plugins Release 20240711 · centreon/centreon-nsclient-build · GitHub) on Windows Server 2022 together with the Icinga 2 for Windows agent.
And we have a problem with this.

If we manually run cmd.exe as administrator and execute the command to call centreon_plugins.exe there, it works successfully:


C:\ProgramData\icinga2\plugins\centreon_plugins\centreon_plugins.exe --plugin os::windows::local::plugin --mode pending-reboot
OK: 'Microsoft Windows Server 2022 Standard': reboot pending is false [Windows Update: false][Component Based Servicing: false][SCCM Client: false][File Rename Operations: false][Computer Name Change: false]

The Icinga 2 for Windows agent service runs on behalf of Local System. Therefore, we try to run cmd.exe on behalf of Local System and execute the same command there. And it works successfully again.

But if we try to call the plugin via the Icinga 2 for Windows agent, we get the message “UNKNOWN: Internal error: execution issue: Access is denied”

Please tell us how to debug this situation. We do not understand where the problem with the lack of rights occurs, because the Icinga 2 for Windows agent service runs with full rights, which means the plugin is called with full rights.

Environment configuration:

Icinga for Windows output (Show-Icinga)
Operating System and version

Icinga for Windows environment:

PowerShell Root => C:\Program Files\WindowsPowerShell\Modules
Icinga for Windows Service Path => C:\Program Files\icinga-framework-service
Icinga for Windows Service User => LocalSystem
Icinga for Windows Service Pid => 19740
Icinga for Windows JEA Pid =>
Icinga Agent Path => C:\Program Files\ICINGA2
Icinga Agent User => NT Authority\SYSTEM
Defined Default User =>
Icinga Managed User => False
PowerShell Version => 5.1.20348.2400
Operating System => Microsoft Windows Server 2022 Standard
Operating System Version => 10.0.20348
JEA Context =>
JEA Session File =>
Api Check Forwarder => True
Debug Mode => False

Icinga for Windows Certificate:

Issuer => CN=Icinga CA
Subject => CN=backup10.holding.com

List of configured background daemons on this system:

Start-IcingaWindowsRESTApi
No arguments defined

List of configured background service checks on this system:
No background service checks configured

List of configured repositories on this system. The list order matches the apply order:
No repositories configured
Installed components on this system:

Component Version Available

agent 2.14.2
framework 1.12.3
plugins 1.12.0
service 1.2.0

Available versions flagged with “*” mean that this component is locked to this version

Version used (icinga2 --version)

icinga2 - The Icinga 2 network monitoring daemon (version: r2.14.2-1)

Copyright (c) 2012-2024 Icinga GmbH (https://icinga.com/)
License GPLv2+: GNU GPL version 2 or later https://gnu.org/licenses/gpl2.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

System information:
Platform: Debian GNU/Linux
Platform version: 12 (bookworm)
Kernel: Linux
Kernel version: 6.1.0-23-amd64
Architecture: x86_64

Build information:
Compiler: GNU 12.2.0
Build host: runner-hh8q3bz2-project-575-concurrent-0
OpenSSL version: OpenSSL 3.0.13 30 Jan 2024

Application information:

General paths:
Config directory: /etc/icinga2
Data directory: /var/lib/icinga2
Log directory: /var/log/icinga2
Cache directory: /var/cache/icinga2
Spool directory: /var/spool/icinga2
Run directory: /run/icinga2

Old paths (deprecated):
Installation root: /usr
Sysconf directory: /etc
Run directory (base): /run
Local state directory: /var

Internal paths:
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

Enabled features (icinga2 feature list)

Disabled features: command compatlog debuglog elasticsearch gelf graphite influxdb journald livestatus opentsdb perfdata syslog
Enabled features: api checker icingadb influxdb2 mainlog notification

Icinga Web 2 version and modules (System - About)

Icinga Web 2 Version 2.12.1
Git commit cd2daeb2cb8537c633d343a29eb76c54cd2ebbf2
PHP Version 8.2.20
Git commit date 2023-11-15
Loaded Libraries
icinga/icinga-php-library 0.14.0
icinga/icinga-php-thirdparty 0.12.1
Loaded Modules
director 1.11.1
grafana 2.0.3
icingadb 1.1.2
incubator 0.22.0
x509 1.3.2

Config validation (icinga2 daemon -C)

[2024-08-01 10:49:38 +0300] information/cli: Icinga application loader (version: r2.14.2-1)
[2024-08-01 10:49:38 +0300] information/cli: Loading configuration file(s).
[2024-08-01 10:49:39 +0300] information/ConfigItem: Committing config item(s).
[2024-08-01 10:49:39 +0300] information/ApiListener: My API identity: MON01.holding.com
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 6 NotificationCommands.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 14088 Notifications.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 IcingaApplication.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 21 HostGroups.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 557 Hosts.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 44 Downtimes.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 Influxdb2Writer.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 4 Comments.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 IcingaDB.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 FileLogger.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 143 Zones.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 CheckerComponent.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 33 Users.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 141 Endpoints.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 3 ApiUsers.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 ApiListener.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 NotificationComponent.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 470 CheckCommands.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 UserGroup.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 2 ServiceGroups.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 1 TimePeriod.
[2024-08-01 10:49:42 +0300] information/ConfigItem: Instantiated 5368 Services.
[2024-08-01 10:49:42 +0300] information/ScriptGlobal: Dumping variables to file ‘/var/cache/icinga2/icinga2.vars’
[2024-08-01 10:49:42 +0300] information/cli: Finished validating the configuration file(s).

lorenz · August 3, 2024, 11:55pm

Hi @Aleksey-Maksimov,
Could you share the exact command which is executed here with us?
I suspect, the problem lies there.

Aleksey-Maksimov · August 4, 2024, 7:42am

I specified the command in the first message. I also had a conversation with a colleague from Centreon and the idea was expressed that the problem is in the Icinga for Windows agent itself.

Aleksey-Maksimov · August 4, 2024, 8:05am

I tried different ways to run the commands

Example 1 (it works).
Simple call to the executable file of the plugin centreon_plugins.exe via the Icinga for Windows agent:

C:\ProgramData\icinga2\plugins\centreon_plugins\centreon_plugins.exe --version

Example 2 (it works).
Call to the executable file of the plugin centreon_plugins.exe via the Icinga for Windows agent. In this case, the called plugin itself uses a call to the system file w32tm.exe (centreon-plugins/src/os/windows/local/mode/ntp.pm at develop · centreon/centreon-plugins · GitHub):

C:\ProgramData\icinga2\plugins\centreon_plugins\centreon_plugins.exe --plugin os::windows::local::plugin -plugin="os::windows::local::plugin" --mode=time --ntp-hostname=-ntp.holding.com

OK: Time offset 0.010 second(s) | 'offset'=0.010s;;;;

Example 3 (it does not work).
Calling the executable file of the plugin centreon_plugins.exe via the Icinga for Windows agent. In this case, the called plugin itself uses a call to powershell.exe (centreon-plugins/src/os/windows/local/mode/pendingreboot.pm at develop · centreon/centreon-plugins · GitHub):

C:\ProgramData\icinga2\plugins\centreon_plugins\centreon_plugins.exe --plugin="os::windows::local::plugin" --mode="pending-reboot" --command "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" --command-options "-InputFormat none -NoLogo -NoProfile -ExecutionPolicy Bypass -EncodedCommand"

UNKNOWN: Internal error: execution issue: Access is denied

It looks like the problem is where the plugin centreon_plugins.exe calls powershell.exe. But I can’t find out what exactly the problem is.

lorenz · August 5, 2024, 2:52pm

Quite curious.

Now I am wondering: Does this fail when a Powershell is executed by the original Plugin? Or does the execution of the encoded command the fail?

Aleksey-Maksimov · August 5, 2024, 3:05pm

I did not understand the question

lorenz · August 5, 2024, 3:14pm

It is probably more a bit of talking to myself than a question. But, to be more elaborate: I see three options/reasons for that error here:

a)When executed by Icinga, the centreon-plugins somehow fail to execute powershell.exe itself. Which would be weird.

b)The powershell executed by centreon-plugins tries to execute some command or script (that’s what the person in the other thread on centreons side is talking about) and somehow we get a permission denied there. No idea why.

c)Something specific in that executed command (executed by …executed by …) needs more permissions.

How any of this occurs when the icinga2 agent is running as SYSTEM user, no idea …