Execvpe(/usr/lib/check_service.exe) failed: No such file or directory

Hello everybody,

General Informations:
version: r2.13.7-1
Ubuntu 22.04.2
Enabled features: api checker icingadb ido-mysql mainlog notification

Im trying to monitor some Windows-Services from my Windows-Agent, but the chekcs cant be executed.
Error:
execvpe(/usr/lib/check_service.exe) failed: No such file or directory

Please dont wonder, I didnt used zones seperations or seperated everything clear in services.conf etc.
I wrote nearly everything directly into the host.conf File (shared below)

object Host "XXX" {
  import "generic-host"
  address = "XXX"
  check_command = "hostalive"
  vars.os = "Windows"
  vars.notify_users = [ "XXX" ]
  vars.notification.teams = true
  vars.notification["mail"] = {
     groups = [ "icinga_mail_notification" ]
  }
  vars.disks_windows["disk /"] = {
      disk_win_path = "C:"
    }
}

object Service "XXX" {
  host_name = "XXX"
  check_command = "service-windows"
  vars.service_win_service = "XXX"
  command_endpoint = host.name
}

This config file is valid.
Icinga is now saying: execvpe(/usr/lib/check_service.exe) failed: No such file or directory

So I tought, he couldnt find the check_service.exe in /usr/lib/ and yes, there is not .exe or something.

I changed in the /usr/share/icinga2/include/command-plugins-windows.conf the

object CheckCommand "service-windows" {
        command = [ PluginDir + "/check_service.exe" ]

to

object CheckCommand "service-windows" {
        command = [ "C:/Program Files/ICINGA2/sbin/check_service.exe" ]

Under my Windows Agent, the exe File (and many others) are available.
But the checks are still not working.

Error message now:
execvpe(C:/Program Files/ICINGA2/sbin/check_service.exe) failed: No such file or directory

So my question is, what could be wrong here?

I have tried nearly everything I found belonging to this topic I found in any forums…

Any Idea on this?
@dnsmichi maybe you can have alook at this.
I have seen you in previous posts.

Hi there

Don’t do that. These aren’t files that are meant to be changed. Your changes will get lost with the next update.

If you want to create your own check command config put it under /etc/icinga2 in the conf.d or even better the zones.d/<your master zone name> folder.

For the command_endpoint option in the service to work you will need to tell Icinga that that host is an agent, meaning you also need an endpoint object and a zone object.
See the docs:
Agent/satellite Setup
Configuration modes
Scenario - Master with Agents

I would also suggest creating a service template and then an apply rule instead of just a single service configuration in future, because it is more flexible while deploying checks to more than one host object.

The PluginDir constant will point to the correct directory depending on the operating system the agent is installed on. No need to change that manually

Hi @log1c
many thanks for your reply and your help.

I changed the /usr/share/icinga2/include/command-plugins-windows.conf back to the default value

object CheckCommand "service-windows" {
        command = [ PluginDir + "/check_service.exe" ]

I already made a zones setup in /etc/icinga2/zones.conf

object Endpoint "XXXMaster" {
}

object Endpoint "XXXAgent" {
  host = "123.123.123.123"
}

object Zone "master" {
        endpoints = [ "XXXMaster" ]
}

object Zone "XXXAgent" {
        endpoints = [ "XXXAgent" ]

        parent = "master"
}

object Zone "global-templates" {
        global = true
}

object Zone "director-global" {
        global = true
}

I understand that for a better overview it would be helpful not to write everything in the host.conf fileu.
However, I want to implement my requirements for the test firstinmal, these I can then at the end, if everything works still rebuild.

After all changes, the check still doenst work and the error message appears:

execvpe(/usr/lib/nagios/plugins/check_service.exe) failed: No such file or directory

Do you have any ideas?

Change this to command_endpoint = host_name.

Run icinga2 daemon -C to check for configuration errors and the reload the icinga2 service to deploy the new configuration.

I think host.name is only usable in apply rules, though I#m not 100% sure.

I made some changes:

  • i moved the hosts.conf from /etc/icinga2/conf.d/ to /etc/icinga2/zones.d/master and I changed in the hosts.conf the command_endpoint = host.name to command_endpoint = host_name but no difference.
    I tried something like “localhost” and agentname and stuff but this syntax didnt work.

icinga2 daemon -c output:

root@XXX:/etc/icinga2/zones.d/master# icinga2 daemon -C
[2023-05-04 12:36:58 +0000] information/cli: Icinga application loader (version: r2.13.7-1)
[2023-05-04 12:36:58 +0000] information/cli: Loading configuration file(s).
[2023-05-04 12:36:58 +0000] information/ConfigItem: Committing config item(s).
[2023-05-04 12:36:58 +0000] information/ApiListener: My API identity: XXX
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 39 Notifications.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 IcingaApplication.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 2 HostGroups.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 2 Hosts.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 FileLogger.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 Downtime.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 IdoMysqlConnection.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 CheckerComponent.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 IcingaDB.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 4 Zones.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 2 Endpoints.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 2 ApiUsers.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 ApiListener.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 NotificationComponent.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 244 CheckCommands.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 2 UserGroups.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 3 ServiceGroups.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 2 TimePeriods.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 3 Users.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 1 ScheduledDowntime.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 24 Services.
[2023-05-04 12:36:58 +0000] information/ConfigItem: Instantiated 3 NotificationCommands.
[2023-05-04 12:36:58 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
[2023-05-04 12:36:59 +0000] information/cli: Finished validating the configuration file(s).

Is

exactly the same as

? If not, the host object is not recognized as an agent.
I missed that until now.

Also check if the agent is correctly connected to your master (see logs on both sides) and received it’s configuration.

If it is both the same, then please test this:

Create two service templates

template Service "generic-service-template" {
    max_check_attempts = "3"
    check_interval = 5m
    retry_interval = 2m
    enable_notifications = true
    enable_active_checks = true
    enable_perfdata = true
}
template Service "windows-agent-service" {
    import "generic-service-template"

    check_command = "service-windows"
    command_endpoint = host_name

and then create a service apply rule

apply Service "service-XXX" {
    import "windows-agent-service"
    vars.service_win_service = "XXX"

    assign where host.name == "XXX"

Yes

object Endpoint "XXXAgent"

is the same as

object Host "XXX"

I think the agent is succesfully connected to the master.
But to make sure, I reconfigured the Agent (Icinga2Setup) on the Windows Client and connected succesfullly to the master.

While checking logs, I coulnt find any logs on the windows agent located in C:\ProgramData\icinga2\var\log\icinga2\icinga2.log

When running the PowerShell Command Read-IcingaAgentLogFile;
I get [Error]: Icinga 2 debug logfile not present. Unable to load it

Maybe the error is inside my agent setup…

What data can I provide you, to get a look at this?

Ok I think it is definitly a agent base problem.
i have now uninstalled every components / frameworks from the Windows Agent and the monitoring (Icinga2Web) is still showing the agent as up.
Im not sure, whats the best way to install icinga2 on a windows host and how to configure it right.

I usually installed the .msi and connected it to the master and then did on the master CLI
icinga2 ca sign ID

Did I forget smth?

Please show what you mean by that. Which check? Where is it executed?

The agent in the newer version (starting with 2.13.4 I think) will log into the eventlog instead of a log file by default.

That should work. Do you get configuration after this under C:\ProgramData\icinga2\var\lib\icinga2\api ? If not then the agent is not connected correctly

Hi @log1c
thanks for your reply.

Please show what you mean by that. Which check? Where is it executed?

When I uninstall the icinga2 software/component/… on the agent, and I still have the Hostname included in the zones.conf with a simple host_alive:

object Host "Agent" {
  import "generic-host"
  address = "10.1.27.23"
  check_command = "hostalive"
}

I still see the Agent in the Monitoring with Green Status (UP).
Why?

I now reinstalled the software on the agent and reconnected it to the master succesfully (i think).
By checking the logs on the agent in Eventlog, there always show up these two messages:

warning/ApiListener: Unexpected certificate common name while connecting to endpoint 'master.oldfqdn': got 'master.newfqdn'
information/ApiListener: Finished reconnecting to endpoint 'master.oldfqdn' via host 'IP' and port '5665'

I had to change the domain name and I changed every config files on the master to the new FQDN, createt new Certificates for master, …

I dont know, why and how he can connect to the old hostname + fqdn (Domainname)?

When I open the C:\ProgramData\icinga2\var\lib\icinga2\api
there are folders:
config (empty)
log (empty)
packages
image
zones (empty)

Because hostalive is just a simple ping to the ip address.

If you want to check if the agent is working correctly you can use the icinga check command as well as the cluster-zone check command combined with the argument var cluster_zone
icinga
cluster-zone

Make sure that you delete the whole %ProgramData%/icinga2 folder after uninstalling, else you could have old certificates or config remaining.

Please share the C:\ProgramData\icinga2\etc\icinga2\zones.conf, C:\ProgramData\icinga2\etc\icinga2\icinga2.conf and C:\ProgramData\icinga2\etc\icinga2\constants.conf files from the agent after you installed

Okay thanks.

Thats not working on my Windows Agent.
wether in powershell or in cmd there is no command like “icinga” or smth.
The icinga2 Service is running.

I have uninstalled the icinga software from agent and deleted the icinga2 folder under %ProgrammData%

Now im installting again the Icinga2-v2.13.7-x86_64.msi

And signed the certificate request on the master via. icinga2 ca sign ID

Event Log on agent says now:
warning/ApiListener: Unexpected certificate common name while connecting to endpoint 'master': got 'hostname.fqdn'

information/ApiListener: Finished reconnecting to endpoint 'master' via host 'hostname.fqdn' and port '5665'

He is reconnecting every 2 min…
(At least he now uses the right FQDN and not the old Domain :slight_smile:

Shared agent data below:
C:\ProgramData\icinga2\etc\icinga2\zones.conf

object Endpoint "master" {
  host = "mastername"
  port = "5665"
}

object Zone "master" {
  endpoints = [ "master" ]
}

object Endpoint "agentname" {
}

object Zone "agentname" {
  endpoints = [ "agentname" ]
  parent = "master"
}

object Zone "global-templates" {
  global = true
}

object Zone "director-global" {
  global = true
}

Is equal to master.

C:\ProgramData\icinga2\etc\icinga2\icinga2.conf

/**
 * Icinga 2 configuration file
 * - this is where you define settings for the Icinga application including
 * which hosts/services to check.
 *
 * For an overview of all available configuration options please refer
 * to the documentation that is distributed as part of Icinga 2.
 */

/**
 * The constants.conf defines global constants.
 */
include "constants.conf"

/**
 * The zones.conf defines zones for a cluster setup.
 * Not required for single instance setups.
 */
include "zones.conf"

/**
 * The Icinga Template Library (ITL) provides a number of useful templates
 * and command definitions.
 * Common monitoring plugin command definitions are included separately.
 */
include <itl>
include <plugins>
include <plugins-contrib>
include <manubulon>

/**
 * This includes the Icinga 2 Windows plugins. These command definitions
 * are required on a master node when a client is used as command endpoint.
 */
include <windows-plugins>

/**
 * This includes the NSClient++ check commands. These command definitions
 * are required on a master node when a client is used as command endpoint.
 */
include <nscp>

/**
 * The features-available directory contains a number of configuration
 * files for features which can be enabled and disabled using the
 * icinga2 feature enable / icinga2 feature disable CLI commands.
 * These commands work by creating and removing symbolic links in
 * the features-enabled directory.
 */
include "features-enabled/*.conf"

/**
 * Although in theory you could define all your objects in this file
 * the preferred way is to create separate directories and files in the conf.d
 * directory. Each of these files must have the file extension ".conf".
 */
// Disabled by the node setup CLI command on 2023-05-09 12:32:16 -0000
// include_recursive "conf.d"

Is equal to master except include_recursive "conf.d" this is on the master icinga2.conf not commented out.

C:\ProgramData\icinga2\etc\icinga2\constants.conf

/**
 * This file defines global constants which can be used in
 * the other configuration files.
 */

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = PrefixDir + "/sbin"

/* The directory which contains the Manubulon plugins.
 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
 */
const ManubulonPluginDir = PrefixDir + "/sbin"

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
 * Check the documentation, chapter "Plugins Contribution", for details.
 */
const PluginContribDir = PrefixDir + "/sbin"

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "agentname"

/* Our local zone name. */
const ZoneName = "agentname"

/* Secret key for remote node tickets */
const TicketSalt = ""

The constants.conf on the master looks different:

/**
 * This file defines global constants which can be used in
 * the other configuration files.
 */

/* The directory which contains the plugins from the Monitoring Plugins project. */
const PluginDir = "/usr/lib/nagios/plugins"

/* The directory which contains the Manubulon plugins.
 * Check the documentation, chapter "SNMP Manubulon Plugin Check Commands", for details.
 */
const ManubulonPluginDir = "/usr/lib/nagios/plugins"

/* The directory which you use to store additional plugins which ITL provides user contributed command definitions for.
 * Check the documentation, chapter "Plugins Contribution", for details.
 */
const PluginContribDir = "/usr/lib/nagios/plugins"

/* Our local instance name. By default this is the server's hostname as returned by `hostname --fqdn`.
 * This should be the common name from the API certificate.
 */
const NodeName = "master"

/* Our local zone name. */
const ZoneName = "master"

/* Secret key for remote node tickets */
const TicketSalt = "0e5360425021b49443555037c06eb9a6"

Ok, this points to different nameings in your config files and what icinga2 has inside its own CA.
When you run the icinga2 node wizard to setup your master the certificate common name (and there later on the endpoint name) for the master will be set to the FQDN (pbtained from hostname --fqdn) unless you specify something else. See Distributed Monitoring - Icinga 2.

Now you configured your agent to connect against “master”, but the master itself knows itself as “hostname.fqdn”. So try to connect your agent against “hostname.fqdn” when running the wizard.
Also (afaik) you need to enable at least one of the “Accept …” boxes to be able to run checks on the agent. There is a detailed doc page about it here
Also I would suggest re-reading the part about the windows agent installation: Distributed Monitoring - Icinga 2

These are both “internal” commands of icinga and can only be executed as checks as far as I know. Both need the command_endpoint attribute in the check config.

Also just to double-check:
The host objects name for your agent host

is exactly the same as

?

conf.d holds example configuration. If you don’t want to use that, comment it out.

That is to be expected, as the NodeName and ZoneName constants are host-specific.

Ok I have now updatetd the names in the configs/certificates on every files (master & agent) and this should be fine now.

Also just to double-check:
The host objects name for your agent host

is exactly the same as

?

Yes, its the same.

I will paste the zones.conf and hosts.conf just to make sure they are fine.

zones.conf

object Endpoint "masterFQDN" {
  host = "masterFQDN_IP"
}

object Zone "master" {
  endpoints = [ "masterFQDN" ]
}

object Endpoint "agentFQDN" {
  host = "agentFQDN_IP"
}

object Zone "agentFQDN" {
  endpoints = [ "agentFQDN" ]
  parent = "master"
}

object Zone "global-templates" {
  global = true
}

object Zone "director-global" {
  global = true
}

hosts.conf

object Host "agentFQDN" {
  import "generic-host"
  address = "agentFQDN_IP""
  check_command = "hostalive"
  enable_notifications = true
  vars.os = "Windows"
  vars.agent_endpoint = name
  vars.notify_users = [ "myuser1" ]
  vars.notification.teams = true
  vars.notification["email"] = {
     groups = [ "icinga_mail_notification" ]
  }
  vars.disks_windows["disk C:"] = {
      disk_win_path = "C:"
    }
}

/*object Service "WinDisk" {
  import "generic-service"
  host_name = "agentFQDN"
  check_command = "disk-windows"
  command_endpoint = host_name
  vars.disk_win_show_used = “true”
  vars.disk_win_warn = “80%”
  vars.disk_win_crit = “90%”
} Doenst work, maybe you can have a look at this too? */

object Service "XY WinService" {
  host_name = "agentFQDN"
  check_command = "service-windows"
  vars.service_win_service = "XY WinService"
  command_endpoint = host.vars.agent_endpoint
}

object Service "XY WinService" {
  host_name = "agentFQDN"
  check_command = "service-windows"
  vars.service_win_service = "XY WinService"
  command_endpoint = host.name
}

I’m really starting to get lost.
I have added now a check_user Service

object Service "User" {
  host_name = "agentFQDN"
  check_command = "users-windows"
  vars.agent_endpoint = name
  command_endpoint = host.agent_endpoint
}

I have tried multiple things for command_endpoint:

command_endpoint = host.name
command_endpoint = host_name
command_endpoint == host.name
command_endpoint = host.vars.agent_endpoint

Nothing worked.
Error message is like for windows-service:

execvpe(/usr/lib/nagios/plugins/check_users.exe) failed: No such file or directory

Even if I rebuild my entire environment, it does not work and the same error message appears.
I have the hosts.conf under zones.d/master/hosts.conf

object Host "agentFQDN" {
  check_command = "hostalive"
  address = "agentIP"
  vars.agent_endpoint = name
}

and under zones.d/master/services.conf

apply Service "WinService Test" {
  check_command = "service-windows"
  command_endpoint = host.vars.agent_endpoint
  assign where host.vars.agent_endpoint
}

stored.
service icinga2 checkconfig runs valid through, reload, no change…
I am getting desperate.

Inexplicably, the check from the agent now works?
Hes telling me now:
Error: Non-optional macro 'service_win_service' used in argument '-s' is missing.

Im gonna work on that and let you know, when I found a solution!

Next error :smiley:
Remote Icinga instance ‘AgentFQDN’ is not connected to 'masterFQDN’Problem handling

Im was checking debug.log and icinga2.log and saw this:

[2023-05-12 11:45:17 +0000] information/IdoMysqlConnection: Pending queries: 0 (Input: 3/s; Output: 3/s)
[2023-05-12 11:45:19 +0000] information/JsonRpcConnection: Closing anonymous connection [::ffff:10.1.27.23]:65509 after 10 seconds.
[2023-05-12 11:45:19 +0000] warning/JsonRpcConnection: API client disconnected for identity 'AgentFQDN'
[2023-05-12 11:45:19 +0000] information/ApiListener: New client connection for identity 'AgentFQDN from [::ffff:10.1.27.23]:49152 (certificate validation failed: code 18: self-signed certificate)
[2023-05-12 11:45:22 +0000] critical/ApiListener: Timeout while reconnecting to endpoint 'AgentFQDNt' via host '10.1.27.23' and port '5665', cancelling attempt
[2023-05-12 11:45:22 +0000] critical/ApiListener: Cannot connect to host '10.1.27.23' on port '5665': Operation canceled
[2023-05-12 11:45:27 +0000] information/ApiListener: Reconnecting to endpoint 'AgentFQDN' via host '10.1.27.23' and port '5665'
[2023-05-12 11:45:29 +0000] warning/JsonRpcConnection: API client disconnected for identity 'AgentFQDN'
[2023-05-12 11:45:29 +0000] information/ApiListener: New client connection for identity 'AgentFQDNt' from [::ffff:10.1.27.23]:49161 (certificate validation failed: code 18: self-signed certificate)
[2023-05-12 11:45:39 +0000] information/ApiListener: New client connection for identity 'AgentFQDN' from [::ffff:10.1.27.23]:49165 (certificate validation failed: code 18: self-signed certificate)
[2023-05-12 11:45:39 +0000] information/JsonRpcConnection: Closing anonymous connection [::ffff:10.1.27.23]:49161 after 10 seconds.
[2023-05-12 11:45:39 +0000] warning/JsonRpcConnection: API client disconnected for identity 'AgentFQDN'
[2023-05-12 11:45:42 +0000] critical/ApiListener: Timeout while reconnecting to endpoint 'AgentFQDN.net' via host '10.1.27.23' and port '5665', cancelling attempt
[2023-05-12 11:45:42 +0000] critical/ApiListener: Cannot connect to host '10.1.27.23' on port '5665': Operation canceled
[2023-05-12 11:45:47 +0000] information/ApiListener: Reconnecting to endpoint 'AgentFQDN.net' via host '10.1.27.23' and port '5665'
[2023-05-12 11:45:49 +0000] warning/JsonRpcConnection: API client disconnected for identity 'AgentFQDN.net'
[2023-05-12 11:45:49 +0000] information/ApiListener: New client connection for identity 'AgentFQDN' from [::ffff:10.1.27.23]:49175 (certificate validation failed: code 18: self-signed certificate)
[2023-05-12 11:45:59 +0000] warning/JsonRpcConnection: API client disconnected for identity 'AgentFQDN'
[2023-05-12 11:45:59 +0000] information/ApiListener: New client connection for identity 'AgentFQDN' from [::ffff:10.1.27.23]:49185 (certificate validation failed: code 18: self-signed certificate)
[2023-05-12 11:46:02 +0000] critical/ApiListener: Timeout while reconnecting to endpoint 'AgentFQDN' via host '10.1.27.23' and port '5665', cancelling attempt
[2023-05-12 11:46:02 +0000] critical/ApiListener: Cannot connect to host '10.1.27.23' on port '5665': Operation canceled
[2023-05-12 11:46:07 +0000] information/ApiListener: Reconnecting to endpoint 'AgentFQDN' via host 'agentIP' and port '5665'

add this as a variable to your service config to define the serviec you want to check.

icinga2 ca list doesn’t show any open signing requests?

Because

signals that the agents certificate is not yet signed by the Icigna CA.

huh, never seen that command before. Not sure what it runs in the background.
I recommend running icinga2 daemon -C to validate the configuration.