Python 3.6 Support for Plugins?

thlucas · April 2, 2023, 5:28am

Hi Everyone.

Can the Icinga Monitoring Service execute custom plugins that are built using Python3 (v3.6)?

I have a custom plugin that I am writing (check_truenas.py) that keeps giving me the following error when trying to execute (e.g. “check now” via Icinga Web):

/usr/bin/env: python3
: No such file or directory

I know I have things defined correctly in Icinga Director for this, as I was executing a similar plugin (check_freenas.rb) that was written in Ruby. I just thought it would be fun to convert the Ruby to Python3, after upgrading my FreeNAS from 11.3-U5 to TrueNAS Core 12.0.

I can execute the plugin from an SSH session, with both the root and “sudo -u icinga …” user, and it returns the expected results. The Icinga Service seems to be trying to use the Python 2.7 environment when executing from there though.

I have restarted the Icinga service after installing Python 3.6, but have not rebooted the host. I did not restart Icinga Director or Icinga Web yet.

Any help you can give would be fantastic.

Thanks - Todd

My Environment:

CentOS Linux release 7.9.2009 (Core), running in a ESXi VM instance
Icinga Web 2 Version 2.11.4
Icinga Director 1.10.2
Icinga Monitoring 2.11.4
PHP Version 7.3.33
Git commit date 2023-01-26
Chrome browser, Version 111.0.5563.147 (Official Build) (64-bit)
Python version 2.7
Python version 3.6

[root@centos7vm scripts]# pip --version
pip 8.1.2 from /usr/lib/python2.7/site-packages (python 2.7)
[root@centos7vm scripts]# pip3 --version
pip 9.0.3 from /usr/lib/python3.6/site-packages (python 3.6)

Pooh · April 2, 2023, 8:07am

What is the very first line (starting with #!) of your custom script?
Where is your python 2.7 binary located?
Where is your python 3.6 binary located?
What does “sudo -u icinga echo $PATH” tell you?

Antony.

rivad · April 3, 2023, 9:12am

Icinga doesn’t care which python version you use as it just gives the shell a file to execute and the shell needs to decide how to handle the file. The shebang #! in the fist line is helping to find the shell how to handle the file. From your error message, I guess your file states #!/usr/bin/env python3 on the first line.

Do you get the same error if you call /usr/bin/env python3 in a terminal?
What do you get in the terminal if you type python and then press TAB twice?

My guess, you miss the symlink for python3 to python3.6.
Have a look at rhel - On RHEL8 how can I use `alternatives` to choose an already listed version as the used version - Unix & Linux Stack Exchange to fix this or alternatively you could change #!/usr/bin/env python3 to #!/usr/bin/env python3.6.

thlucas · April 3, 2023, 4:40pm

Hi Dominik - Thanks for your reply.

Here’s what I get when I call “/usr/bin/env python3” from a Putty SSH terminal:
[root@centos7vm ~]# /usr/bin/env python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

Here’s what I get when entering “python ”:
[root@centos7vm ~]# python
python python2.7 python2-config python3.6 python-config
python2 python2.7-config python3 python3.6m
[root@centos7vm ~]# python

I will take a look at the link you supplied concerning the alternatives

My concern is that I’m not sure if I want to set the system-wide “default” Python version to 3.6.8 if it’s currently set to 2.7. There are other things running on this host, and I don’t want to cause any issues with those applications. I am fairly new to Python, so not sure if that is a concern since it appears the shebang statement indicates if it will use Python3 or a lower version of Python?

I appreciate your help.

thlucas · April 3, 2023, 4:51pm

Hi Antony - answers to your questions …

#1 - first line is: #!/usr/bin/env python3

#2 - Python 2.7 binary location:

[root@centos7vm ~]# which python
/usr/bin/python

#3 - Python 3.6 binary location:

[root@centos7vm ~]# which python3
/usr/bin/python3

#4 - “sudo -u icinga echo $PATH” results:

[root@centos7vm ~]# sudo -u icinga echo $PATH
/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/plugins/ruby-build/bin:/root/.rbenv/shims:/root/.rbenv/bin:/usr/local/rvm/gems/ruby-2.7.2/bin:/usr/local/rvm/gems/ruby-2.7.2@global/bin:/usr/local/rvm/rubies/ruby-2.7.2/bin:/opt/rh/rh-ruby25/root/usr/local/bin:/opt/rh/rh-ruby25/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin
[root@centos7vm ~]#

Could it simply be that I need to add “/usr/bin/python3” to the PATH?

I’m new to Python, and fairly new to CentOS7, so that was not a rhetorical question.

Another odd thing is that the “python” and “python3” commands appear to start correctly when issued from a sudo -u icinga command. One would think that I would not have to modify $PATH if that were not the case.

[root@centos7vm ~]# sudo -u icinga python
Python 2.7.5 (default, Jun 28 2022, 15:30:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
[root@centos7vm ~]# sudo -u icinga python3
Python 3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()

thlucas · April 3, 2023, 11:54pm

So I tried modifying the 1st line of the script to use #!/usr/bin/python3 instead of the #/usr/bin/env python3 line. It changed the error message, but still returns an error.

To be clear:

line 1:  #!/usr/bin/python3
result:  execvpe(/etc/icinga2/scripts/check_truenas.py) failed: No such file or directory

line 1:  #/usr/bin/env python3
result:  /usr/bin/env: python3
            : No such file or directory

I rebooted the host, and that did not help.

I also did a yum install for every ‘imports’ that appears in my script, thinking the “No such file” error was due to a missing import. The only thing installed was “json”.

Here’s the script execution and result executed from a Putty SSH shell using root:

[root@centos7vm scripts]# python3 check_truenas.py -s nas2.netlucas.com -u root -p xx -w 80 -c 90 -m zpool_usage -z ZMPR502
OK - ZMPR502: status ONLINE, 67% used (3.34T available, 7.07T used). | ZMPR502=67%;80;90

Here’s the script execution and result executed from a Putty SSH shell using sudo -u icinga:

[root@centos7vm scripts]# sudo -u icinga python3 check_truenas.py -s nas2.netlucas.com -u root -p xx -w 80 -c 90 -m zpool_usage -z ZMPR502
OK - ZMPR502: status ONLINE, 67% used (3.34T available, 7.07T used). | ZMPR502=67%;80;90

Here’s a directory listing of the /usr/bin/python* directory entries:

[root@centos7vm scripts]# ls -la /usr/bin/python*
lrwxrwxrwx. 1 root root     7 Jan 26 17:09 /usr/bin/python -> python2
lrwxrwxrwx. 1 root root     9 Jan 26 17:09 /usr/bin/python2 -> python2.7
-rwxr-xr-x. 1 root root  7144 Jun 28  2022 /usr/bin/python2.7
-rwxr-xr-x. 1 root root  1835 Jun 28  2022 /usr/bin/python2.7-config
lrwxrwxrwx. 1 root root    16 Jan 27 01:12 /usr/bin/python2-config -> python2.7-config
lrwxrwxrwx. 1 root root     9 Apr  1 23:20 /usr/bin/python3 -> python3.6
-rwxr-xr-x. 2 root root 11328 Nov 16  2020 /usr/bin/python3.6
-rwxr-xr-x. 2 root root 11328 Nov 16  2020 /usr/bin/python3.6m
lrwxrwxrwx. 1 root root    14 Jan 27 01:12 /usr/bin/python-config -> python2-config

I also just noticed there is no “python3-config” file or symlink in /usr/bin. Could that be the problem? A which python3-config command just displays the $PATH variable contents of where it searched.

[root@centos7vm scripts]# which python-config
/usr/bin/python-config
[root@centos7vm scripts]# which python2-config
/usr/bin/python2-config
[root@centos7vm scripts]# which python3-config
/usr/bin/which: no python3-config in (/usr/local/rvm/gems/ruby-2.7.0/bin:/usr/local/rvm/gems/ruby-2.7.0@global/bin:/usr/local/rvm/rubies/ruby-2.7.0/bin:/usr/local/rvm/bin:/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/plugins/ruby-build/bin:/root/.rbenv/shims:/root/.rbenv/bin:/opt/rh/rh-ruby25/root/usr/local/bin:/opt/rh/rh-ruby25/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)

thlucas · April 4, 2023, 12:14am

Dunno why, but Akismet (spam filter) hid my latest post. Probably because I edited it 3 times after noticing typos and such. Hopefully it will re-appear soon. Sorry for the trouble.

moreamazingnick · April 4, 2023, 5:30am

Can you try to put that in the first line of the script instead of #!/usr/bin/env python3

#!/usr/bin/env /usr/bin/python3

another thing, do you have env on your system? try which env

rivad · April 4, 2023, 7:04am

line 1:  #!/usr/bin/python3
result:  execvpe(/etc/icinga2/scripts/check_truenas.py) failed: No such file or directory

is /etc/icinga2/scripts/check_truenas.py executable?
if not you can make it executable by:
sudo chmod +x /etc/icinga2/scripts/check_truenas.py

I would not change python with alternatives but if python3 would not point to python3.6 it would have helped maybe.

Did you try #!/usr/bin/python3.6 in the first line?

thlucas · April 4, 2023, 4:34pm

@Dominik - script is executable (see below). I have also ran the script from the Putty SSH session using root and “sudo -u icinga” successfully.

[root@centos7vm scripts]# ls -la /etc/icinga2/scripts
-rwxr-xr-x. 1 root   root   18006 Apr  4 11:28 check_truenas.py

I did try the #!/usr/bin/python3.6 in the first line, but still received the error:
execvpe(/etc/icinga2/scripts/check_truenas.py) failed: No such file or directory

thlucas · April 4, 2023, 4:41pm

@Moreamazingnick -
I get the following error when using the #!/usr/bin/env /usr/bin/python3 as line 1:

/usr/bin/env: /usr/bin/python3
: No such file or directory

I also tried #!/usr/bin/env /usr/bin/python3.6 that yields the same error:

/usr/bin/env: /usr/bin/python3.6
: No such file or directory

The which env command displays the following:

[root@centos7vm scripts]# which env
/usr/bin/env

rivad · April 5, 2023, 6:24am

So it tells you, the file that’s there isn’t but only for Icinga run by systemd.
Did you jail it? Can Icinga execute other files in the same folder? Any log entries with this filename - especially selinux?

Peer-Mario · April 5, 2023, 12:40pm

Hi, just to jump in:

I used to use for my python 3.9 this 1 line:

#!/usr/bin/env python3.9

and it works like a charm

Regards
Peer-Mario

thlucas · April 5, 2023, 4:17pm

@rivad - I have not jailed it, as there should not be a need to.

Icinga can execute other files in the etc/icinga2/scripts/ folder, as this is where the “check_freenas.rb” Ruby script was executing for my old version of FreeNAS.

Log entries:

[root@centos7vm scripts]# tail /var/log/icinga2/icinga2.log
[2023-04-05 10:22:53 -0500] warning/PluginCheckTask: Check command for object 'NAS2!TrueNAS Check ZPOOL ZMPR501' (PID: 18996, arguments: '/etc/icinga2/scripts/check_truenas.py' '-s' 'NAS2' '-u' 'root' '-p' 'xx' '-k' 'true' '-m' 'zpool_usage') terminated with e: No such file or directory bin/env: python

I’m pretty sure this is a Python install related error, and not Icinga itself. This is not a new Icinga build, and has been working fine for years monitoring my FreeNAS server. The only things I did on the CENTOS7 Icinga2 Host server after upgrading my FreeNAS server (different server) to TrueNAS was:

install stable release of Python 3 via a “yum install python3” command.
convert the old FreeNAS Ruby script to Python3 script (check_truenas.py).

So to narrow things down further, I created a very simple Python3 script that simply displays the python version in effect at run-time. I know this does not conform to Icinga2 monitoring methods, but I don’t care about that at this point. Here is the script:

#!/usr/bin/env python3
import platform;
print(platform.sys.version);

Here is the file permissions:

[root@centos7vm scripts]# ls -l
-rwxr-xr-x. 1 root root 18141 Apr  4 14:23 check_truenas.py
-rwxr-xr-x. 1 root root    71 Apr  5 10:47 testver.py

Here is the output in the Icinga2 web interface after the check (same as my script):

/usr/bin/env: /usr/bin/python3
: No such file or directory

Here is the output in the Putty SSH session when running from the command line:

[root@centos7vm scripts]# python3 testver.py
3.6.8 (default, Nov 16 2020, 16:55:22)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
[root@centos7vm scripts]# python testver.py
2.7.5 (default, Jun 28 2022, 15:30:04)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]

I appreciate everyone’s help, but at this point I think I need to figure out if something is wrong with my Python3 install. The problem with that is I am totally new to Python, and fairly new to Unix systems, so am fumbling around trying different things. I will keep you updated if I make any progress.

Thanks for your help.

thlucas · April 5, 2023, 5:12pm

@Peer-Mario
Can you show me what your “CheckCommand” object looks like when executing your Python3 script? I am wondering if I need to specify the “python” or “python3” prefix in order to execute, but am not for sure how that would look in the CheckCommand object arguments. I tried the following, but it did not like the space between the python and the script path:
python3 /etc/icinga2/scripts/testver.py

Note that I am using Icinga Director to configure my objects.

Here is my CheckCommand object for my current test of a Python3 script:

object CheckCommand "TrueNAS_Check_Python_Version" {
    import "plugin-check-command"
    command = [ "/etc/icinga2/scripts/testver.py" ]
    timeout = 30s
    arguments += {
        "-k" = {
            description = "This is a test argument."
            value = "true"
        }
    }
}

moreamazingnick · April 5, 2023, 5:20pm

you don’t need too, because of the first line of the script + the script has execute permissions.

but you can try something else:

object CheckCommand "TrueNAS_Check_Python_Version" {
    import "plugin-check-command"
    command = [ "/usr/bin/python3" ]
    timeout = 30s
    arguments += {
        "-c" = {
            description = "the script to execute"
            value = "/etc/icinga2/scripts/testver.py"
            order = 0
        },
        "-k" = {
            description = "This is a test argument."
            value = "true"
        }
    }
}

thlucas · April 5, 2023, 5:56pm

I found the problem! It was a very stupid overlook on my part, but at the end of the day I learned something.

As it turns out, the issue was caused by a CRLF vs LF setting when editing my Python script!

I am doing all of my development on Windows using Visual Studio 2022. As part of my setup, I chose to edit files using standard Windows line endings which are carriage-return linefeed (CRLF) sequence. The Unix standard is linefeed (LF) sequence. Me being a Windows guy, this was not much of a concern to me - but apparently it’s a big deal to the Unix command-line interpreter!

The fix was simple: inform my Visual Studio 2022 to use LF sequence, save the changes, and redeploy the script to the CentOS7 server.

It is now working great with the #!/usr/bin/env python3 shebang line.

For other Windows VS 2022 friends, you simply have to change the “CRLF” to “LF” in the lower-right hand corner of the VS 2022 editor window. Here’s a screen capture of those settings:

I appreciate everyone’s help with this, and many apologies for my simple mistake.

@rivad - I was able to get the alternatives processing to work, so am ready now for more / future versions of Python. Thank you for pointing me in that direction.

rivad · April 6, 2023, 7:01am

Invisible characters in error messages are a problem. Some times hd (hexdump) helps to spot things.

Knowing about alternatives is important for a administrator and it’s also present in the Debian family of distributions.