[Solved] Run Python CGI in virtual environment?
Hi -- I am running python scripts successfully as CGI. Now my python process needs a library installed in a virtual environment. It works fine from the CLI with the following shebang:
#!/usr/bin/env -S VIRTUAL_ENV=/path/to/venv/ /path/to/venv/bin/python3
But it won't run via CGI (breakage.log indicates the python library can not be found). I changed lighttpd.conf to include mod_setenv:
server.modules = (
"mod_indexfile",
"mod_access",
"mod_alias",
"mod_setenv",
"mod_redirect",
"mod_auth",
"mod_cgi",
)
setenv.add-environment = (
"VIRTUAL_ENV" => "/path/to/venv/"
)
I also tried changing the python executable:
cgi.assign = ( ".sh" => "/bin/bash", ".py" => "/path/to/venv/bin/python3" )
...but that gave "permission denied".
What's the proper way to do this? Thanks!
Replies (9)
RE: Run Python CGI in virtual environment? - Added by chconnor 10 days ago
Apologies: lighttpd version 1.4.74 on ubuntu 24.04.1.
RE: Run Python CGI in virtual environment? - Added by gstrauss 10 days ago
You could try executing the .py directly if executing the .py directly from the CLI works.cgi.assign = ( ".sh" => "/bin/bash", ".py" => "" )
It works fine from the CLI
Does it work from CLI with /usr/bin/env - VIRTUAL_ENV=/path/to/venv/ /path/to/venv/bin/python3 ? If it does not work, then figure out what other env vars are needed.
env | sort
RE: Run Python CGI in virtual environment? - Added by chconnor 10 days ago
Thank you for the reply!
I tried ".py" => "", but same result.
I tried this shebang:
#!/usr/bin/env - VIRTUAL_ENV=/path/to/venv/ /path/to/venv/bin/python3
...but on the CLI it says:
/usr/bin/env: invalid option -- ' ' /usr/bin/env: use -[v]S to pass options in shebang lines Try '/usr/bin/env --help' for more information.
...I tried other stuff with the env command (e.g. tried -i and -i -S) but made no progress... it doesn't seem to like the -, and oddly if I pass just -i it also says "invalid option -- ' '" even though there is no blank option...
RE: Run Python CGI in virtual environment? - Added by gstrauss 10 days ago
The CLI is a shell. The shebang line inside a script is not the same thing.
$ /usr/bin/env - VIRTUAL_ENV=/path/to/venv/ /path/to/venv/bin/python3 /path/to/your-script.py
or
$ /usr/bin/env - VIRTUAL_ENV=/path/to/venv/ /path/to/your-script.py
or even (with your original shebang line)
$ /usr/bin/env - /path/to/your-script.py
Also, make sure you can run a trivially basic "Hello World" script using the default python interpreter on the system before you try to get your venv working.
RE: Run Python CGI in virtual environment? - Added by chconnor 10 days ago
Ah, sorry, misunderstood; from the CLI proper:
/usr/bin/env - VIRTUAL_ENV=/path/to/venv/ /path/to/venv/bin/python3 /path/to/your-script.py
...that works fine.
/usr/bin/env - VIRTUAL_ENV=/path/to/venv/ /path/to/your-script.py
...also works.
/usr/bin/env - /path/to/your-script.py
...also works.
In general, scripts with the default python interpreter work fine through lighttpd/CGI.
RE: Run Python CGI in virtual environment? - Added by gstrauss 10 days ago
If all of those worked, then ".py" => "" should have worked, but "something" in your system is getting in the way, giving you
...but that gave "permission denied".
Keep in mind that you have hidden information in your "summary" of /path/to/venv/ which is likely relevant, so please troubleshoot more on your own. You don't have to share precise details since you consider them private, but you also should take the hint and troubleshoot those details on your own.
If the (non-root) user under which lighttpd runs does not have permission to files under your home directory, and you put files under your home directory (hint: venv), you really should not be surprised by "permission denied" and you should carefully review how you do your troubleshooting before posting, and why when asking for help, you thought it was okay to hide useful information by posting /path/to/venv instead of /home/<user>/.venv.
RE: Run Python CGI in virtual environment? - Added by stbuehler 10 days ago
I work a lot with python virtual environments, and I never had to pass a VIRTUAL_ENV variable; running the interpreter or other (e.g. via [project.scripts]) installed scripts like gunicorn, django manage from venv/bin/ always worked just as it should.
In case you need to set "default" env vars in your python venv environment you could try adding / installing a `myenv.pth` file with content like this:
# *.pth files get autoloaded when present in python site-packages
# https://docs.python.org/3/library/site.html
# lines starting with "import" are executed.
#
# we assume we're always installed into a dedicated venv - default to our settings module
import os; os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
# could also do things like this:
# # also disable *.pyc generation
# import sys; sys.dont_write_bytecode = True
(This mechanism is also used when you install a package "editable" to extend the package search path.)
All in all this does sound like a permission problem (as gstrauss said); try running the script with the same user lighttpd is running as (probably www-data on debian based systems).
RE: Run Python CGI in virtual environment? - Added by chconnor 9 days ago
Thanks, all -- it was indeed permissions. Sorry for the obfuscation: just habitual anonymizing for the company I'm doing this for and I didn't realize it was relevant. I see now that it was.
I'm still not totally sure I'm doing things the smart way: I did "chown -R www-data:www-data venv" (venv is now in /var/www/html/bin/venv) and have the shebang set VIRTUAL_ENV and the executable to that venv's python3, and .py => "", so that sounds like what you all recommended, but wanted to make sure I wasn't doing anything stupid there.
RE: [Solved] Run Python CGI in virtual environment? - Added by gstrauss 9 days ago
Sorry for the obfuscation: just habitual anonymizing for the company I'm doing this for
With no acknowledgement that /home/<user>/.venv is also reasonably anonymized, your statement is a non-apology. Please do better.
(venv is now in /var/www/html/bin/venv)
It is generally a good idea to avoid putting files under the web server document-root (/var/www/html) if those files are not resources intended to be served by the web server. Did you notice that the files under venv are not html, and yet you have chosen to place them under /var/www/html/...? Have you protected URLs starting with /bin/ in the web server config to protect /var/www/html/bin/venv?
Similarly, it is generally a good idea for a company to do due diligence to determine that the person they are paying to set up their web server has a decent base of knowledge to do so. I recommend subcontracting to someone more knowledgeable in setting up the server-side of web servers to help you configure things in smarter ways.