Install in Python virtual environment

Bug #1784132 reported by Matthias Homann
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Rapid Photo Downloader
Fix Released
Wishlist
Matthias Homann

Bug Description

If I install Rapid Photo Downloader on Ubuntu 16.04, then it creates conflicts with other Python based programs like Retext. An existing Restext installation doesn't work afterwards because RPD is istalling another (incolmplete?) version of PyQT5 causing the Retext error "ImportError: No module named 'PyQt5.QtWebKit'" at startup.

It would be much better if the RPD install script would allow to install in a Python virtual environment to avoid such issues. I tried to run the script in a venv (https://docs.python.org/3/library/venv.html), but it still installed all libraries outside the venv.

Tags: venv

Related branches

Revision history for this message
Damon Lynch (dlynch3) wrote :

Hi Matthias,

Thanks for your bug report. This is one of those features that would be nice to have, but to be honest I have a list of tasks for the project that are much higher priority to implement. And right now, I have no time whatsoever to implement any of them. So realistically there are two possibilities: (1) you enhance the install.py script to add the feature and send me the improved script for review, or (2) you go through the script and figure out which packages to install in the virtual environment on your system, which is not difficult.

Thanks for your kind understanding.

Changed in rapid:
status: New → Triaged
importance: Undecided → Wishlist
Revision history for this message
Matthias Homann (palto) wrote :

Hi Damon,
Thanks for your quick feedback & also thanks for this great software!
I will try to enhance the scrip, not sure if I can get it work with venv.

Revision history for this message
Damon Lynch (dlynch3) wrote :

Great, thanks. I updated the install.py script today to fix a critical bug. Be sure to make any enhancements to the latest version.

Changed in rapid:
assignee: nobody → Matthias Homann (palto)
Revision history for this message
Matthias Homann (palto) wrote :

Meanwhile I managed to install RPD in venv by forcing the script to use "unknown distro", removing the "--user" option from pip command (otherwise the libs are not installed in venv) and some manual installs. Now I need to see how to correctly modify the script to make it all work automatic (e.g. add venv install option, modified starter script).

One issue I found was that the lib "gi" (gobject-introspection) doesn't work in venv. The solution was to use vext https://pypi.org/project/vext/.

apt also doesn't work in venv, but this is anyway Ubuntu/Debian specific and I could install all the required libs with pip.

Revision history for this message
Damon Lynch (dlynch3) wrote :

Thanks for investigating. Some suggestions:

Any enhancements to the script must work seamlessly on all currently supported platforms, i.e. Debian family, openSUSE, Fedora. The goal is for the script to seamlessly install required system packages and then install packages into a venv.

The install.py script is more sophisticated than it might first appear. It's truly vital to understand how every part of it works before trying to make changes to it. For instance the function restart_script() is core to how the script functions, and is called under a few different circumstances.

Revision history for this message
Matthias Homann (palto) wrote :

You can find a first version of the modified install script (based on version 0.2.0) for usage of Python virtual environments at https://github.com/palto42/rapid-photo-downloader

I only tested it on Ubuntu 16.04 yet, any feedback is welcome.

Revision history for this message
Damon Lynch (dlynch3) wrote :

I apologize but I will not be able to provide any feedback until my PhD dissertation is submitted, which will be later this year.

Revision history for this message
Damon Lynch (dlynch3) wrote :

I did take a very brief look at the code, without running it.

First, it's important to test it on the supported Linux distributions. For example, the --user command to pip is essential on systems that do not patch pip like Debian does (which pretty much means any non-Debian distro). Without the --user flag, the script is broken on non-Debian family distros.

Second, a much more minor issue is that there are some spelling mistakes I noticed.

Meanwhile I have updated the install.py script to version 0.2.1

Revision history for this message
Matthias Homann (palto) wrote :

Thanks for your feedback, I'm still working on the script and just trying it on my upgraded Ubuntu 18.04.
In active virtual environment pip must be used without "--user" to install the files to the defined path of the virtual environment. Otherwise they will be put into the default user location which is used without active virtual environment.

Can I download the new script version somewhere? The download from the main web page still provides 0.2.0 version

Revision history for this message
Damon Lynch (dlynch3) wrote :

The latest version of the code is always available at https://bazaar.launchpad.net/~dlynch3/rapid/zeromq_pyqt/files

You can download the file using a web browser if you prefer not to use bzr. Having said that, in my experience bzr's merge tool is pretty good.

Changed in rapid:
status: Triaged → In Progress
Revision history for this message
Matthias Homann (palto) wrote :

I updated my script based on latest 0.2.1 version and did some minor fixes and improvements.
https://github.com/palto42/rapid-photo-downloader

Revision history for this message
Damon Lynch (dlynch3) wrote :

Thanks for updating your enhancement. On which platforms has it been tested?

Revision history for this message
Matthias Homann (palto) wrote :

Latest version was tested on Ubuntu 18.04, I have no other platforms available.

Revision history for this message
Damon Lynch (dlynch3) wrote :

As I indicated above, without running it it looks very much like the script will break on non-Debian systems. Developing the script to run on all supported platforms (now including CentOS 7.5) takes time, but it needs to be done. If you are unable to do this by using VirtualBox or another hypervisor, then I'll need to do it, which means incorporating the enhancement into a release will not occur until after several months.

Revision history for this message
Matthias Homann (palto) wrote :

No sure why you think that my changes will break things. Actually it should not change anythin as long as non of the two new arguments "--virtual-env" and "--no-dist" are added.
If the script is run without the "--virtual-env" option, then the pip commands are still constructed with the "--user" flag as in your original scripts.
The other changes are to allow the script if it runs in virtual environment (your original script terminates with message "..installer does not support a custom python installation..") and to add required packages vext.gi if needed.

Since I can't do testing on more platforms, this needs to wait some month, but I hope that my script and the comments here may help anyone else who wants to try running Rapid Photo Downloader in Python venv.

Revision history for this message
Damon Lynch (dlynch3) wrote :

Taking a look at it now. Why do you attempt to install vext.gi and not just plain vext? The former does not work. I have my doubts the author cares about it enough to fix the basic installation bug.

Revision history for this message
Matthias Homann (palto) wrote :

vext and vext.gi was required because gi was previously not working in a virtual environment. I have just tried again and could successfully install pygobjects in venv (as documented here https://pygobject.readthedocs.io/en/latest/getting_started.html). I need to modify the script accordingly and try it out with pygobjects instead of vext.gi.

Revision history for this message
Matthias Homann (palto) wrote :

I updated my modified install script to original version 0.2.5 and replaced vext.gi with pygobject. On Ubuntu 18.04 (64) it works fine (sorry, no time to test other OS).

Link to diff page on Github: https://github.com/palto42/rapid-photo-downloader/commit/1ec020815d3684bb399a4c2e067e10e1fd4ac9b0#diff-9de40ef13f0271a46b14b8c27bbf004d

Revision history for this message
Damon Lynch (dlynch3) wrote :

A hint for any future code you develop for any project: check your code works in a clean environment.

This page lists many build requirements:

https://pygobject.readthedocs.io/en/latest/devguide/dev_environ.html#devenv

You cannot assume your users will have already installed them.

Damon Lynch (dlynch3)
Changed in rapid:
milestone: none → 0.9.13
Revision history for this message
Damon Lynch (dlynch3) wrote :

I committed the changes to the install.py script. You can test them if you like. You can also the changes I made to the code.

The --system-site-packages option is not supported.

Changed in rapid:
status: In Progress → Fix Committed
Damon Lynch (dlynch3)
Changed in rapid:
status: Fix Committed → Fix Released
Revision history for this message
Matthias Homann (palto) wrote :

Thanks for fixing this issue and all the other improvements!

Regarding the support of --system-site-packages this seems to require just a small fix in the section where the script queries the site packages. If Python runs in a venv, then the site module works slightly different. If you are interested, please have a look at this patch: https://github.com/palto42/rapid-photo-downloader/commit/5b075ece7cd1af02b22839bcb24bbf53fdb6c58f#diff-9de40ef13f0271a46b14b8c27bbf004d

Another change which might be interesting for some users is the capability to install RPD without sudo. This is possible if all mandatory files can be installed by pip and the script treats the distribution as "unknown". Based on your script 0.2.6 I added such option in another patch: https://github.com/palto42/rapid-photo-downloader/commit/56cb6602c4ae9e9b329b96098b46152cf19a2366#diff-9de40ef13f0271a46b14b8c27bbf004d

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.