Ubuntu 20.04: virtualenv: pep517 requires toml, but it isn't bundled in venvs

Status changed to 'Confirmed' because the bug affects multiple users.

Please backport the fixes, this affects how we install our toolset at my company.

It would be great to get this fixed as more people are upgrading servers and CI environments to Ubuntu 20.04 and starting to find weird edge cases related to this bug (besides not being able to install many popular packages from PPAs that haven't been updated like Ansible's).

Thank you for taking the time to report this bug and helping to make Ubuntu better. However, I am closing it because the bug has been fixed in the latest development version of Ubuntu.

If you need a fix for the bug in previous versions of Ubuntu, please perform as much as possible of the SRU Procedure [1] to bring the need to a developer's attention.

[1]: https://wiki.ubuntu.com/StableReleaseUpdates#Procedure

For the record: This is still broken in Ubuntu 20.04 (Focal Fossa). It breaks Ansible being able to support 20.04 in their official PPA. Tracking ticket on the Ansible side: https://github.com/ansible/ansible/issues/69203

It would be awesome if someone with the time and energy would do the work suggested by Kai to get this fix into a Focal Fossa update as well, so we can get these packages working there. Otherwise, people on Focal Fossa are bound to run the repo-provided Ansible packages (i.e. some older version).

20.04 is an LTS release, so it's the next step in upgrading from 18.04 and all new server builds.
Sadly, this fix did not make it into 20.04.1, according to the Release Notes.

As mentioned by Kai, does "latest development version of Ubuntu." mean 20.10+ releases?

Status changed to 'Confirmed' because the bug affects multiple users.

Almost half a year now and the LTS version is still not fixed?

This is sad, because it keeps Ansible from bringing out their PPA for focal.
I guess most users switching to pip installations will never automatically apply released security patches for it.

Still waiting for this bug to be resolve in Ubuntu 20.04.1 Focal Fossa.

> You will notice there are a plethora of extra packages in the virtual environment that should not normally be there, resulting in a dirty virtual environment.

Looking at if this can be changed for Ubuntu 20.04. I'm afraid it may be complex. I think we may need to patch pip to be able to apply these changes to virtualenv.

> To make matters worse, the latest release of pip bundles incompatible versions of libraries. The net result is that `pip install pep517` will show that it is already installed, and but `import pep517` will result in an ImportError.

I can't reproduce that:

1. pep517 imports successfully.
2. pip can upgrade it, without error.

root@actual-krill:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.1 LTS
Release: 20.04
Codename: focal
root@actual-krill:~# python3 -m virtualenv foo && source foo/bin/activate
created virtual environment CPython3.8.5.final.0-64 in 590ms
  creator CPython3Posix(dest=/root/foo, clear=False, global=False)
  seeder FromAppData(download=False, CacheControl=latest, appdirs=latest, certifi=latest, chardet=latest, colorama=latest, contextlib2=latest, distlib=latest, distro=latest, html5lib=latest, idna=latest, ipaddr=latest, lockfile=latest, msgpack=latest, packaging=latest, pep517=latest, pip=latest, pkg_resources=latest, progress=latest, pyparsing=latest, pytoml=latest, requests=latest, retrying=latest, setuptools=latest, six=latest, urllib3=latest, webencodings=latest, wheel=latest, via=copy, app_data_dir=/root/.local/share/virtualenv/seed-app-data/v1.0.1.debian)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
(foo) root@actual-krill:~# pip list
Package Version
------------- ----------
appdirs 1.4.3
CacheControl 0.12.6
certifi 2019.11.28
chardet 3.0.4
colorama 0.4.3
contextlib2 0.6.0
distlib 0.3.0
distro 1.4.0
html5lib 1.0.1
idna 2.8
ipaddr 2.2.0
lockfile 0.12.2
msgpack 0.6.2
packaging 20.3
pep517 0.8.2
pip 20.0.2
pkg-resources 0.0.0
progress 1.5
pyparsing 2.4.6
pytoml 0.1.21
requests 2.22.0
retrying 1.3.3
setuptools 44.0.0
six 1.14.0
urllib3 1.25.8
webencodings 0.5.1
wheel 0.34.2
(foo) root@actual-krill:~# python
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pep517
>>>
(foo) root@actual-krill:~# pip install pep517
Requirement already satisfied: pep517 in ./foo/lib/python3.8/site-packages (0.8.2)
(foo) root@actual-krill:~# pip install -U pep517
Collecting pep517
  Downloading pep517-0.9.1-py2.py3-none-any.whl (18 kB)
Collecting toml
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Installing collected packages: toml, pep517
  Attempting uninstall: pep517
    Found existing installation: pep517 0.8.2
    Uninstalling pep517-0.8.2:
      Successfully uninstalled pep517-0.8.2
Successfully installed pep517-0.9.1 t...

Stefano,

For the pep517 issue, try importing pep517.check or pep517.build:

(foo) root@3edb3d8dca37:/# python3 -m pep517.build
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/foo/lib/python3.8/site-packages/pep517/build.py", line 6, in <module>
    import toml
ModuleNotFoundError: No module named 'toml'

Upgrading it does indeed work fine, the main issue is that it's reported as being installed already but the installed version doesn't work. This is because it's bundled with pip (and gets injected into the virtualenv even though it shouldn't (the main reason for this bug report)), and this version of pip bundled conflicting versions of libraries. They bundled a pep517 version that wants toml, but bundled pytoml alongside it instead of toml.

Sorry, posted that from the wrong account, that comment was by me (the original bug reporter).

OK, so sounds like we're missing a toml wheel in python-pip-whl.

What is the impact of this? It seems to happily install PEP 517 packages as-is.

The impact of the pep517 issue isn't too big of an issue for us- we can work around it (installing toml).

The "impact" is that we have a test scenario which runs `pip install pep517` (reports already installed) and then tries to do `python -m pep517.build --binary --out-dir dist .` (which fails due to the toml issue).

As mentioned, we can work around this by installing toml. I'm more worried about the broader issue in the report, that virtualenvs are dirty in the first place, this limits their utility :(

I think we can get the missing toml resolved.

Agreed that that's really ugly that the virtualenvs include all of the pip deps. That was an issue in the early virtualenv 20 days, that took a while to get the necessary upstream support for cleaner de-vendoring. But messyness on its own can't justify broader changes to a stable release. The process for these changes is a Stable Release Update (SRU), see the policy linked below.
We need to avoid breaking things for users of stable releases, once they are released. The change has to be proven to be worth the risk. I'll need some more concrete examples of the impact, to push through an SRU. And to show that the SRU resolves the issues.

https://wiki.ubuntu.com/StableReleaseUpdates

Retargetting this bug at just the missing toml. Use LP: #1904945 for the dirty virtualenvs.

@Rick Elrod: It would be useful if you could describe something that's concretely broken by toml being missing? I haven't been able to trigger any issue, installing packages with pip.

It's always a bit tricky reviewing SRUs without a visible user impact. But I would say that since the vendoring of pytoml instead of toml *is* a bug, let's get it included in the SRU for fixing the dirty venvs. I can't think of any use-cases this could break. Let's see how it goes.

Hello Rick, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted python-pip (20.0.2-5ubuntu1.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

python3.8/3.8.5-1~20.04 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#python-pip

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Re-tried the armhf autopkgtest and it passed.

> What is the impact of this? It seems to happily install PEP 517 packages as-is.

@Stefano sorry for the delay here. Installing PEP517 packages is fine without the patch, but building them (building a wheel) is where the issue comes in.

If you need a concrete example, download the .tar.gz of any project that is normally capable of producing wheels, such as 'sampleproject' from pypi (https://pypi.org/project/sampleproject/#files).
Activate a fresh virtualenv, untar it the .tar.gz, cd into the project, and run:

  python -m pep517.build --binary --out-dir dist .

Before the patch mentioned above, it will fail with: ModuleNotFoundError: No module named 'toml'

The .deb's from the SRU above get us a step closer but seem to still have an issue/traceback (see attachment which includes the commands to reproduce). It looks to be due to the vendored pytoml being listed in `lib/python3.8/site-packages/pip/_vendor/__init__.py` in the virtualenv.

Whereas doing similar steps in a ubuntu:20.10 container (and installing pep517 explicitly in the virtualenv) produces a successful result.

@Rick: Thanks for the response.

A few things:
1. I don't think you're using the current wheels.
   Try with --clear-app-data (added to the SRU instructions in the description).
2. What this still doesn't tell me is how *pip* is impacted by this breakage.
   LP: #1904945 means that we won't expose pep517 to the virtualenv outside of pip, any more.
   So, pep517 being broken outside of pip wouldn't be an issue.

We have -proposed enabled in one of our focal OpenStack deployments and this update appears to have regressed the setup of virtualenv's for reactive charms (which ship with a wheelhouse of python module dependencies including pip) - we see this error as new charms attempt to setup a venv using pip:

ImportError: cannot import name 'pytoml' from 'pip._vendor'

Downgrading the python3-pip + hard versioned depends resolves this issue for us.

I've marked the verification tags as 'failed' as releasing this update to -updates will have a major impact on charm deployments on focal.

Full error in attached logfile

Thank you for flagging this regression. @Stefano, can you take a look at these issues? I think we might need to drop this update from -proposed.

After Rick's failed verification comment I had been wondering if we'd run into trouble with stale seed-app-data, and yes we have 2 different regressions:

1. The new python-pip-whl breaks new virtualenv creation if you have existing seed-app-data.
   This was James's issue. Traceback ends with:
   File "lib/python3.8/site-packages/pip/_internal/pyproject.py", line 8, in <module>
     from pip._vendor import pytoml, six
   ImportError: cannot import name 'pytoml' from 'pip._vendor' (lib/python3.8/site-packages/pip/_vendor/__init__.py)

2. The new virtualenv breaks new virtualenv creation if you have existing seed-app-data.
   File "lib/python3.8/site-packages/pip/_internal/exceptions.py", line 10, in <module>
     from pip._vendor.six import iteritems
   ModuleNotFoundError: No module named 'pip._vendor.six'

Attached is my script to reproduce.

This is all pretty horrible. I'm tempted to patch out seed-app-data entirely for our use-case. But
I see the whole seed-app-data mechanism was re-done in 20.0.24, hopefully it will be more friendly to patching...

My solution is:

1. Bump the seed-app-data version in python3-virtualenv, to cache-bust.
2. Add a Breaks: python3-virtualenv (<< 20.0.17-1ubuntu0.3) (the version in 1) to the new python-pip-whl.

Hello Rick, or anyone else affected,

Accepted python-pip into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-pip/20.0.2-5ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Rick, or anyone else affected,

Accepted python-virtualenv into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-virtualenv/20.0.17-1ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Nobody has reported any more issues, and the tests we've got pass for me, so marking verification-done

This bug was fixed in the package python-pip - 20.0.2-5ubuntu1.3

---------------
python-pip (20.0.2-5ubuntu1.3) focal; urgency=medium

  * Add Breaks: python3-virtualenv (<< 20.0.17-1ubuntu0.3) to python-pip-whl,
    which bumps the seed-app-data version. Our LP: #1880749 change broke
    existing seed-app-data caches.

python-pip (20.0.2-5ubuntu1.2) focal; urgency=medium

  * Switch from vendoring pytoml to toml, following pep517's dependency.
    (LP: #1880749)
  * Use sys.base_prefix instead of sys.prefix in debundle.patch. Back-ported
    from 20.1-1 to allow virtualenvs to avoid needing to install pip's
    dependencies in new venvs. (LP: #1904945)

 -- Stefano Rivera <email address hidden> Fri, 26 Feb 2021 18:38:56 -0800

The verification of the Stable Release Update for python-pip has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-virtualenv - 20.0.17-1ubuntu0.3

---------------
python-virtualenv (20.0.17-1ubuntu0.3) focal; urgency=medium

  * Bump the seed-app-data version, because we changed the bundled wheels in
    LP: #1880749.

 -- Stefano Rivera <email address hidden> Fri, 26 Feb 2021 18:09:22 -0800

I'm not able to create a python 3.7 virtualenv on ubuntu20 with python3-virtualenv 20.0.17-1ubuntu0.3 installed.

Downgrading to 20.0.17-1 did the job. Probably this issue is introduced in this issue?

virtualenv -p /home/laurens/projects/tcit/cpython/out/bin/python3 venv
created virtual environment CPython3.7.10.final.0-64 in 63ms
  creator CPython3Posix(dest=/xxx/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/laurens/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
Traceback (most recent call last):
  File "/xxx/venv/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
  File "/xxx/venv/lib/python3.7/site-packages/pip/_internal/cli/main.py", line 10, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/xxx/venv/lib/python3.7/site-packages/pip/_internal/cli/autocompletion.py", line 9, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/xxx/venv/lib/python3.7/site-packages/pip/_internal/cli/main_parser.py", line 7, in <module>
    from pip._internal.cli import cmdoptions
  File "/xxx/venv/lib/python3.7/site-packages/pip/_internal/cli/cmdoptions.py", line 24, in <module>
    from pip._internal.exceptions import CommandError
  File "/xxx/venv/lib/python3.7/site-packages/pip/_internal/exceptions.py", line 10, in <module>
    from pip._vendor.six import iteritems
ModuleNotFoundError: No module named 'pip._vendor.six'

I've built python 3.7.10 from source because I'm running on a platform that uses 3.7 instead of the built-in 3.8.