Documentation for a newer release is available. View Latest

Python

Setuptools 59

Fedora Linux 36 provides python-setuptools 59. This includes the removal of the use_2to3 setup command. For more details, see the upstream changelog.

New installation scheme of Python packages

In Fedora Linux 36, Python changes the way installation paths of Python packages are handled. These changes affect the main Python in Fedora 36, Python 3.10, as well as any newer Python version included. Most Fedora Linux users should not be affected by the change but there are situations where there might be slight differences.

When Python packages are installed by sudo pip, sudo python setup.py install or similar methods, Python packages are installed to /usr/local/lib(64)/python3.10/site-packages/. This has already been happening since Fedora Linux 27. However the way this is achieved has been significantly re-implemented in Fedora Linux 36 and that has created several minor differences.

The sysconfig Python module from the standard library defines several installation schemes. By default, the installation scheme used on Fedora 36 when installing Python packages using root privileges (for example via sudo) is {prefix}/lib(64)/python3.10/site-packages/, where {prefix} is defined as /usr/local by default. When building RPM packages, {prefix} is defined as /usr by default. When Python itself runs from a Python virtual environment, {prefix} is defined as the path for the virtual environment.

Previously, the /local/ part was only artificially added when installing packages via distutils, now it is also defined in sysconfig. This was changed to be more consistent with what other Python distributors are doing, so that the scheme is more likely to be accepted in upstream Python and to work well with upstream tools like setuptools or pip, which we cannot modify when they are installed or upgraded using pip directly from the Python Package Index.

Here are the differences that might be observed with the new approach:

sysconfig.get_path(key) returns paths with /local/

Previously, on Fedora Linux 35 outside of a virtual environment:

>>> import sysconfig
>>> for key in sysconfig.get_path_names():
...     print(f'{key} = {sysconfig.get_path(key)}')
...
stdlib = /usr/lib64/python3.10
platstdlib = /usr/lib64/python3.10
purelib = /usr/lib/python3.10/site-packages
platlib = /usr/lib64/python3.10/site-packages
include = /usr/include/python3.10
scripts = /usr/bin
data = /usr

Now, on Fedora Linux 36 (except during RPM build):

>>> import sysconfig
>>> for key in sysconfig.get_path_names():
...     print(f'{key} = {sysconfig.get_path(key)}')
...
stdlib = /usr/lib64/python3.10
platstdlib = /usr/lib64/python3.10
purelib = /usr/local/lib/python3.10/site-packages
platlib = /usr/local/lib64/python3.10/site-packages
include = /usr/include/python3.10
scripts = /usr/local/bin
data = /usr/local

The values now reflect the reality of where packages are actually going to be installed with pip, setuptools, distutils, etc. However, if your Python code uses the values to determine where to load Python packages from, it won’t see dnf-installed packages, which are installed in /usr/lib(64)/python3.10/site-packages/. Generally, sysconfig.get_path(key) gives results that determine where the Python packages should be installed to. To fix affected code, avoid assumptions that "where to install packages to" is the same as "where to load Python modules from".

Example fixes from affected open source projects:

If you need to restore the previous behavior of sysconfig.get_path(key), you may explicitly set the {base} and {platbase} variables to /usr:

>>> for key in sysconfig.get_path_names():
...     print(f'{key} = {sysconfig.get_path(key, vars={"base": "/usr", "platbase": "/usr"})}')
...
stdlib = /usr/lib64/python3.10
platstdlib = /usr/lib64/python3.10
purelib = /usr/lib/python3.10/site-packages
platlib = /usr/lib64/python3.10/site-packages
include = /usr/include/python3.10
scripts = /usr/bin
data = /usr

RPM build–related caveats

When Python runs during RPM build, it sets the defaults for {base} and {platbase} variables to /usr. This behavior is triggered when the $RPM_BUILD_ROOT environment variable is set. That has several caveats:

Executing Python in Python’s subprocess

If the Python code that runs in RPM build (for example in %check) executes another Python instance via a subprocess, it is relatively easy to inadvertently unset all environment variables. When this happens, the inner Python will not know it runs within RPM build and will return paths with the /usr/local/ prefix.

In the most trivial example, the surrounding environment variables are implicitly passed to subprocess and everything works as expected:

>>> import os, subprocess, sys
>>> 'RPM_BUILD_ROOT' in os.environ
True
>>> command = [sys.executable, '-c', 'import sysconfig; print(sysconfig.get_path("purelib"))']
>>> subprocess.check_output(command)
b'/usr/lib/python3.10/site-packages\n'

But when a custom environment is passed, it breaks the detection, because $RPM_BUILD_ROOT is no longer set:

>>> subprocess.check_output(command, env={'FOO': 'bar'})
b'/usr/local/lib/python3.10/site-packages\n'

A solution is to always make a copy of the surrounding environment, then editing it and passing it to the subprocess. That is a generally valid Python advice.

>>> env = os.environ | {'FOO': 'bar'}
>>> subprocess.check_output(command, env=env)
b'/usr/lib/python3.10/site-packages\n'

Example fixes from affected open source projects:

%(…​) RPM macros

When RPM macros in the form of %(command …​) are expanded, $RPM_BUILD_ROOT is not yet set. Hence, Python does not know it is invoked from RPM build and the paths returned by sysconfig.get_path(…​) contain /local/. To fix this, set $RPM_BUILD_ROOT in the macro definition (to any value, even empty). For example a macro defined like this:

%global python3_scripts_dir %(python3 "import sysconfig; print(sysconfig.get_path('scripts'))")

Needs to be changed to this:

%global python3_scripts_dir %(RPM_BUILD_ROOT= python3 "import sysconfig; print(sysconfig.get_path('scripts'))")

The affected RPM macros supplied by Fedora’s python-rpm-macros packages have all been changed accordingly.

Paths for bootstrapping Python virtual environments

If your Python code uses installation schemes to determine paths to be used in created virtual environments, and the Python interpreter executing that code does not run from a Python virtual environment itself, the paths will not match.

To bootstrap Python virtual environments, the code should use the venv installation scheme (but only if it exists).

>>> scheme = 'venv' if 'venv' in sysconfig.get_scheme_names() else None
>>> sysconfig.get_path('purelib', scheme=scheme, vars={'base': '<venv>'})
'<venv>/lib/python3.10/site-packages'
>>> sysconfig.get_path('scripts', scheme=scheme, vars={'base': '<venv>'})
'<venv>/bin'

The venv scheme is currently Fedora specific, but other Python distributors (such as Ubuntu Deadsnakes) may define it as well. The venv scheme is generally available starting with Python 3.11. Due to checking if the venv install scheme exists the code is functional on other operating systems as well, as it falls back to a backwards compatible behavior.

Example fixes from affected open source projects:

Django 4.0

Fedora Linux 36 provides the latest version of Django 4.0. Few highlights of this version are:

  • The new RedisCache backend provides built-in support for caching with Redis.

  • To ease customization of Forms, Formsets, and ErrorList they are now rendered using the template engine.

  • The Python standard library’s zoneinfo is now the default timezone implementation in Django.

For more details, see the upstream release notes.

Want to help? Learn how to contribute to Fedora Docs ›