Reconsider how we create requirements.txt from a given venv #962
Description
Today I ran into the following problem: a venv with a bunch of packages was installed on a x64 macOS machine and on a arm64 macOS machine, and then make requirements was used on both to generate the requirements.txt file.
One of the installed packages is SQLAlchemy. Querying PyPI for all available SQLA wheels and their host ABIs and platforms returns the following:
> curl -s https://pypi.org/pypi/sqlalchemy/json | jq -r '.releases["2.0.41"].[].filename'
sqlalchemy-2.0.41-cp313-cp313-macosx_10_13_x86_64.whl
sqlalchemy-2.0.41-cp313-cp313-macosx_11_0_arm64.whl
sqlalchemy-2.0.41-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
sqlalchemy-2.0.41-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
sqlalchemy-2.0.41-cp313-cp313-musllinux_1_2_aarch64.whl
sqlalchemy-2.0.41-cp313-cp313-musllinux_1_2_x86_64.whl
sqlalchemy-2.0.41-cp313-cp313-win32.whl
sqlalchemy-2.0.41-cp313-cp313-win_amd64.whl
...
sqlalchemy-2.0.41.tar.gz
The problem is that SQLA installed on an x64 platform installs the greenlet dependency (which seems to be used only for async SQLA?) and on an arm64 platform it does not!
Therefore, the generated requirements.txt files are different on x64 platform vs. ARM platform. Therefore, building a requirements.txt file on an arm64 macOS machine and then using it to install into a x64 Linux container fails.
I think the solution would be to build a requirements.txt file for a dedicated target (see also pip install --platform).1 But then which ones? Should this template provide for building multiple platform specific requirements.txt files?
Addendum: This is interesting: on x64 macOS
> uname -a
Darwin pooh 22.6.0 Darwin Kernel Version 22.6.0: Thu Apr 24 20:25:14 PDT 2025; root:xnu-8796.141.3.712.2~1/RELEASE_X86_64 x86_64 i386 Darwin
> pip install sqlalchemy
...
Installing collected packages: typing-extensions, greenlet, sqlalchemy
Successfully installed greenlet-3.2.3 sqlalchemy-2.0.41 typing-extensions-4.14.0
and then on arm64 macOS
> Darwin user.fritz.box 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:48:46 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T8103 arm64
> pip install sqlalchemy
...
Installing collected packages: typing-extensions, sqlalchemy
Successfully installed sqlalchemy-2.0.41 typing-extensions-4.14.0
No greenlet package 🤔
Related issue: sqlalchemy/sqlalchemy#7714
Footnotes
-
This feels very similar to the problem of cross-compiling code for a target architecture different than the host it’s being compiled on. ↩