feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master) - #38915
feat: consolidated pip-compile -> uv migration (all 5 stacked PRs, rebased onto master)#38915irfanuddinahmad wants to merge 27 commits into
Conversation
openedx-webhooks
added
open-source-contribution
|
Thanks for the pull request, @irfanuddinahmad! This repository is currently maintained by Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review. 🔘 Get product approvalIf you haven't already, check this list to see if your contribution needs to go through the product review process.
🔘 Provide contextTo help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:
🔘 Get a green buildIf one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green. DetailsWhere can I find more information?If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources: When can I expect my changes to be merged?Our goal is to get community contributions seen and reviewed as efficiently as possible. However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:
💡 As a result it may take up to several weeks or months to complete a review and merge your PR. |
irfanuddinahmad
changed the title
|
Re: the Good catch, and worth noting: the org already ran a dedicated SHA-pinning sweep for this exact reason (openedx/.github#165, prompted by the tj-actions/changed-files supply-chain incident), but openedx-platform wasn't part of that ~121-repo effort. That said, I don't think pinning just |
…ration 1/5) Populates [project.dependencies] (from kernel.in + bundled.in), adds [project.optional-dependencies] for the legacy openstack storage backend, adds PEP 735 [dependency-groups] (coverage/testing/doc/assets/development/ semgrep/ci/dev, mirroring the current .in file composition), and [tool.edx_lint].uv_constraints + generated [tool.uv].constraint-dependencies for the ~20 repo-specific version pins, with a committed uv.lock. This is purely additive: the Makefile, tox.ini, and CI workflows are untouched and continue to use pip-compile/requirements/*.txt as the source of truth. Part of the pip-compile -> uv migration tracked in openedx/public-engineering#543 (1 of 5 PRs). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ions Found via real CI runs: a fresh uv resolution picked social-auth-core 5.0.2 (previously locked at 4.9.1 via pip-compile), which changes the OAuth pipeline's post-login redirect behavior and breaks common/djangoapps/third_party_auth's integration test suite (AzureAD, Google, LinkedIn, Twitter full-pipeline specs all failed the same assertion in tests/specs/base.py's assert_logged_in_cookie_redirect). This migration is meant to be a tooling swap, not a dependency upgrade, so pin back to the 4.x line rather than bundle an investigation into social-auth-core 5.x's behavior change into this PR. Mirrors the existing social-auth-app-django<=5.4.1 constraint, pinned for a related, already-deferred migration in this same dependency family. Follow-up tracked at #38841. Verified: all 46 previously-failing third_party_auth tests pass with social-auth-core==4.9.1 restored via this constraint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Rewrites the Makefile's requirements targets, tox.ini, and ~13 CI workflows to use uv instead of pip-compile/pip-sync for the main app. Deletes requirements/edx/*.in and *.txt (superseded by pyproject.toml + uv.lock, added in PR 1 / #38835). requirements/constraints.txt, common_constraints.txt, and pip-tools.{in,txt} are intentionally kept for now: requirements/edx-sandbox and scripts/* still pip-compile against them and aren't migrated until PR 3/4. requirements/edx/{base,assets,development}.txt are regenerated as `uv export` compatibility artifacts (via the Makefile's compile-requirements target) since external tooling -- notably tutor's Dockerfile -- installs from those exact paths with plain pip, not uv. check_python_dependencies.yml is disabled (workflow_dispatch only, job gated with if: false) since find_python_dependencies can't scan pyproject.toml yet; tracked at openedx/repo-tools#725. User-confirmed before committing since this removes a CI safety net. Part of openedx/public-engineering#543 (2 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found via real CI runs after opening the PR (unit tests, Quality Others, and the ReadTheDocs build all failed): 1. Makefile's test-requirements used `uv sync --only-group testing`, which EXCLUDES [project.dependencies] entirely (confirmed: `--only-group` replaces the dependency set rather than adding to it, unlike `--group`). This meant Django, XBlock, and the rest of the actual application were never installed for test runs -- ModuleNotFoundError: No module named 'xblock' on every unit test shard. Fixed to `--no-default-groups --group testing`, matching what tox-uv itself generates for the equivalent tox environment. 2. .readthedocs.yaml had the same bug in its doc-requirements.txt export (`--only-group doc`). Since [project.dependencies] were excluded from that export, the subsequent `pip install -e .` resolved the whole dependency tree completely unconstrained by uv.lock's [tool.uv].constraint-dependencies -- picking setuptools==82.0.1 (violates setuptools<82) and Django==6.0.6 (violates Django<6.0). The setuptools violation broke fs/pyfilesystem2's pkg_resources import, crashing the Sphinx build via Django app loading. Fixed to `--group doc` plus `--no-deps` on the `pip install -e .` step, so dependencies only ever come from the properly-constrained export. 3. scripts/xsslint_config.py's SKIP_DIRS didn't exclude .venv. Under the old pip-compile system, dependencies installed into the system Python outside the repo checkout, so this never mattered. Now that `uv sync` creates .venv/ inside the checkout, xsslint's directory walk (which defaults to scanning the whole cwd) swept up thousands of vendored third-party files, inflating violations from 64 (the accepted baseline) to 316. Verified all three: real pytest run (59 passed) plus full Django `manage.py check` for both LMS and CMS against the corrected test-requirements; a local simulation of the RTD build job sequence confirms Django==5.2.15/setuptools==81.0.0 (both constraint-compliant) and a successful Sphinx build. Audited every other `--only-group` usage introduced in this migration (assets.txt compat export, semgrep.yml) -- both are intentionally project-dependency-free, matching the original files' documented behavior, and their CI checks already passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
requirements/edx/{base,development}.txt were regenerated from the
uv.lock that predated PR 1's social-auth-core<5.0.0 constraint, so
they still referenced social-auth-core==5.0.2 -- caught by
check-consistent-dependencies.yml's re-run of `make compile-requirements`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives requirements/edx-sandbox/ its own standalone pyproject.toml + uv.lock, independent of the main app's dependency graph (codejail intentionally runs untrusted code in a separate, isolated venv). [tool.edx_lint].uv_constraints holds only the subset of the root constraints relevant to this environment's deps (numpy, lxml, setuptools) -- uv/edx-lint have no cross-project constraint chaining equivalent to pip-compile's "-c ../constraints.txt", so root and sandbox constraints are now independently maintained (documented in requirements/edx-sandbox/README.rst). base.txt is regenerated as a `uv export` compatibility artifact (the README documents it as a supported, if unstable, direct pip-install target). releases/*.txt are untouched -- they're frozen historical snapshots, not part of any active compile loop; README now documents cutting future ones via `uv export` instead of pip-compile. Part of openedx/public-engineering#543 (3 of 5). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…on 4/5)
Gives scripts/xblock, scripts/user_retirement, and scripts/structures_pruning
each their own standalone pyproject.toml + uv.lock, mirroring the
codejail sandbox pattern from PR 3. structures_pruning's local
[tool.edx_lint].uv_constraints keeps the pymongo<4.4.1 pin it inherited
via the old "-c ../../../requirements/constraints.txt" chain.
These scripts are documented (in their own READMEs) to support git
sparse-checkout usage -- cloning only e.g. scripts/user_retirement/
without the rest of edx-platform. A self-contained pyproject.toml is
actually an improvement here over the old relative "-c
../../../requirements/constraints.txt" reference, which wouldn't even
resolve in a sparse checkout that excludes the root requirements/ dir.
Compatibility .txt exports are kept at their previously-documented
paths (e.g. scripts/user_retirement/requirements/{base,testing}.txt)
since each script's own README explicitly instructs `pip install -r`
against those exact paths.
Also fixes check-consistent-dependencies.yml's path filter and the two
PR-creating workflows' add-paths, neither of which would have picked
up changes to the new scripts/*/pyproject.toml or uv.lock files.
Part of openedx/public-engineering#543 (4 of 5).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Deletes requirements/constraints.txt, common_constraints.txt, and
pip-tools.{in,txt} -- these were kept alive through PR 2-4 because
requirements/edx-sandbox and scripts/* still pip-compiled against
them, but PR 4 was the last consumer, so they're now fully unused.
Removes the correspondingly-vestigial Makefile machinery: the
pre-requirements target, the COMMON_CONSTRAINTS_TXT curl-fetch-and-sed
target, and the CUSTOM_COMPILE_COMMAND/COMPILE_OPTS variables that only
existed to feed pip-compile invocations which no longer exist anywhere
in this repo.
Finalizes requirements/README.rst for the fully-migrated state and
fixes a couple of remaining stale references (constraints.txt ->
[tool.edx_lint].uv_constraints).
This is the last of 5 PRs migrating openedx-platform from pip-compile
to uv + PEP 621/735 pyproject.toml, tracked in
openedx/public-engineering#543. Two follow-up
items remain outside this repo's control:
- openedx/repo-tools#725: find_python_dependencies needs pyproject.toml
support before check_python_dependencies.yml can be re-enabled.
- Tutor's Dockerfile installs from requirements/edx/{base,assets,development}.txt
with plain pip; those are kept as `uv export` compatibility artifacts
(see PR 2 / #38836) rather than deleted, so no action is required there,
but tutor maintainers should be aware these are now generated files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI's "Compile requirements" check re-runs `make compile-requirements` and fails if it produces any diff, to catch exactly this kind of inconsistency. The compat-export files in this PR were originally generated with uv 0.11.26; a newer uv (0.11.30, matching what astral-sh/setup-uv installs in CI) resolves grpcio/grpcio-status with an explicit `; platform_python_implementation != 'PyPy'` marker that 0.11.26 omitted. Regenerated with uv 0.11.30 to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The uv migration for scripts/user_retirement dropped the lxml pin that requirements/constraints.txt previously carried, letting the lockfile resolve to lxml 6.1.1. The pin exists to avoid a libxml2 version mismatch at runtime (#36695), and this script transitively depends on lxml via simple-salesforce -> zeep, so the same pin applies here as it does at the repo root.
|
|
||
| # The ^"? is because git may quote weird file paths | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/))'; then | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/)|(scripts/[^/]+/pyproject\.toml)|(scripts/[^/]+/uv\.lock)|(pyproject\.toml)|(uv\.lock))'; then |
There was a problem hiding this comment.
You should add a comment here, as this doesn’t seem to be a permanent solution. In the upcoming follow-up PRs, we plan to remove the requirements folder, so this will need to be updated again.
There was a problem hiding this comment.
Added a comment explaining this is transitional (requirements/edx-sandbox + the compat .txt exports still live under requirements/, but a planned follow-up per public-engineering#543 removes the folder entirely, at which point this pattern needs updating again). See 1c592ea.
|
|
||
| # The ^"? is because git may quote weird file paths | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/))'; then | ||
| if git diff --name-only "$BASE_SHA" | grep -P '^"?((requirements/)|(scripts/.*?/requirements/)|(scripts/[^/]+/pyproject\.toml)|(scripts/[^/]+/uv\.lock)|(pyproject\.toml)|(uv\.lock))'; then |
There was a problem hiding this comment.
You should add a comment here, as this doesn’t seem to be a permanent solution. In the upcoming follow-up PRs, we plan to remove the requirements folder, so this will need to be updated again.
There was a problem hiding this comment.
Same fix applies here (duplicate thread) -- see 1c592ea.
| - name: Install Python dependencies | ||
| run: | | ||
| pip install -r requirements/edx/coverage.txt | ||
| pip install coverage diff-cover |
There was a problem hiding this comment.
why we are using pip here?
There was a problem hiding this comment.
Fixed -- this now uses uv sync --no-default-groups --only-group coverage --no-install-project --frozen, picking up the coverage dependency-group already defined in pyproject.toml for exactly this purpose. Restores the pinning that got lost when this was a bare pip install. See 1c592ea.
…coverage tools, clearer no-op) - check-consistent-dependencies.yml: note that the requirements/ watch pattern is transitional -- requirements/edx-sandbox and the compat .txt exports still live there, but a planned follow-up (see public-engineering#543) removes the folder entirely. - unit-tests.yml coverage job: replace the unpinned `pip install coverage diff-cover` with `uv sync --only-group coverage`, using the `coverage` dependency-group that already exists in pyproject.toml for exactly this purpose -- restores the version-pinning/reproducibility the rest of this migration preserved everywhere else. - Makefile: clarify that `local-requirements` is an intentional no-op kept for backwards compatibility, not dead code, so an external caller relying on it doesn't read as silently broken.
|
Items not addressed in this PR:
|
- setup-uv installs and manages its own Python via the `python-version` input (sets UV_PYTHON, which uv resolves/downloads lazily on first use) -- actions/setup-python alongside it is redundant. Verified per file that no bare `python`/`pytest` invocation runs before the uv-managed venv is created and put on PATH (via the existing `echo "$PWD/.venv/bin" >> "$GITHUB_PATH"` step in each job), so this doesn't change what Python interpreter later steps actually get. js-tests.yml was missing that GITHUB_PATH line (harmless today since nothing in that job calls bare python, but added for consistency with every sibling job and defense against a future indirect call). - Bumped `astral-sh/setup-uv@v7` -> `@v9` everywhere it's used, for consistency (v9.0.0 is the current latest release). - Left the `coverage` job's `actions/setup-python` step in unit-tests.yml untouched -- not flagged, out of scope for this pass.
astral-sh/setup-uv stopped publishing floating major-version tags after v7 -- v8.x/v9.x only exist as full semver tags. @v9 broke every job in the previous commit ("Unable to resolve action astral-sh/setup-uv@v9, unable to find version v9"). Corrected to @v9.0.0 everywhere.
|
Quick correction to my last batch of replies: the v7→v9 bump broke every job ( |
Manual workflow testing neededbuild imageI think building the docker image should be tested before merging of the PR Two workflows changed in this PR are
|
…migration-consolidated # Conflicts: # requirements/common_constraints.txt # requirements/edx-sandbox/base.txt # requirements/edx/base.txt # requirements/edx/coverage.txt # requirements/edx/development.txt # requirements/edx/doc.txt # requirements/edx/semgrep.txt # requirements/edx/testing.txt # requirements/pip-tools.txt # scripts/user_retirement/requirements/base.txt # scripts/user_retirement/requirements/testing.txt # scripts/xblock/requirements.txt
| (cd $$d && uv export --frozen --no-hashes --group test --no-emit-project); \ | ||
| } > $$d/requirements/testing.txt; \ | ||
| done | ||
|
|
There was a problem hiding this comment.
The two loops over the same directory list can be merged into one — each export only depends on its own lock, not on the other directory. The second loop also has no error handling: a failed uv export silently writes an empty .txt and the target still exits 0.
Suggested replacement:
@# scripts/user_retirement and scripts/structures_pruning: base + testing (test group) compat exports.
@for d in scripts/user_retirement scripts/structures_pruning; do \
echo ; \
echo "== $$d ===============================" ; \
uv run --no-project --with edx-lint edx_lint write_uv_constraints $$d/pyproject.toml && \
(cd $$d && uv lock ${UV_LOCK_OPTS}) && \
{ \
echo "# GENERATED FILE, DO NOT EDIT DIRECTLY."; \
echo "# Compatibility export for anyone still 'pip install -r $$d/requirements/base.txt'"; \
echo "# directly instead of using uv. Source of truth: $$d/pyproject.toml / uv.lock."; \
(cd $$d && uv export --frozen --no-hashes --no-emit-project); \
} > $$d/requirements/base.txt && \
{ \
echo "# GENERATED FILE, DO NOT EDIT DIRECTLY."; \
echo "# Compatibility export for anyone still 'pip install -r $$d/requirements/testing.txt'"; \
echo "# directly instead of using uv. Source of truth: $$d/pyproject.toml (test group) / uv.lock."; \
(cd $$d && uv export --frozen --no-hashes --group test --no-emit-project); \
} > $$d/requirements/testing.txt \
|| exit 1; \
doneThree minimal changes from the current code: \ → && \ after uv lock, ; → && between the two export blocks, and || exit 1 after the last export.
- upgrade-one-python-dependency.yml's add-paths was missing root pyproject.toml/uv.lock, unlike its sibling compile-python-requirements.yml. make upgrade-package edits both (via the tomlkit script and uv lock --upgrade-package), so without these entries the generated PR would silently omit the actual dependency-upgrade changes. - docs/concepts/testing/testing.rst told contributors to run `uv sync --only-group testing`, which excludes [project.dependencies] entirely (verified via `uv sync --dry-run`: acid-xblock, analytics-python, bridgekeeper, and codejail-includes -- all direct project dependencies, acid-xblock explicitly needed for tests -- are absent under --only-group but present under --no-default-groups --group, which is what the Makefile's test-requirements target actually uses). Fixed to match.
|
Could we add an explicit dependency mapping table to the PR description? The current description mentions files being "absorbed into pyproject.toml dependency-groups" but a reviewer has to trace through the code to figure out what replaced what. Something like this would make it self-contained:
The last row is especially worth calling out — it is the only intentional behavior change that affects individual developers and is currently invisible in the description. |
- README.rst's "production" install instruction (`uv sync --no-default-groups`) omitted the `bundled` group. Confirmed master's requirements/edx/base.in was always `-r kernel.in -r bundled.in`, so production base.txt has always included bundled packages -- this silently dropped them. Fixed to match the Makefile's base-requirements target (`--no-default-groups --group bundled`). - Documented `uv sync --inexact` in requirements/README.rst as the replacement workflow for the dropped requirements/edx/private.in mechanism (keeping extra personal packages installed across syncs) -- there's no direct successor file, but the same need is covered by this flag rather than being a dropped capability with no answer.
|
Added the mapping table to the PR description, with one correction: |
| detect_changed_source_translations: ## check if translation files are up-to-date | ||
| i18n_tool changed | ||
|
|
||
| pre-requirements: ## install Python requirements for running pip-tools |
There was a problem hiding this comment.
I think perhaps both local-requirements: and pre-requirements were used in this make file and both are not required any more.
Let's provide no-op implementation of both for time bieng, deprecate them and add in comment to remove them in future,
There was a problem hiding this comment.
On reflection, went with deleting pre-requirements outright rather than keeping both as no-ops. Its only job was installing pip-tools itself (pip install -r requirements/pip.txt etc.), which has no uv equivalent need — every target that depended on it (dev-requirements, base-requirements, test-requirements, compile-requirements) has been updated to not reference it at all, so there's no dangling dependency. local-requirements is different: it's still a plausible thing an external caller might invoke by name expecting some action, so a no-op with a comment makes sense there. pre-requirements was purely an internal implementation detail of the old pip-tools flow with no equivalent caller expectation, so I don't think a deprecated no-op earns its keep the same way. Happy to add it back as a no-op too if you'd rather keep both for a transition period.
| uv run --no-project --with edx-lint edx_lint write_uv_constraints pyproject.toml | ||
| uv lock ${UV_LOCK_OPTS} | ||
|
|
||
| @# Compatibility exports for external tooling (e.g. tutor's Dockerfile) that |
There was a problem hiding this comment.
Should we add a TODO here to make it explicit that these exports are transitional and should be removed once external consumers (Tutor, Devstack, etc.) migrate to uv sync? e.g.
| @# Compatibility exports for external tooling (e.g. tutor's Dockerfile) that | |
| @# Compatibility exports for external tooling (e.g. tutor's Dockerfile) that | |
| @# still does `pip install -r requirements/edx/<name>.txt` directly. These are | |
| @# GENERATED FILES -- see the header comment in each for what regenerates them. | |
| @# TODO: Remove these exports once external consumers (Tutor, Devstack, etc.) | |
| @# have migrated to `uv sync`. Tracked in https://github.com/openedx/public-engineering/issues/543 |
| installed:: | ||
|
|
||
| pip install -r requirements/edx/base.txt -e . | ||
| uv sync --no-default-groups --group bundled --group assets |
There was a problem hiding this comment.
Why are we adding assets group while it was not required before?
There was a problem hiding this comment.
This is a real new organizational split, not scope creep — it separates frontend-build tooling from the runtime/dev dependency sets that already existed before this migration.
Previously, the Python packages needed to build/compile static assets (libsass, nodeenv, etc. — see scripts/compile_sass.py) weren't materialized as their own concept in the old pip-tools requirements layout; they were bundled into whichever broader requirements file happened to include them. PEP 735 dependency-groups let us pull that out into its own minimal assets group, so uv sync --no-default-groups --only-group assets installs only the ~4 packages actually needed to build assets, rather than the runtime/dev superset. docs/references/static-assets.rst documents this as the intended way to set up an asset-building-only environment.
|
@irfanuddinahmad Please confirm why check_dependencies (3.12) does not execute and keeps showing a pending status. |
…rt failure - SHA-pin astral-sh/setup-uv (was @v9.0.0, a floating tag) to its actual commit SHA across all workflows that use it, matching every other action pin in this repo. Verified via `git ls-remote --tags` + `gh api repos/.../commits/<sha>` that the SHA is a real, valid commit (not a tag-object SHA, a documented failure mode when hand-resolving tags). - unit-tests.yml's `coverage` job still had actions/setup-python alongside setup-uv -- redundant now that setup-uv's python-version input handles it, and already removed from every other job in this PR. Missed spot. - Makefile compile-requirements: merge the two scripts/user_retirement + scripts/structures_pruning loops into one, and add `&&`/`|| exit 1` error propagation to the second loop's exports. Previously a failed `uv export` in that loop would silently write an empty file and the target would still exit 0.
|
|
||
| - name: Set up Python environment | ||
| uses: actions/setup-python@v6 | ||
| - name: install system requirements |
There was a problem hiding this comment.
I think we can remove this step install system requirements as this workflow only runs make compile-requirements, which internally calls uv lock + uv export. Neither operation installs packages or builds C extensions, so libmysqlclient-dev and libxmlsec1-dev are not needed here.
Please check it.
| fi | ||
|
|
||
| - uses: actions/setup-python@v6 | ||
| - name: install system requirements |
There was a problem hiding this comment.
I think we can remove this step install system requirements as this workflow only runs make compile-requirements to check for lockfile drift. That call only does uv lock + uv export — no package installation, no C extension builds — so libmysqlclient-dev and libxmlsec1-dev are not needed here.
Please check it.
|
|
||
| - name: Set up Python environment | ||
| uses: actions/setup-python@v6 | ||
| - name: install system requirements |
There was a problem hiding this comment.
I think we can remove this step install system requirements as this workflow only runs make upgrade-package, which resolves to uv lock --upgrade-package + uv export. Neither operation installs packages or builds C extensions, so libmysqlclient-dev and libxmlsec1-dev are not needed here.
Please check it.
| The three standalone script directories at the repo root (``scripts/xblock``, | ||
| ``scripts/user_retirement``, ``scripts/structures_pruning``) each have their | ||
| own ``pyproject.toml`` + ``uv.lock`` too, independent of both the main app and | ||
| each other. This completes the migration from pip-compile tracked in |
There was a problem hiding this comment.
This completes the migration from pip-compile tracked in
public-engineering#543_.
I think in general readme rst docs, these lines should not be placed.
| ========================================= | ||
|
|
||
| If you don't have write permissions to openedx/edx-platform, you'll need to run these workflows on a fork. | ||
| The old ``requirements/edx/private.in``/``private.txt`` mechanism (an |
There was a problem hiding this comment.
This strategy should be discussed and properly announced in the PR description if its not.
farhan
left a comment
farhan
left a comment
There was a problem hiding this comment.
I am done with the first pass.
Inline comments and following 2 comments need to be addressed
5 participants
Summary
Migrates edx-platform from pip-compile/pip-tools to
pyproject.toml+uv, tracked in openedx/public-engineering#543.Old file -> new equivalent
requirements/edx/kernel.in[project.dependencies]requirements/edx/bundled.in[dependency-groups].bundledrequirements/edx/testing.in[dependency-groups].testingrequirements/edx/doc.in[dependency-groups].docsrequirements/edx/assets.in[dependency-groups].assetsrequirements/edx/development.in[dependency-groups].developmentrequirements/edx/coverage.in[dependency-groups].coveragerequirements/edx/semgrep.in[dependency-groups].semgreprequirements/edx/github.in[project.dependencies]requirements/constraints.txt+requirements/common_constraints.txt[tool.edx_lint].uv_constraints(hand-maintained) +[tool.uv].constraint-dependencies(machine-generated)requirements/pip-tools.{in,txt}requirements/edx/private.in/private.txtuv sync --inexact-- seerequirements/README.rstfor the exact workflowrequirements/edx/{base,assets,development}.txtand the four sub-projects' compat.txtfiles stay as machine-generateduv exportoutputs at their historical paths (for external tooling, e.g. Tutor's Dockerfile, that still installs from them directly) -- see the file-by-file breakdown below for exactly what each one exports.File-by-file summary
Root packaging & build config
[project.dependencies]list (previously just["setuptools"]), a new[dependency-groups]tree (bundled,testing,docs,assets,development,semgrep,ci,default),[tool.uv].default-groups/constraint-dependencies(machine-managed byedx_lint write_uv_constraints),[tool.edx_lint].uv_constraints(the hand-maintained version pins, each with a dated comment and issue link, carried over from the oldconstraints.txt), and a new[tool.coverage.*]tree.[tool.coverage.*]in pyproject.toml.pre-requirements/pip-synctargets replaced withuv sync --group ...targets;compile-requirementsrewritten to runuv lockfor the root project and loop over the 4 uv-managed sub-projects, then re-export compatibility.txtfiles at the old paths for external tools (e.g. Tutor's Dockerfile) that stillpip install -r requirements/edx/base.txtdirectly.runner = uv-venv-lock-runner+dependency_groups = testing; droppedqualityfromenvlist(quality now runs via a dedicated CI workflow/Makefile target, not tox) and the now-redundantusedevelop/commands_pre = make test-requirements.pip install -r requirements/edx/*.txttouv sync --group ....CI workflows
All follow the same mechanical pattern: drop the manual
pip cachesteps in favor ofastral-sh/setup-uv's built-in cache (enable-cache: true), replacepip install/pip check/pip freezewithuv pipequivalents, and addecho "$PWD/.venv/bin" >> "$GITHUB_PATH"so subsequent bare commands (pylint, tox, etc.) resolve inside the uv-managed venv.check-consistent-dependencies.yml: trigger-detection regex extended to also watchpyproject.toml/uv.lock/sub-project files, not justrequirements/.check_python_dependencies.yml: disabled (if: false), with a comment explainingfind_python_dependencies(edx-repo-tools) can't scanpyproject.toml/uv.lockyet; linked to the upstream tracking issue/fix PR.ci-static-analysis.yml,js-tests.yml,lint-imports.yml,migrations-check.yml,pylint-checks.yml,quality-checks.yml,semgrep.yml,unit-tests.yml: mechanical pip->uv swap as described above.compile-python-requirements.yml,upgrade-one-python-dependency.yml: updated to operate onpyproject.toml/uv.lockinstead of.in/.txt/constraints.txt. The dependency-downgrade script now edits[tool.edx_lint].uv_constraintsvia a propertomlkitTOML round-trip instead ofsed-patching a text file.static-assets-check.yml: same pip->uv swap, plus scoped the Python-deps-install step touv sync --no-default-groups --group bundled --group assets --frozen(rather than pulling in the fulldefaultgroup of dev/test/docs/ci tooling that this job doesn't need).units-test-scripts-structures-pruning.yml,units-test-scripts-user-retirement.yml: intentionally untouched -- they stillpip install -r scripts/.../requirements/testing.txt, and those compatibility files continue to be regenerated (viauv export) at the same paths, so these two workflows keep working unmodified.Documentation
.in/.txtworkflow, including how to replace the droppedprivate.inmechanism withuv sync --inexact.uv syncequivalents, scoped to the specific dependency-groups each workflow step actually needs.Dependency files (requirements/, uv.lock, scripts/*)
requirements/edx/*.in(all deleted): superseded by[project.dependencies]/[dependency-groups]in pyproject.toml.requirements/edx/base.txt,assets.txt,development.txt(kept, regenerated): machine-generateduv exportcompatibility exports at their historical paths for external tooling that still installs from them directly.base.txtexports[project.dependencies]plus thebundledgroup.requirements/edx/coverage.{in,txt},doc.{in,txt},testing.{in,txt},semgrep.{in,txt},bundled.in,github.in,kernel.in,openstack.txt(all deleted): fully absorbed into pyproject.toml dependency-groups; no external tooling installed from these paths directly, so no compatibility export was needed.requirements/edx/private.readme(deleted): documented theprivate.in/private.txtmechanism, which itself was never checked into git (git-ignored). No direct successor file -- see the mapping table above for theuv sync --inexactreplacement workflow.requirements/common_constraints.txt,requirements/constraints.txt(deleted): absorbed into[tool.uv].constraint-dependencies(machine-managed) and[tool.edx_lint].uv_constraints(hand-maintained) in pyproject.toml, with all original rationale comments and issue links preserved.requirements/pip-tools.{in,txt}(deleted): pip-tools itself is no longer needed.uv.lock,requirements/edx-sandbox/{pyproject.toml,uv.lock},scripts/{xblock,user_retirement,structures_pruning}/{pyproject.toml,uv.lock}(new): machine-generated lockfiles for the root project and its 3 independent standalone sub-projects (codejail sandbox, XBlock scripts, user-retirement scripts, structures-pruning scripts). Each sub-project sets[tool.uv] package = false(they're script bags, not installable packages) and carries only the subset of root constraints relevant to its own deps.scripts/*/requirements/{base,testing}.txt(kept, regenerated): same compatibility-export treatment asrequirements/edx/*.txt.Misc scripts
uv sync --no-default-groups --only-group assets --no-install-project), verified to actually compile Sass correctly with that reduced environment.pip install -r ...message updated touv sync --group default..venvto the linter's skip-dirs list.python.installpip-requirements mechanism with apost_create_environment/post_installhook that exports thedocsgroup viauv exportto a plainrequirements.txt, then lets RTD's own pip install it -- this preserves the constraint that Django/XBlock/etc. get resolved againstuv.lock's pins rather than picking up an unconstrainedsetuptools>=82that breaksfs/pyfilesystem2'spkg_resourcesimport.What's intentionally NOT done here (tracked externally)
find_python_dependenciesneedspyproject.tomlsupport beforecheck_python_dependencies.yml(disabled as part of this migration) can be re-enabled. Fix proposed in feat: teach find_python_dependencies to scan pyproject.toml and uv.lock repo-tools#735; both still open as of this writing.requirements/edx/{base,assets,development}.txtwith plainpip. Those stay asuv exportcompatibility artifacts rather than being deleted, so no action is required on Tutor's side right now -- but Tutor maintainers should be aware these paths are now machine-generated, not hand-compiled.Verification
uv lockresolves cleanly (445 packages, root project); all 4 sub-projects (requirements/edx-sandbox,scripts/xblock,scripts/user_retirement,scripts/structures_pruning) sync cleanly withuv sync --frozen.make compile-requirementsend-to-end for the root project and all 4 uv sub-projects -- regenerated compatibility export files are byte-identical to what's committed, confirming consistency.🤖 Generated with Claude Code