Run e2e test buckets in parallel with ctest -j - #8253
Closed
Amaury Chamayou (achamayou) wants to merge 3 commits into
Closed
Run e2e test buckets in parallel with ctest -j#8253Amaury Chamayou (achamayou) wants to merge 3 commits into
Amaury Chamayou (achamayou) wants to merge 3 commits into
Conversation
CI wall-clock time is dominated by the two virtual e2e jobs, which sit at 15.2 min (bucket_b) and 15.5 min (bucket_c + partitions) against a 15 min target. Both jobs pay ~5 min of fixed overhead (checkout, dependencies, build), so the tests themselves have to fit in well under 10 min. ctest ran the buckets strictly serially, while individual tests use wildly different amounts of the 16-core runner. Measuring node process lifetimes from the logs uploaded by run 33500039436 shows bucket_b spending 50s, 26s and 9s stretches running a single three-node network, and `partitions` occupying a whole 282s step at an average of 4.2 live nodes. Declare each e2e test's load as a PROCESSORS weight, in units of concurrently live CCF node processes, and run the buckets with `ctest -j`. ctest then keeps the sum of the weights of running tests within the budget, packing light tests alongside heavy ones without overcommitting the runner. The budget is 1.5 nodes per core, reflecting that CCF nodes in e2e tests mostly wait on timers and sockets rather than burning CPU. `partitions` is long but light, so label it bucket_c and select its configuration in the same ctest invocation. It now overlaps the rest of the bucket instead of running in a serial step of its own. It stays gated on the `partitions` configuration, so runs that do not ask for it are unaffected, and its iptables DROP rules only match its own nodes' ports. Raise the per-test timeout to 600s, since tests now share the runner. Add scripts/e2e-test-load.py to recompute the weights from a CI run, so they can be refreshed rather than guessed when a test's sub-tests change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24ad6978-a80f-45bc-b73b-268aeaf2b30e
Running bucket_c in parallel made `modules_test` and `auth` fail together with: npm error path /__w/CCF/CCF/js/ccf-app npm error command sh -c npm run build npm error > tsc npm error sh: line 1: tsc: command not found Three tests build the same npm trees outside their own workspaces: `modules_test` and `programmability_and_jwt` both call `npm_tests.build_npm_app`, which installs `js/ccf-app` and then builds `tests/npm-app`, and `auth`'s `run_interpreter_reuse` installs `tests/js-interpreter-reuse`, whose `file:../../js/ccf-app` dependency rebuilds the same package. Concurrently, one test's install replaces `js/ccf-app/node_modules` while another's triggers the package's `prepare` script, so `tsc` is missing when it runs. Give those three tests a shared ctest RESOURCE_LOCK, which guarantees ctest never schedules them concurrently. They stay free to overlap the rest of the bucket, including the long `partitions` test, so the lock costs little: the three total 79s against a bucket that is now paced by `partitions` at 282s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24ad6978-a80f-45bc-b73b-268aeaf2b30e
With the buckets running in parallel, VMSS Virtual B drops to 11.0 min and VMSS Virtual C to 9.9 min, which leaves VMSS Virtual A as the critical path at 14.1 min - under the 15 min target, but not by enough to call it reliable. Job A is paced by one thing: the Debug build with clang-tidy takes 7.9 min of its 14.1 (13:22:30 to 13:30:27 in run 33512624021). `ci-checks.sh`, the Python package tests and the unit tests together account for under a minute, so the only meaningful work that can leave the job is the e2e bucket. Run bucket_a on job C instead, which has the most headroom. ctest's `-L` takes a regex, so one invocation covers both buckets. bucket_a is small (lts_compatibility at 76s and tls_groups_test at 3s) and packs into the gaps alongside `partitions`. This leaves ACI SNP Milan at ~12.8 min as the slowest job, so there is no value in trimming job A further. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 24ad6978-a80f-45bc-b73b-268aeaf2b30e
Amaury Chamayou (achamayou)
marked this pull request as ready for review
September 1, 2026 14:05
Copilot started reviewing on behalf of
Amaury Chamayou (achamayou)
September 1, 2026 14:06
View session
Contributor
There was a problem hiding this comment.
Pull request overview
This PR reduces CI wall-clock time for virtual end-to-end jobs by enabling parallel scheduling of e2e test buckets via ctest -j, using per-test PROCESSORS weights (and RESOURCE_LOCK where needed) to avoid overcommitting shared resources.
Changes:
- Add per-test
PROCESSORSweights (and a sharedRESOURCE_LOCKfor npm build contention) to enable weighted parallel execution of e2e buckets withctest -j. - Introduce a CI helper (
scripts/ci-test-jobs.sh) to compute thectest -jbudget, with an override viaCCF_CI_TEST_JOBS. - Update CI workflows to run bucket tests in parallel and rebalance where bucket labels run (moving
bucket_aexecution to VMSS Virtual C and overlappingpartitionsvia configuration gating).
Custom instructions used
- None (no files from
.github/copilot-instructions.mdor.github/instructions/were loaded during this review).
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/e2e-test-load.py | Adds a utility to recompute PROCESSORS weights from CI node logs and ctest summaries. |
| scripts/ci-test-jobs.sh | Computes the ctest -j budget for CI e2e buckets, with an environment override. |
| CMakeLists.txt | Assigns PROCESSORS weights and adds a shared RESOURCE_LOCK for tests that build shared npm trees; relabels partitions into bucket_c while keeping it gated by configuration. |
| cmake/gersemi_definitions.cmake | Extends add_e2e_test argument definitions to include PROCESSORS and RESOURCE_LOCK. |
| cmake/common.cmake | Implements PROCESSORS (with a default) and optional RESOURCE_LOCK as CTest properties for add_e2e_test. |
| .github/workflows/ci.yml | Switches bucket execution to ctest -j with higher timeout, and runs `bucket_a |
| .github/workflows/ci-al4.yml | Switches bucket execution to ctest -j with higher timeout for the AL4 workflow. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CI wall-clock is dominated by the virtual e2e jobs. On run 33500039436 (
main, post-#8236/#8240), VMSS Virtual B and C were at 15.2 and 15.5 min against a 15 min target.This PR takes the slowest job from 15.5 min to 12.4 min, and leaves the whole workflow paced by ACI SNP Milan, which it does not touch.
Validated on run 33515162772, all green.
Root cause
ctestran the buckets strictly serially (-jwas only ever used for unit tests), while individual e2e tests use wildly different amounts of the 16-core runner. Most e2e tests already drive several CCF networks at once viainfra.runner.ConcurrentRunner, so their load varies by more than an order of magnitude.Measuring node process lifetimes from the uploaded logs (sum of node lifetimes / test duration = average concurrently live nodes):
The bucket_b timeline contained a contiguous 85s stretch running one small three-node network at a time on a 16-core box.
partitionswas the worst case: a whole 282s step at an average of 4.2 live nodes.Change
1. Weighted parallelism. Each e2e test declares a
PROCESSORSweight in units of concurrently live CCF node processes, and the buckets run withctest -j. ctest keeps the sum of the weights of running tests within the budget, packing light tests alongside heavy ones without overcommitting the runner. The budget is 1.5 nodes per core (scripts/ci-test-jobs.sh, overridable viaCCF_CI_TEST_JOBS), reflecting that CCF nodes in e2e tests mostly wait on timers and sockets rather than burning CPU.partitionsis long but light, so it is labelledbucket_cand its configuration is selected in the same ctest invocation, letting it overlap the rest of the bucket instead of occupying a serial step of its own. It stays gated on thepartitionsconfiguration, soctest -L bucket_cwithout-C partitions(ci-al4, the bucket snapshot check, coverage) is unaffected, and its iptables DROP rules only match its own nodes' ports.2. Resource lock for the shared npm tree. The first parallel run failed with
modules_testandauthracing injs/ccf-app: one test'snpm installreplacednode_moduleswhile the other's triggered the package'spreparescript, sotscwas missing.modules_test,authandprogrammability_and_jwtall build that tree, so they now share a ctestRESOURCE_LOCK. They remain free to overlap everything else, so the lock costs almost nothing.3. bucket_a moved to job C. With B and C fixed, job A became the critical path. It is paced entirely by its clang-tidy build (7.9 of 14.1 min), so the only meaningful work that can leave is the e2e bucket.
ctest -Ltakes a regex, so one invocation covers both buckets.The per-test timeout goes from 360s to 600s, since tests now share the runner.
scripts/e2e-test-load.pyrecomputes the weights from a CI run so they can be refreshed rather than guessed when a test's set of sub-tests changes. It also flags nodes that outlive their test.Coverage and testing
Test buckets match snapshotpasses, sotests/ci-buckets.txtis unchanged.recovery_test166.9 to 173.0s,e2e_logging84.0 to 87.5s,partitions281.6 to 283.8s), confirming the runner is not being starved.add_e2e_testapplies explicit weights, defaults unmeasured tests to 4, applies resource locks only when given, and preserves labels.ctest -L bucket_cwithout-Cstill excludespartitions,ctest -C partitions -L bucket_cincludes it, and-L "bucket_a|bucket_c"selects both buckets without touching bucket_b.PROCESSORS(four weight-8 tests take 13.6s at-j 8, 7.5s at-j 16, 4.5s at-j 32, and a test heavier than the budget still runs on its own) andRESOURCE_LOCK(three 3s tests sharing a lock take 9.2s at-j 32).scripts/e2e-test-load.pyreproduces the measured weights.gersemi,black,shellcheckandprettierclean; fullci-checks.shgreen in CI.Follow-ups (not in this PR)
governance_testnodes outlive their test by ~8.5 min, logging ~660 lines/min on a retry loop through the rest of the bucket. Cause:gov()rebindsnetworkto the object returned bytest_all_members, which stops the original network and recovers into a new one; the enclosing context manager only owns the original, so the recovered network, including the deliberately-untrusted nodes added after it, is never torn down.reconfiguration.run_allrunsrun_join_old_snapshot(22s) andrun_join_no_snapshot_against_original_primary(7s) serially after its main network, and runs the whole ~25-test reconfiguration suite twice (IPv4 and IPv6).partitions_test.runrunsrun_ledger_chunk_bytes_check(44s) andrun_in_place_restart_uncommittable_ledger_check(13s) serially after its main network; overlapping those needs per-instance iptables chain names, sincePartitionershares a globalCCF-TESTchain and flushes it on construction.