fix: parallelize R2 uploads in md-exports child-listings phase #16357
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The child-listings phase uploaded ~1651 section index files to R2 sequentially (one await per file inside a for...of loop). At ~300ms per request, this added ~8 minutes to every other production build, causing an alternating SLOW/FAST pattern (19 min vs 12 min). Collect uploads during the loop and batch them with pLimit(50), reducing the upload phase from ~8 min to ~10-20 seconds. Co-Authored-By: Claude <[email protected]>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DESCRIBE YOUR PR
Fix alternating slow/fast production build times (~19 min vs ~12 min) by parallelizing R2 uploads in the md-exports child-listings phase.
Root cause: The child-listings phase in
scripts/generate-md-exports.mjsuploads ~1651 section index files to R2 sequentially — oneawait uploadToCFR2()at a time inside afor...ofloop. At ~300ms per R2 request, this adds ~8 minutes to every other production build.Fix: Collect uploads during the loop, then batch them with
pLimit(50)(already imported in the file). This reduces the upload phase from ~8 min to ~10-20 seconds.pLimit(50)for concurrency control (same pattern as cache cleanup)IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
🤖 Generated with Claude Code