Skip to content

feat(CLI): add --parallel flag to phrase pull for concurrent downloads#1067

Open
Thibaut Etienne (tetienne) wants to merge 1 commit intophrase:mainfrom
tetienne:feat/pull-parallel
Open

feat(CLI): add --parallel flag to phrase pull for concurrent downloads#1067
Thibaut Etienne (tetienne) wants to merge 1 commit intophrase:mainfrom
tetienne:feat/pull-parallel

Conversation

@tetienne
Copy link

@tetienne Thibaut Etienne (tetienne) commented Mar 19, 2026

We run phrase pull in CI pipelines with 379 locale files across 3 projects. Sequential downloads take over a minute per run, and the latency adds up across branches and environments. This PR adds parallel downloads to dramatically reduce pull time.

Pairs well with #1066 (--cache for ETag-based conditional requests): parallel mode makes the initial pull fast, and caching makes subsequent pulls nearly free by returning 304 Not Modified (which don't count against rate limits).

Summary

  • Add --parallel flag (-p) to phrase pull that downloads locale files using up to 4 concurrent requests (matching the Phrase API concurrency limit)
  • Uses errgroup with SetLimit(4) for bounded parallelism
  • Results are collected and printed in original locale order after all downloads complete (clean, deterministic output)
  • A shared mutex coordinates rate-limit pauses across all workers
  • Only supported in sync mode; --parallel with --async warns and ignores
  • Extracts buildDownloadOpts helper to eliminate code duplication between sequential and parallel paths

Usage

# Parallel download (4 concurrent requests)
phrase pull --parallel

# Short flag
phrase pull -p

# Without flag: sequential as before (no regression)
phrase pull

Real-world results

Tested against a production project with 379 locale files across 3 Phrase projects:

$ time phrase pull
Downloaded en to locales/en/tradfile.json
Downloaded fr to locales/fr/tradfile.json
...
(379 files downloaded sequentially in ~1m 10s)

$ time phrase pull --parallel
Downloaded en to locales/en/tradfile.json
Downloaded fr to locales/fr/tradfile.json
...
(379 files downloaded in parallel in ~11s)
Mode Time Speedup
phrase pull 1m 10s baseline
phrase pull --parallel 11.5s ~6x faster

Design decisions

  • 4 concurrent requests: Matches the Phrase API's documented concurrency limit. Going higher would result in rejected requests.
  • errgroup with SetLimit: Standard Go concurrency pattern. Provides bounded parallelism, fail-fast error propagation via context cancellation, and clean Wait() semantics.
  • Ordered output: Results are stored in a pre-allocated slice indexed by position. Output is printed after all downloads complete, preserving the original locale file order regardless of goroutine completion order.
  • Rate-limit mutex: When any worker hits HTTP 429, it holds a shared mutex while waiting for the rate limit window to reset. Other workers block on the mutex before making their next request, preventing a thundering herd.
  • Shared buildDownloadOpts: Extracted the download options preparation (params, file format, tags) into a helper used by both sequential and parallel paths, eliminating ~20 lines of duplication.
  • Timeout: Parallel path uses context.WithTimeout matching the existing 30-minute timeout from the sequential path.

Test plan

  • go build ./cmd/internal/... compiles
  • go test ./cmd/internal/... passes
  • go vet ./cmd/internal/ clean
  • Manual test: phrase pull --parallel downloads 379 files in 11.5s
  • Manual test: phrase pull without --parallel behaves identically
  • phrase pull --parallel --async warns and ignores parallel

@tetienne Thibaut Etienne (tetienne) force-pushed the feat/pull-parallel branch 2 times, most recently from e4541d8 to 2fa4000 Compare March 19, 2026 14:44
@tetienne Thibaut Etienne (tetienne) marked this pull request as ready for review March 19, 2026 14:53
Download locale files using up to 4 concurrent requests (matching the
Phrase API concurrency limit) via errgroup.

Results are collected and printed in order after all downloads complete
for clean output. A shared mutex coordinates rate-limit pauses across
all workers.

Only supported in sync mode; --parallel with --async warns and ignores.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant