[core] Fix flaky timing-sensitive tests: events-consumer deferred-check budgets and TTL-expiration e2e timeout - #3531
Conversation
The 'duplicate event classes' tests reach their outcome through the
deferred check's multi-stage timer chain (promise queue -> setTimeout(0)
-> idle poll -> delay timer), which a loaded CI runner with coarse
timers can starve for whole seconds. The afterDeferredCheck poll capped
that at 2s and two follow-up assertions used vi.waitFor's 1s default,
inside the 5s default test timeout - regularly starved through on
Windows runners ('does not track hook deliveries' and 'leaves a
duplicate run_cancelled' flaked ~1.5x/day over the last 10 days,
failing with strandedEvent/parkedSummary still undefined).
The polls return as soon as their assertions hold, so raising the poll
timeout to 15s and the suite budget to 30s costs healthy runs nothing
while bounding only genuinely stalled runners.
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
The other distributedAbortController tests run with a 60s budget; the TTL-expiration one got 30s. Its 3s TTL is trivial, but on a fresh prod deployment cold starts plus queue backlog routinely push run start + first stream delivery past 30s - it timed out on two apps simultaneously in a single Tests run over the last week. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
🦋 Changeset detectedLatest commit: e6b258e The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 228513ms → this run 182329ms (Δ -46184ms, -20%) ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
|
No backport to This is a test-only flaky-test fix, which would normally qualify, but neither test exists on To override, re-run the Backport to stable workflow manually via |
2 participants
Summary & Motivation
Two timing-sensitive tests flaked on loaded runners, and both polls return as soon as their assertions hold, so the larger budgets cost healthy runs nothing.
duplicate event classessuite: the deferred check's timer chain can be starved for whole seconds on Windows, so the poll bound goes to 15s (and the two follow-up assertions use it instead ofvi.waitFor's 1s default) under a 30s suite budget.distributedAbortControllerTTL-expiration e2e test gets the 60s its siblings already have, since cold starts on a fresh prod deployment push run start plus stream delivery past 30s.Test Plan
events-consumer suite passes locally; e2e test collection verified.