July release candidate (DO NOT MERGE)#919
Draft
ChristianPavilonis wants to merge 135 commits into
Draft
Conversation
Fastly dynamic backend names embed the first-byte and between-bytes timeouts so a registration can never be silently reused with a different transport configuration. Deriving those timeouts from the remaining wall-clock auction budget minted a new backend name on nearly every request, defeating cross-request TCP/TLS connection reuse (Fastly pools connections per backend name) and accumulating registrations toward the per-service dynamic backend limit. Compute the effective transport timeout from the configured provider timeout verbatim when the budget allows, floor the budget-bound value to 250ms buckets otherwise, and pass sub-quantum remainders through exactly so publishers with sub-250ms configured budgets keep launching. The quantized value feeds both the backend name and the registered configuration, so they cannot diverge. Rounding down never extends a transport cap past the auction deadline, which the mediator and dispatched-collect paths rely on to bound the </body> hold. Also add a Fastly platform test pinning predict_name == ensure for the same spec, since the orchestrator maps responses back to providers by predicted backend name. Fixes #847
Publisher pages were fully buffered before the first byte reached the client: the platform client materialized the origin body (10 MiB cap), the rewrite pipeline ran over an in-memory cursor, and the EdgeZero finalize buffered the assembled response while awaiting auction collection. TTFB therefore tracked full origin transfer plus the auction instead of origin first byte. - Add supports_streaming_responses() to PlatformHttpClient (default false, Fastly true) and request with_stream_response() on the publisher origin fetch only where honored - Teach the pipeline to consume Body::Stream asynchronously: BodyChunkSource (cumulative raw-byte cap via publisher.max_buffered_body_bytes), push-style BodyStreamDecoder/BodyStreamEncoder in streaming_processor - Replace the Fastly buffered finalize with publisher_response_into_streaming_response: a lazy Body::Stream that commits headers at origin first byte, streams rewritten chunks, and holds only the </body> tail for auction collection; bids still inject before body close - Share one hold implementation (hold_step_decoded_chunk / hold_finish_segments) between the lazy body and the writer-driven loop so the paths cannot drift; collect_non_html_auction dedupes the collect-before-stream path - Finalize brotli decode with close() so truncated origin streams error instead of silently truncating; decode failures emit stream_decode_error telemetry - Guard bodiless (HEAD/204/304) responses and log wasted auction dispatch, matching the buffered finalizer Local A/B on a 183 KB gzip publisher page with a live 3-slot auction (release builds, 20 interleaved rounds): TTFB median 741 ms buffered vs 161 ms streamed (-78%); guest wall time and wasm heap unchanged.
Address the deep-review findings on the streaming cutover: - Cap cumulative decoded bytes in BodyStreamDecoder against publisher.max_buffered_body_bytes: the chunk source only bounds raw compressed bytes, so a decompression bomb could expand ~1000x past it and push unbounded decoded volume through the rewrite pipeline - Detect truncated deflate streams: write::ZlibDecoder::try_finish accepts truncated input silently, so the deflate arm now drives flate2::Decompress directly and requires Status::StreamEnd at finalization; trailing bytes after the end marker stay ignored. Add truncated-gzip and truncated-deflate regression tests - Make BodyChunkSource::next_chunk cancellation-safe by polling the body in place instead of moving it out across an await; a cancelled pull no longer turns into a silent EOF - Log dispatched auctions dropped uncollected (client disconnect mid-stream or never-polled body) via DispatchedAuctionGuard; the guard is created before the lazy stream so unpolled drops log too - Share the pull+decode step between the lazy publisher body and the write-sink drivers (hold_step_next_chunk / passthrough_step), removing the unreachable!() error plumbing and the triplicated processor selection; document body_close_hold_loop_stream as groundwork for the buffered adapters' streaming cutover - Pass identity-encoded chunks through zero-copy and finish encoders by consuming them instead of allocating a throwaway replacement - Add a Fastly dispatch test asserting the publisher fallback returns Body::Stream without a stale Content-Length, plus a comment on why the publisher fetch gates streaming on capability while the asset path does not Behavior note: gzip bodies with trailing garbage after the trailer now error mid-stream; the old read-path decoder ignored them.
The buffered finalizer abandons a dispatched auction with processor_init_error telemetry when HTML processor construction fails; the streaming finalizer dropped the in-flight SSP responses silently. Make publisher_response_into_streaming_response async and emit the same abandonment before returning the construction error.
Publisher-specific bundles need diagnostics based on the modules they actually contain. Otherwise, missing identity integrations can be hidden by the default preset. Refresh the comparison when auctions begin so late publisher configuration is visible without repeating warnings. Resolves: #886
The server-side auction stream path only emitted a summary counter (ssp/mediator/winning/time), so an operator seeing winning=0 could not tell whether prebid returned nothing, errored, or bid below the floor. Serialize the full provider_responses (and mediator_response) into the ts-debug HTML comment so the SSAT surfaces the same prebid server response detail available from the /auction endpoint. Bid creative and metadata are attacker/partner-influenced, so neutralize the '-->' and '--!>' comment terminators before embedding to keep the dump inside the comment and out of the live DOM.
When Prebid Server returns a non-2xx status, the parser returned a bare
AuctionResponse::error with empty metadata — indistinguishable in the
ts-debug dump from a transport, parse, or timeout failure, all of which
tag error_type. An operator seeing status=error with metadata={} had no
way to know the upstream HTTP code without log access.
Attach error_type=http_status, the status code, and a 512-byte body
snippet to the error response metadata so the auction dump shows exactly
why prebid errored (e.g. a 4xx from a PBS rejecting the request).
Address PR review of the SSAT debug-dump change:
- The PBS non-2xx response body was attached to AuctionResponse.metadata,
which ProviderSummary clones verbatim into ext.orchestrator.provider_details
on the public /auction response — violating the documented invariant in
auction/orchestrator.rs. Drop the body from metadata, keep only the numeric
status, and log the snippet server-side at warn.
- Register ERROR_TYPE_HTTP_STATUS and match it in provider_status so PBS HTTP
errors get their own telemetry bucket instead of the transport_error
fallback.
- Bound the ts-debug dump: compact serialization capped at 256 KiB, and skip
the mediator_response line when no mediator ran.
- Correct the auction_html_comment and prepend_auction_debug_comment docs to
state the comment now embeds raw SSP creative markup (never enable in prod).
- Keep the targeted two-replace terminator neutralisation: a single
replace("--", ...) re-forms -->/--!> at odd dash-run junctions and is not
equivalent. Add a table-driven test over the comment-terminator vectors.
- Hoist test-local imports to module scope per CLAUDE.md.
…debug' into feat/ssat-write-prebid-response-debug
- Drop render_adm param: always include adm when creative present
- Gate GAM-bypass on per-bid debug_bid instead of a global window.tsjs flag
(removes SPA-staleness edge case, TsjsApi change, and whole flag-emit task)
- Correct fallback scope: cache fallback only when adm absent; render failure
after adm is supplied is not detectable
- Qualify sandbox claim: TS guarantees script-context escaping; bridge frame
isolation depends on the Prebid Universal Creative
- Reconcile with existing ad_init.test.ts coverage (rename, no duplicates)
- Fix test assertion messages to expect("should ..."); list exact clippy gates
- Task 3: assert observable DOM (GAM iframe src) instead of spying on the module-private injectAdmIntoSlot - Use existing make_bid helper + set .creative (make_test_bid_with_creative does not exist on this branch) - Hostile-adm regression test covers both U+2028 and U+2029 - Add 'cd docs && npm run format' to verification
build_bid_map now always inserts the winning creative as adm so the pbRender bridge can render it locally (no PBS Cache round trip); the verbose debug_bid blob and the GAM-bypass gate stay behind inject_adm_for_testing. Rename the param include_adm -> include_debug_bid and thread it through write_bids_to_state. Reconcile the by-default test to the new behavior, drop the now-redundant debug-only-adm test, and pin script-context escaping for a hostile adm (</script> + U+2028/U+2029).
The direct GAM-replace path (injectAdmIntoSlot) now fires only when the bid carries debug_bid, which is present only under inject_adm_for_testing. In production the always-present adm is rendered by the pbRender bridge and GAM stays in the loop. Add observable-DOM tests (GAM iframe src unchanged without debug_bid, rewritten with it), strengthen the bridge test to prove inline adm is preferred even when cache coords are present, and rename debug-adm terminology to inline adm.
Resolve five correctness and resource-safety findings from the PR #867 review of the end-to-end Fastly publisher streaming path. - Drive the deflate decoder to StreamEnd at finalization so a valid stream that exactly fills the internal output buffer is no longer rejected as truncated; the inflater is also drained after all input is consumed within a chunk. - Decode concatenated (multi-member) gzip bodies via MultiGzDecoder on both the streaming decoder and the buffered read pipeline so adapters agree. - Enforce the decoded-body cap during decompression through a bounded sink shared by the gzip and brotli codecs, so a compression bomb errors before its expanded bytes are buffered instead of after a full chunk expands; the deflate codec charges each produced block as it is emitted. - Drop the body of bodiless responses (HEAD, 204, 205, 304) in both the streaming and buffered finalizer Buffered arms, and add RESET_CONTENT to response_carries_body, so a buffered-unmodified stream body is never streamed to the client for a response that must be bodiless. - Keep the dispatched-auction guard armed across the collection await and disarm it only once collection reaches a terminal result, so a body dropped while collection is pending still logs the discarded SSP work. Add regression tests for the deflate output-buffer boundary, multi-member gzip, bodiless buffered stream bodies, and the auction guard sentinel.
Resolve the PR review by making transport-timeout canonicalization a platform capability and hardening auction backend-name correlation. - Move quantization behind PlatformBackend::canonicalize_transport_timeout_ms. Fastly floors budget-derived timeouts to a 250ms quantum with a bounded sub-quantum ladder [200,150,100,50]; other adapters use the exact remaining budget so bidder deadlines (Prebid tmax, APS timeout) are not shortened where no connection-pooling benefit exists. - Bound sub-quantum backend-name cardinality: exact 1-249ms values no longer pass through, capping the budget-derived names a single origin can mint toward the per-service dynamic backend limit. - Add a provider discriminator to PlatformBackendSpec, folded into every adapter's backend name, so two providers sharing one origin no longer collide on the response-correlation key. Reject a duplicate backend_to_provider insertion with an attributed launch failure instead of silently overwriting and misattributing a response. - Make the orchestrator call-site tests deterministic: record predicted and registered transport timeouts separately and assert exact equality via a controllable platform backend, and enumerate the sub-quantum ladder to assert a bounded name cardinality. - Correct the timeout-semantics comments that overstated absolute-deadline enforcement; the Fastly connect/first-byte/between-bytes timeouts bound connection, first-byte, and inactivity, not total response time. A true absolute deadline carried through the platform HTTP API remains follow-up work (#849).
Prebid non-2xx responses were reduced to bare provider errors, making intermittent failures difficult to diagnose. Surface safe HTTP metadata and bounded debug details while correlating server logs with the auction ID.
A configured bidder with no inline params and no matching override
expanded to `"bidder": {}`, which PBS rejects. After applying overrides,
drop fabricated empty bidders, preserve an explicitly supplied empty
object so genuine misconfiguration stays visible, and fall back to the
stored-request path when no eligible bidders remain.
…debug' into merge/pr-890-893-899-910-912
# Conflicts: # crates/trusted-server-adapter-fastly/src/platform.rs # crates/trusted-server-core/src/publisher.rs
…isolation Creative sanitization ran unconditionally on every markup bid, stripping `script`/`object`/`embed`/`form` and friends together with their inner content. For script-based creatives — the majority of programmatic display — that leaves nothing renderable, and the slot goes blank with no error: the leftover markup is usually a tracking pixel, so the ad server reports a successful render. Measured over 291 creative deliveries on a live publisher: a median 43% of bytes removed, 29 creatives reduced by more than 80%, and 20 reduced below 500 bytes. One bidder lost 100% of every creative; another lost 76% across 43 of them. Add `auction.sanitize_creatives` so sanitization can be disabled where creatives render in a foreign-origin frame (the Prebid Universal Creative inside the ad server's iframe), and make both creative controls opt-in: `sanitize_creatives` and `rewrite_creatives` now default to false, so a creative ships exactly as the bidder returned it unless a publisher asks for processing. Removing `allow-same-origin` from the creative iframe sandbox is part of the same change rather than a follow-up. Sanitization was documented as "the primary defense against malicious markup", with the sandbox as defense-in-depth — but the sandbox granted `allow-same-origin` alongside `allow-scripts`, which removes its origin isolation entirely. With sanitization now optional, that pairing would leave creative markup able to reach publisher cookies, storage, and same-origin fetches. The two sibling sandboxes (APS_RENDERER_SANDBOX, ADM_IFRAME_SANDBOX) already omit the token for exactly this reason; this brings the third in line, so the origin boundary no longer depends on an optional transform. Note the default change alters behaviour for deployments that never set `rewrite_creatives`: creative URL rewriting is now off unless enabled explicitly. Verified end to end: creatives pass through byte-for-byte (triplelift 8902 -> 8902, openx 22069 -> 22069, previously 100% and 35% losses), page renders with ads serving and no hydration errors.
Introduce query-activated diagnostic sessions, trace identities and telemetry, bounded GPT and Prebid render ownership, creative acknowledgements, and the browser timeline overlay. Add deterministic integration fixtures and coverage across edge adapters and browser render paths.
# Conflicts: # crates/trusted-server-adapter-fastly/src/backend.rs # crates/trusted-server-adapter-fastly/src/platform.rs # crates/trusted-server-core/src/auction/orchestrator.rs
…ping APS bids are bid-by-reference: interpretResponse sets the renderer descriptor on the Prebid bid as the custom trustedServerRenderer field, and a bidResponse listener registers it in window.tsjs.apsPrebidRenderers keyed by Prebid's generated adId. Prebid drops unknown top-level fields when it normalizes bids during addBidResponse, so the custom field can be gone before the bidResponse listener runs (observed in production: absent as early as bidAccepted). The listener then saw renderer === undefined and returned without registering; the registry stayed empty, the Universal Creative's request found nothing, and Prebid's default renderer threw "Missing ad markup or URL" (reason noAd) for every APS bid. Also stash the descriptor keyed by requestId (a first-class field Prebid preserves) when the bids are built, and fall back to it in the bidResponse listener when the custom field is absent. Additive: the existing field path is unchanged; the fallback engages only when Prebid has stripped the field. Bounded map. Adds a unit test.
- Remove now-dead resolved_gam_unit_path (superseded by
render_gam_unit_path) and its two tests; the default
/<network_id>/<slot_id> path now lives solely in render_gam_unit_path
- Tighten compile_unit_template and render_gam_unit_path to pub(crate)
- Document that validate_runtime must run after compile_unit_templates
for the {section} -> section_root check to fire
- Use fictional gam_network_id 99999 in the new build_slot_json test
…nto 954-per-section-gam-unit-path
# Conflicts: # crates/trusted-server-core/src/integrations/gpt_bootstrap.js # crates/trusted-server-js/lib/src/integrations/gpt/index.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
main.Included pull requests
setConfig()Test plan
cargo fmt --all -- --checkcargo test-fastly && cargo test-axum && cargo test-cloudflare && cargo test-spincargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare && cargo clippy-cloudflare-wasm && cargo clippy-spin-native && cargo clippy-spin-wasmcd crates/trusted-server-js/lib && npx vitest run