Request for feedback: Fetch posture proposal

[jasnell]

Within WinterTC, and eventually WHATWG if we can get consensus on the approach in WinterTC, I'm working on a proposal to alter the normative requirements for fetch for non-web browser runtimes based on the reality of implementations across the various runtimes.

The draft can be found here: WinterTC55/fetch-workstream#14

I'm planning to present it to WinterTC at the upcoming meeting on July 31st.

Before doing so, I'd like to collect feedback / comments on the approach.

[KhafraDev]

I hate to be the 'guy who disagrees with everything', but I have a few highly negative opinions on the suggestions...

The sandboxing constraints differ from browsers. Some runtimes (e.g. Workers) do sandbox untrusted code, but under a different security model with different requirements.

If server environments also differ in these requirements, how will there ever be compatibility or conformance between them?

There is no single global user. Browsers act on behalf of one user whose credentials (cookies, client certs) are attached to outgoing requests automatically. Server-side runtimes typically handle requests from many users and manage credentials per-request.
There is no browser cookie jar. Cookie management is the application's responsibility.
There is no same-origin policy. The application controls all outbound requests.

There's no origin, it has nothing to do with security or users. In undici, we implement setGlobalOrigin which changes the behaviors of some apis slightly (ie. new Request('/') will work instead of throwing, mostly used for WPTs). Deno, from what I remember, implements a --location flag. You mention later that relative URLs are implementation-defined, but all implementations must now implement CORS and other security policies that explicitly rely on the existence of an origin to function.

These other runtimes have independently diverged from the spec by relaxing forbidden headers, ignoring CORS, and stripping navigation-related properties. The divergence is undocumented and inconsistent between runtimes. Code that moves between runtimes encounters subtle incompatibilities, and none of these runtimes can claim Fetch spec conformance despite implementing most of the spec correctly.

Rather than forking the Fetch spec, the goal is to define an alternative conformance mode within the Fetch spec itself, relaxing browser-specific constraints while preserving the core fetch model and API surface.

The divergence is documented in undici and node.

I heavily disagree with the conformance argument as it's completely subjective. Does Safari conform to the fetch spec despite failing ~14% of WPTs? Does Chrome, despite them failing ~7%? How can the goal to be conformance when later on you propose making a handful of getters implementation defined and changing the values for others?

This is answered later by stating "A conformant implementation operates in one of two fetch posture. Sandboxed (default): The implementation enforces the multi-origin sandboxing model. Unrestricted: The implementation does not enforce the multi-origin sandboxing model." This seems to unfairly benefit Cloudflare's model which, as mentioned, is sandboxed, where node's (and other server environments) is not. It also suffers the same problem: what is the origin? How can node support a multi-origin model if it does not have an origin?

Object.getOwnPropertyDescriptor(Request.prototype, 'mode')

This will return different values in different server environments, along with all of the other optional ones: (credentials, destination, referrer, referrerPolicy, keepalive, isReloadNavigation, isHistoryNavigation).

A fetch posture is a property of the environment in which fetch() is invoked. A fetch posture is one of:

'sandboxed': The current (default) posture. The environment enforces the browser's multi-origin sandboxing model. All existing spec behavior applies unchanged.
'unrestricted': The environment does not enforce multi-origin sandboxing. Browser-specific restrictions are relaxed as described below.

In browser terms, the posture would be associated with the settings object. For everything else, it is simply a constant internal property of the runtime.

Every algorithm and normative requirement in the spec assumes 'sandboxed' unless it explicitly states otherwise. This means:

Existing spec text does not need to be updated to say "if sandboxed, do X." The existing text is the sandboxed behavior by default.

Does this mean we now have to implement CORs, a cookie jar, forbidden request/response headers, etc.? This is a ton of work and will break a super-majority of people using undici and node's fetch. Why would we ever want to enable this, when it puts us at a major disadvantage? Node is the only project being maintained by volunteers, and now those volunteers (me) will have to spends months implementing dumb browser security policies. Many of these won't even cause obvious errors either; headers silently being dropped will cause untold hours of annoying debugging. CORS and the other security policies suddenly being enabled... do I even need to explain...?

Headers guard simplification:

This will just cause unnecessary struggles between DOM types and server types. We've had multiple issues and PRs opened in the past to align our types with the DOM types.

No Referer header is automatically appended (the application may set it manually via the now-unrestricted headers).

There's an entire algorithm for deciding the value of the Referer header, expecting users to implement this themselves, especially with redirects that they cannot hook into, is optimistic...

Port Blocking is implementation-defined
RequestInit.cache default is implementation-defined
Subresource Integrity is implementation-defined
Keepalive body size limits are implementation-defined
fetchLater / Deferred Fetch are not available
Navigation-related Request properties are presence-optional, value-undefined
file: URL Support is implementation-defined
Relative URL resolution is mplementation-Defined

This is just every-runtime-for-themselves. Many of these also, objectively, lead an implementation towards not being conformant with browsers and other server environments. This means that server environments will still have the problems mentioned above (differing slightly in semantics) while also being separate from browser environments.

[jasnell]

I think you may have significantly misread/misunderstood the draft proposal given some of the conclusions you appear to have jumped to.

There's no origin, it has nothing to do with security or users. In undici, we implement setGlobalOrigin which changes the behaviors of some apis slightly (ie. new Request('/') will work instead of throwing, mostly used for WPTs). Deno, from what I remember, implements a --location flag. You mention later that relative URLs are implementation-defined, but all implementations must now implement CORS and other security policies that explicitly rely on the existence of an origin to function....

I'm not following. Where is it asserting that all implementation must now implement CORS? That is only a requirement for browsers, as it currently is.

This is answered later by stating "A conformant implementation operates in one of two fetch posture. Sandboxed (default): The implementation enforces the multi-origin sandboxing model. Unrestricted: The implementation does not enforce the multi-origin sandboxing model." This seems to unfairly benefit Cloudflare's model which, as mentioned, is sandboxed, where node's (and other server environments) is not. It also suffers the same problem: what is the origin? How can node support a multi-origin model if it does not have an origin?

Again... what? The 'sandboxed' posture is the status quo of the current fetch spec. It's literally what the spec says today. There's no new requirement asserted. The 'unrestricted' posture is what multiple runtimes currently implement. There's absolutely nothing here favoring Cloudflare's model. Cloudflare Workers would use the 'unrestricted' posture just like Node.js, Deno, Bun, and other non-browser implementations.

Does this mean we now have to implement CORs, a cookie jar, forbidden request/response headers, etc.?

No. The proposal is pretty clear on this.

[jasnell]

To be clear:

'sandboxed' posture == The current fetch spec as written today. This posture would apply only to web browsers or anything that WHATWG would defined as "The Web" within the scope of its charter. The 'sandboxed' posture imposes zero new requirements on anyone.

'unrestricted' posture == The subset of features/requirements that are not needed by any non-web browser implementation. This includes Node.js, Workers, Deno, Bun, etc using fetch APIs on either client or server side. This posture is a subset of the full spec and imposes zero new requirements on anyone. What it does do is normatively relax requirements where implementations already were doing their own thing.

[KhafraDev]

Okay, I understand it a bit more. I read about it being on the settings object a dozen times and just didn't understand its purpose. The rest of my post is still relevant, including it being Cloudflare focused, most of it is a checklist for what I assume Cloudflare wants (Request.cache, implementation-defined navigation getters, non-strict request method, etc.). Not that it's a bad thing, but why would we in undici want to differ from browsers (or in other words, align with WinterCG) in that case?

My two main issues therefore are:

1. why does this need to be part of WinterCG? It could be handled in a couple of notes in the WHATWG fetch spec.
2. this could be handled by a wintercg test suite instead of changes to the spec.

I was expecting it to be a configurable option on RequestInit because I didn't understand why there needed to be normative changes to the spec lol

[jasnell]

... including it being Cloudflare focused... most of it is a checklist for what I assume Cloudflare wants

It's entirely unclear how you're arriving at this conclusion or even what relevance that would have to the conversation. Everything in this has been discussed/explored as part of the WinterTC fetch-workstream looking at multiple non-browser implementations across multiple runtimes. There's absolutely nothing "Cloudflare focused" about the proposal.

... but why would we in undici want to differ from browsers

undici already differs from browsers in broad and significant ways.

• There is no CORS
• Forbidden headers are not enforced
• HTTP cache is not implemented
• redirect manual returns the actual response (not opaque)
• GC semantics are different
• Service worker mechanisms are not implemented
• CSP, mixed content, HSTS, etc all not implemented
• Incomplete fetch metadata headers
• file:// and about: not implemented
• Cookie jar not implemented
• Accept header always /

These variances mean that undici is already overwhelmingly closer in alignment with the proposed 'unrestricted' posture than with browsers.

... why does this need to be part of WinterCG

It wouldn't be? The entire point of the fetch-workstream under WinterTC is to propose updates to the WHATWG spec.

[KhafraDev]

These variances mean that undici is already overwhelmingly closer in alignment with the proposed 'unrestricted' posture than with browsers.

It has been my position for many years that people should expect it to work as it does in browsers. There are some cases in which we could do better and there are some cases in which it is not possible to do so. Why align further away from browsers?

It wouldn't be? The entire point of the fetch-workstream under WinterTC is to propose updates to the WHATWG spec.

PRs that browsers do not have motivation in implementing are very rarely added to the spec from my experience. Maybe it's different if you have the right connections. It could be handled in a handful of notes in the fetch spec or just a test suite. Not really important though. I don't understand why there has to be a change to the spec to handle this when the end goal is that this is added to the spec and WPTs are added. You could skip the middleman (browsers) and add a test suite.

[mcollina]

@KhafraDev I think clarifying and harmonizing the behavior of fetch() on server-side platform is a valid goal. So users can rely on that to build cross-runtime applications.

For obvious reasons we are already deviating from the spec. Clarifying that and putting it behind a shared test suite is worthwhile.

[KhafraDev]

But it's not just about clarifying and standardizing, it's asking us to move further away from the spec in a number of ways.

[jasnell]

... it's asking us to move further away...

No, it's not. Whatever level of conformance an impl wants to achieve is entirely up to the impl. This is asking for feedback on the proposed spec changes. If I was asking undici to move further away from the spec I would have opened a PR with specific changes. It's exactly about clarifying and standardizing. The proposal is about spec changes not undici code changes.

[KhafraDev]

Yes it is. There are a large number of changes that will actively break our users.

Browser-specific Request properties (mode, credentials, destination, referrer, referrerPolicy, keepalive, isReloadNavigation, isHistoryNavigation) follow a presence-optional, value-undefined rule:

    Implementations MAY include these properties on the Request prototype, or MAY omit them.
    If a property IS present, its getter MUST return undefined.

These getters move from functional (even if the value is 'lying') to returning undefined, unless I'm misreading.

RequestInit.mode is silently ignored. The internal request mode is effectively always 'cors' in the sense that no CORS-safelisted restrictions are applied, but the CORS enforcement is bypassed.

I can recall issues where changing mode broke users as we didn't properly implement parts.

edit: Yes, from Sol 5.6:

• It changes the outgoing Sec-Fetch-Mode header (lib/web/fetch/util.js:243).

• After a cross-origin redirect:

  • same-origin rejects the fetch.
  • no-cors returns an opaque response (status: 0, empty body).
  • cors returns a CORS-typed response.

• no-cors rejects methods other than GET, HEAD, or POST (lib/web/fetch/request.js:474).

• Streaming request bodies require cors or same-origin (lib/web/fetch/request.js:562).

---

In 'unrestricted' posture, there are no forbidden methods. The Request constructor does not throw for CONNECT, TRACE, or TRACK.

try { new Request('https://a', { method: 'TRACE' }); assert(false) } catch (e) { assert.ok(e) }

Referrer and referrer policy are not applicable

We respect both of these in undici.

If a non-empty integrity string is provided and the implementation does not support that integrity algorithm, it MUST throw (not silently ignore).
Implementations MAY support any subset of integrity algorithms (including none beyond the empty string).

This sounds like a breaking change, haven't looked at SRI in a while though.

[jasnell]

Yes it is.

Please don't presume to tell me what my own proposal is proposing.

Spec changes that are optional for implementers do not "actively break" users.

At this point, I think you've made your position clear. I'd like to hear from the rest of the undici developers.

[KhafraDev]

You're talking to the sole maintainer of fetch, EventSource, and WebSocket. Other contributors occasionally make changes, but it's more or less just me. I don't mean any disrespect, but I pointed out a number of changes we have to make that would break our users. If we don't, then we do not conform to the unrestricted posture.

Spec changes that are optional for implementers do not "actively break" users.

The end goal is for undici (and node) to implement this. If we implement this, it breaks our users.

[jasnell]

The end goal ...

The end goal is to clarify/refine normative requirements in the spec for non-browser implementations. What undici chooses or does not choose to implement is an entirely different conversation.

... changes we have to make ...

There are no changes you would have to make.