DENO_TLS_CA_STORE="system,mozilla" silently drops system certs (order-dependent regression from upstream Deno)

[vergarabcs]

Summary

In crates/deno_facade/cert_provider.rs, the "mozilla" match arm assigns to root_cert_store, overwriting it, while the "system" arm correctly appends. As a result, the documented value DENO_TLS_CA_STORE="system,mozilla" silently discards every cert that was just loaded from the system trust store — leaving only the embedded webpki bundle.

This is order-dependent ("mozilla,system" works, "system,mozilla" does not), and the _ match arm even advertises "system,mozilla" as the allowed format in its error message — so the value most users will copy from the error string is the broken one.

Repro

Corporate proxy (in our case Cato Networks) doing TLS MITM. Steps:

1. Set DENO_TLS_CA_STORE=system,mozilla in supabase/functions/.env
2. docker exec into the running supabase_edge_runtime_* container, copy the corporate CA into /usr/local/share/ca-certificates/, run update-ca-certificates. The bundle at /etc/ssl/certs/ca-certificates.crt now contains the corporate CA.
3. Restart the container so the runtime re-runs get_root_cert_store_provider().
4. From inside the container, openssl s_client -connect api.openai.com:443 -showcerts successfully verifies the chain via the system store (proving the cert is correctly installed).
5. A Deno fetch to the same host from a function still fails with:

   TypeError: error sending request for url (https://api.openai.com/...): client error (Connect): invalid peer certificate: UnknownIssuer
   

6. Change the env var to DENO_TLS_CA_STORE=system (or mozilla,system) and re-create the container — fetch succeeds.

Root cause

cert_provider.rs:

for store in ca_stores.iter() {
  match store.as_str() {
    "mozilla" => {
      root_cert_store = deno_tls::create_default_root_cert_store();  // ← REPLACES the store
    }
    "system" => {
      let roots = load_native_certs().expect("could not load platform certs");
      for root in roots {
        root_cert_store.add((&*root.0).into())...;                   // ← APPENDS
      }
    }
    _ => bail!("... (allowed: \"system,mozilla\")"),                 // ← suggests the broken value
  }
}

The comment in the file links to denoland/deno@v1.37.0 cli/args/mod.rs#L467 as the reference implementation. Upstream Deno is append-only on both arms:

"mozilla" => {
  root_cert_store.add_trust_anchors(            // ← appends, not assigns
    webpki_roots::TLS_SERVER_ROOTS.iter().map(...)
  );
}
"system" => {
  let roots = load_native_certs().expect(...);
  for root in roots {
    root_cert_store.add(...)...;
  }
}

So the regression was introduced when porting to use deno_tls::create_default_root_cert_store() — that helper builds a fresh store, but the surrounding loop assumed append semantics like upstream.

Suggested fix

Either:

(a) Extend rather than assign in the mozilla arm:

"mozilla" => {
  let mozilla_store = deno_tls::create_default_root_cert_store();
  root_cert_store.roots.extend(mozilla_store.roots);
}

(b) Mirror upstream by calling add_trust_anchors directly with webpki_roots (would need to depend on webpki-roots directly rather than going through deno_tls's helper).

Either way, order-independence should be a test: "system,mozilla" and "mozilla,system" must produce the same effective trust set.

Affected version

Reproduced on supabase-edge-runtime-1.73.13 (also visually confirmed present on main at the time of this report). Supabase CLI v2.95.4 on Windows 11 + Docker Desktop.

[AyaanFaisal21]

I'm interested in taking a look at this problem,

I’m going to verify the current behavior in crates/deno_facade/cert_provider.rs, then try the minimal fix of making the mozilla branch preserve the existing RootCertStore rather than replacing it. I’ll also look for the smallest regression test that proves DENO_TLS_CA_STORE=system,mozilla and mozilla,system preserve the same effective trust set.

If there’s a preferred test location or approach for cert-provider behavior, happy to follow that.