Skip to content

feat(stack): prepare slim resources on demand - #6250

Merged
jgoux merged 69 commits into
developfrom
juliengoux/cli-2123-stack-prepare-slim-resources-safely-on-demand-or-upfront
Aug 21, 2026
Merged

feat(stack): prepare slim resources on demand#6250
jgoux merged 69 commits into
developfrom
juliengoux/cli-2123-stack-prepare-slim-resources-safely-on-demand-or-upfront

Conversation

@jgoux

@jgoux jgoux commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make the @supabase/stack runtime core Effect-native across Node and Bun, with Promise adapters only at the public non-Effect edge
  • add package-owned off, lazy, and eager preparation policies with dependency closure, bounded concurrency, coalesced work, and deterministic disposal
  • resolve native slim-service releases through exact manifests and checksums, asynchronous in-process zstd decompression, host compatibility gates, typed post-processing, and atomic cache publication
  • select one exclusive native or Docker/Podman runtime, persist that concrete selection for managed stacks, and use it consistently for preparation, execution, logs, and cleanup
  • centralize managed identity, lifecycle documents, sticky port intent, control ownership, stale-owner recovery, and detached supervision in one manager and lifecycle facade
  • harden process-compose lifecycle transitions with Effect fibers, scopes, semaphores, typed failures, and interruption-safe resource cleanup

Context

Direct and managed stacks now share one service graph, preparation pipeline, port allocator, proxy, and process lifecycle. Direct callers own a scoped in-process handle. Managed callers use a detached supervisor with one durable document and one loopback control owner; CLI handlers delegate to that facade instead of maintaining PID-based liveness or parallel metadata.

When mode is omitted, selection prefers a usable Docker or Podman runtime. If neither is usable on a supported host, the first launch selects native mode and disables Docker-only services before port planning or managed state acquisition. Explicit native and Docker choices remain strict, preparation never falls back across modes, and a managed stack pins its selected runtime after ownership is acquired. Changing modes requires deleting and recreating that private managed stack state.

Preparation is driven by the service catalog. Eager resources are prepared during stack start, lazy resources use the same activation path as proxy and programmatic callers, and concurrent requests join the same owned work. Native archives are verified before extraction, decompressed through an interruptible callback boundary, validated for the current host, post-processed with checked exit codes, and published from private staging through atomic rename. Docker services resolve one canonical GHCR image and retry only classified transient pull failures.

Port allocation owns bound sockets until each runtime consumer takes over. Explicit ports remain exact; automatic managed assignments remain sticky and are coordinated with per-user claims. Managed control uses a deterministic sequence of eight loopback candidates, read-only probes never claim ownership, and mutations fail closed when no unambiguous owner or free candidate exists.

Native mode currently supports Postgres, Auth, and PostgREST. Docker-only services remain container-backed. On Linux, Docker Postgres performs only the required image setup as root and then drops to the host UID before touching bind-mounted data. Database bootstrap is a resumable observable one-shot dependency, and cleanup is scoped to exact owned containers, ports, processes, and auto-managed paths.

The managed document and identity markers are private unreleased state. This change intentionally implements the current model directly without legacy migration or compatibility adapters, while preserving fail-loud ownership and destructive-cleanup safeguards.

@jgoux

jgoux commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Automated review triage is complete for this batch.

Fixed:

  • socket-less Docker or Podman now uses internal Vector logs instead of crash-looping the stack
  • a Podman-selected stack no longer implicitly mounts the Docker default socket
  • download completion restores public state atomically and cannot overwrite a newer lifecycle transition
  • the stale Downloading coverage now exercises real download events, including a concurrent explicit stop

Deliberately deferred:

  • graceful stop cancelling an in-progress cold start remains a product lifecycle decision; disposal is the force-cancelling boundary
  • layer-scope single-flight alignment, version-prefix leniency, a catalog mandatory trait, and archive streaming remain outside this tracer-bullet scope

This is the final automated-fix pass for this PR. The remaining items need human product and architecture review rather than another mechanical review loop.

@jgoux

jgoux commented Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Final automated-review disposition:

  • Applied usable-socket checks before Vector selects container logs, label=disable for every explicitly selected Podman socket, and an atomic Downloading projection transition.
  • Added focused coverage for inaccessible-socket degradation and an explicit Docker-compatible Podman socket override.
  • Deliberately did not add an exhaustive XDG/per-UID path matrix or warning plumbing. Those path permutations do not add meaningful user-surface confidence, and a warning belongs at a later UX boundary.
  • The uncached core run exposed a separate test-only managed-control flake: its 2-second guard left little margin over a deliberate ~1.5-second retry schedule. The guard is now 5 seconds, still well below the real 35-second parent handshake; production timing is unchanged.

This is the final automated-review pass. The remaining recorded deferrals are unchanged; the PR is ready for human review.

@jgoux
jgoux marked this pull request as ready for review August 19, 2026 21:11
@jgoux
jgoux requested a review from a team as a code owner August 19, 2026 21:11
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@5da95a1542b68571ca5ddc04b3e00904cbd58fd1

Preview package for commit 5da95a1.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

// A fully staged replacement is now available, so an incomplete
// destination can be reclaimed without risking the last usable
// cache entry. Retry publication once; persistent filesystem
// failures still surface instead of looping forever.
yield* fs.remove(cacheDir, { recursive: true, force: true });
const retry = yield* fs.rename(stagingDir, cacheDir).pipe(Effect.result);

P2 Badge Avoid deleting a concurrently published cache

When two processes repair the same incomplete cache, both can fail the initial rename and observe the destination as incomplete; if one publishes after the other's check, this unconditional removal deletes the newly complete directory that the winner may already have returned to a starting service. That produces intermittent missing executables despite successful resolution. Use a cross-process lock or a replacement protocol that cannot remove a destination after another contender has published it.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/managed/document.ts
Comment thread packages/stack/src/BinaryResolver.ts Outdated
Comment thread packages/stack/src/versions.ts Outdated
Comment thread packages/stack/src/managed/lifecycle.ts
Comment thread packages/stack/src/StackPreparation.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8372805bf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/createStack.ts Outdated
@jgoux

jgoux commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the cache-publication finding from review 4976855133 in 4d31f6c. Invalid-destination repair now takes an atomic cross-process publication claim and revalidates the destination under that claim before removing anything. Private staging and the uncontended atomic-rename path remain unchanged. A deterministic resolver integration scenario covers a stale repairer observing another resolver publish and verifies that the winner is reused rather than replaced.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4d31f6c4aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/managed/manager.ts Outdated
Comment thread packages/stack/src/BinaryResolver.ts
Comment thread packages/stack/src/BinaryResolver.ts Outdated
Comment thread apps/cli/src/next/config/stack-config.ts
Comment thread packages/stack/src/LocalStack.ts
Comment thread packages/stack/src/LocalStack.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5f1603bf1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/PortAllocator.ts Outdated
Comment thread packages/stack/src/StackConfigResolver.ts Outdated
Comment thread packages/stack/src/ServiceCatalog.ts
Comment thread packages/stack/src/services/postgres-init.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 066c466b66

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/PortAllocator.ts Outdated
Comment thread packages/stack/src/supervisor.ts Outdated
Comment thread packages/stack/src/PortAllocator.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

# Check if already migrated (authenticator role created by initial-schema.sql)
if ${psql} -U supabase_admin -d postgres -tAc "SELECT 1 FROM pg_roles WHERE rolname='authenticator'" 2>/dev/null | grep -q 1; then
echo "Database already initialized, updating passwords..."
else

P1 Badge Use a completion marker after all native migrations

If initial native initialization is interrupted or a migration fails after initial-schema.sql creates the authenticator role, the next start takes this branch and skips every remaining init script and migration. The one-shot can then report success against a partially initialized database, leaving later services to fail on missing schema objects until the data directory is deleted; detect completion using a marker written only after the entire migration sequence succeeds rather than an early-created role.


const psql = `${pgBinDir}/psql -h 127.0.0.1 -p ${opts.dbPort}`;
const psqlOpts = `-v ON_ERROR_STOP=1 --no-password --no-psqlrc`;

P2 Badge Quote native migration executable and file paths

When cacheRoot contains whitespace, the generated shell expands this composite psql command as multiple words, so native Postgres initialization tries to execute only the path prefix and fails. The later init_flags and migrate_flags strings also append unquoted SQL paths, causing the same splitting for every -f argument; invoke the quoted executable separately and preserve migration paths as shell-safe arguments or arrays.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/LocalStack.ts
Comment thread packages/stack/src/PortAllocator.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 213f057a33

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/docs/architecture.md Outdated
Comment thread packages/stack/src/managed/document.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b2cb6382c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/LocalStack.ts Outdated
Comment thread packages/stack/src/errors.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed798cf19b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/ContainerRuntime.ts Outdated
Comment thread packages/stack/docs/architecture.md Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b9f7309865

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/PortAllocator.ts Outdated
Comment thread packages/stack/src/services/postgres-init.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05ced50120

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/BinaryResolver.ts

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f109e34a01

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/services/postgres.ts
Comment thread packages/stack/src/BinaryResolver.ts Outdated
Comment thread packages/stack/src/BinaryResolver.ts Outdated
Comment thread packages/stack/src/cleanup.ts Outdated

@avallete avallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The architecture holds: checksum-before-extract, exact-identity cleanup, omitted-mode fallback order, and name-only docker exec secrets.

Requesting changes on remaining start/attach/secret/port issues. Inline comments cover the ones I want fixed. The rest was checked on 1f3a1431a and is listed below so it is not silently dropped.

Please address

  • Attach can still skip the mode check on first launch (reply on the existing thread).
  • Native fallback still strips an explicit edgeRuntime (e.g. functions dev on a Docker-less host).
  • Native postgres-init never writes app.settings.jwt_secret.
  • docker run still puts JWT/passwords on argv (envArgs).
  • resolveConfig treats schema-default inspectorPort as exact; the managed plan does not. functions dev cold start can fail after the claim when config.toml omits the key.
  • process-compose graceful stop can run cleanup twice.
  • Control status timeout is treated as “no owner”.
  • Interrupt during allocateManagedPorts leaks claims.
  • Leaf chown of the Postgres bind mount does not fix root-owned parent dirs on Linux Docker (reply on the existing thread).

Seen, not requesting a change

  • PID-reuse leaving a live-looking port claim: real, but already deferred on the existing thread. Not re-opening.
  • Recursive chown of persisted PGDATA: the leaf chown is the intended first-init fix. Migrating a Docker-UID tree is a different product decision.
  • Empty init-scripts glob still committing cli_init: the bash is real, but it needs a published cache whose migrations dir exists with no scripts. Completeness only checks the directory; official bundles plus checksum-before-extract make this a mispackaged-cache path, not a normal start.
  • Cache repair fs.remove(cacheDir) under a repairer-only lock: path is version/target, not full identity. Adjacent to the publication-lock / identity item already deferred.
  • DX leftovers (duplicated docker-only sets, hand-rolled isNotFound, Effect runSync in listeners, tautological tests): mechanical, not merge-blocking.

Checked and dropped

  • “Every default functions dev cold start fails”: the omitted-key bug above is real. Blast radius is not — supabase init writes inspector_port = 8083, and supabase start does not set inspectorPort.
  • Replay native init on pre-cli_init data dirs: matches the stated no-legacy-migration policy for unpublished managed state.
  • /usr/local/bin/entry.sh crash-looping GHCR Postgres: catalog image is ghcr.io/supabase/cli; docker e2e starts through that entrypoint; CI is green.
  • createStack({ storage: {} }) must auto-strip Docker-only services: explicit enable should fail on native, not be removed. Default createStack() already leaves those off.
  • JWT on docker exec argv: already fixed (-e KEY + ServiceDef.env). Remaining leak is docker run / envArgs above.
  • Streaming zstd, 24h publication-lock reclaim, legacy formatVersion / auto launch migration: already deferred; not re-raising.

Comment thread packages/stack/src/supervisor.ts Outdated
Comment thread packages/stack/src/services/postgres-init.ts
Comment thread packages/stack/src/services/service-utils.ts Outdated
Comment thread packages/stack/src/StackConfigResolver.ts Outdated
Comment thread packages/process-compose/src/supervisor-runtime.ts Outdated
Comment thread packages/stack/src/platform-node.ts Outdated
Comment thread packages/stack/src/managed/manager.ts Outdated
@avallete
avallete self-requested a review August 21, 2026 20:08

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 441c767d4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread apps/cli/src/shared/telemetry/error-actionability.ts
Comment thread apps/cli/src/next/commands/functions/dev/functions-dev-runtime.ts Outdated
Comment thread packages/stack/src/Stack.ts
@jgoux
jgoux enabled auto-merge August 21, 2026 21:48
@jgoux
jgoux added this pull request to the merge queue Aug 21, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5da95a1542

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/stack/src/ContainerRuntime.ts
Merged via the queue into develop with commit 803e9ff Aug 21, 2026
37 checks passed
@jgoux
jgoux deleted the juliengoux/cli-2123-stack-prepare-slim-resources-safely-on-demand-or-upfront branch August 21, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants