diff --git a/openspec/changes/add-bundled-extension-distribution/design.md b/openspec/changes/add-bundled-extension-distribution/design.md index a309435e..8838dba1 100644 --- a/openspec/changes/add-bundled-extension-distribution/design.md +++ b/openspec/changes/add-bundled-extension-distribution/design.md @@ -1,27 +1,97 @@ ## Context -`add-extension-mechanism` delivers the extension *mechanism* and bundled-directory *resolution*: lstk runs an `lstk-` found next to its binary, ahead of `PATH`. It intentionally defers *distribution* so the first release can validate bundled extensions by manual placement. This change automates getting LocalStack's bundled extensions into the install artifacts and keeping them version-matched. The code for these decisions was prototyped during `add-extension-mechanism` and then removed from that change; this change re-introduces it. +`add-extension-mechanism` (PR #340) delivered the extension *mechanism* and everything lstk needs at runtime: bundled-directory resolution (`extension.BundledDir` — the directory of the symlink-resolved lstk executable, searched ahead of PATH), descriptions loading (`extension.LoadDescriptions` reads `lstk-extensions.toml` from that directory, degrading to an empty map on any failure), and help wiring (`cmd/extension.go`). It intentionally deferred *distribution* so the first release could validate bundled extensions by manual placement. This change automates getting LocalStack's bundled extensions into the install artifacts and keeping them version-matched with lstk. + +The distribution code was prototyped during `add-extension-mechanism` and removed before the squash-merge; it is **not recoverable from git history** (the squashed commit contains only the runtime code and these openspec docs). Everything below is designed against the current tree. + +Because `BundledDir` is "the directory containing the symlink-resolved lstk executable", the whole distribution problem reduces to one rule: **each channel must land the bundled files in the directory where the real lstk binary lives.** Every decision below follows from that rule plus the constraints of the individual channel. ## Decisions -### Decision 1: Atomic, version-matched update of the `lstk`/`lstk-*` set +### Decision 1: Channel placement — one staging dir feeds all three channels (corrects two assumptions from the original draft) + +A single release-time staging tree is the source for every channel: `bundled/_/lstk-[.exe]` plus one os/arch-independent `bundled/lstk-extensions.toml`. + +**Binary archive** — the staged binaries and the descriptions file are added at the archive root as siblings of `lstk`, via `archives.files` entries in `.goreleaser.yaml` with a templated source (`bundled/{{ .Os }}_{{ .Arch }}/lstk-*`), `strip_parent: true`, and `info: { mode: 0o755 }` so execute bits survive download and staging. + +**Homebrew** — lstk ships as a **cask** (`homebrew_casks` in `.goreleaser.yaml`), not a formula; the original draft's "libexec" plan described a formula layout that does not exist here. A cask stages the whole release archive under the Caskroom and symlinks only the declared `binary "lstk"` into `bin`. Since lstk resolves its bundled dir through `EvalSymlinks`, the Caskroom staged directory — containing every archive-root sibling — **is** the bundled dir, with zero layout work. Two cask constraints: + +- The post-install hook currently de-quarantines only `#{staged_path}/lstk`; it must cover the staged directory recursively (`xattr -dr com.apple.quarantine "#{staged_path}"`), otherwise Gatekeeper blocks the first run of every bundled extension on macOS (the binaries are not notarized; quarantine stripping is the standard cask workaround, and it must now cover the whole set). +- Bundled extensions must **not** gain `binary` stanzas: they stay un-symlinked and are resolved only through the bundled dir. GoReleaser's generated cask declares only `lstk`; keep it that way (release-candidate checklist item). + +**npm** — the real binary lives in the platform-specific optional-dependency package (`@localstack/lstk--`), not the `@localstack/lstk` wrapper: `npm/launcher.js` execs it from there, so `os.Executable()` — and therefore the bundled dir — is the *platform package* directory. Bundled files must be copied into each platform package. `goreleaser-npm-publisher build` has no per-platform extra-files mechanism, so the release job post-processes its `dist/npm/lstk--/` output with a copy step before `npm publish` — the same seam already used to swap in the signal-forwarding launcher — mapping Node platform names to Go names (`win32`→`windows`, `x64`→`amd64`; `darwin`/`linux`/`arm64` map to themselves). + +**Rationale**: one resolution rule at runtime, one staging tree at build time; no channel grows its own layout concept, and the archive (already SHA-256-verified by the self-updater and hash-pinned by the cask) carries the extensions under the existing integrity checks for free. -`internal/update` treats `lstk` and its bundled `lstk-*` set (binaries + the descriptions file) as one unit. For the self-managed binary channel, the extractor stages every new `lstk`/`lstk-*` member next to its destination (`.lstk-new` siblings) and renames each into place, so an interrupted update never leaves `lstk` and a bundled extension at mismatched versions. For Homebrew and npm, the package manager replaces the whole package — and therefore the whole bundled set — atomically. +### Decision 2: Bundled binaries come from the private repo's releases, pinned by a file in this repo -**Rationale**: a bundled extension and the `lstk` that conveys its contract are released together; a mismatched pair could violate the `LSTK_EXT_API_VERSION` contract. Staging-then-rename keeps the swap crash-safe within a directory. +The private extensions repository — the same source of truth that builds the closed-source binaries and hand-authors the descriptions file — publishes **tagged releases** whose assets are the per-platform binaries (`lstk-__[.exe]`), `lstk-extensions.toml`, and a `checksums.txt` manifest covering them. -### Decision 2: Hand-authored descriptions file, release-validated by a shell script +This repo carries a **pin file**, `bundled/extensions.version` — a single line naming the private release tag. Each lstk release therefore maps deterministically and reproducibly to one extensions bundle; bumping the pin is an ordinary reviewable PR (automatable later from the private repo's release workflow). `scripts/fetch-bundled-extensions.sh` reads the pin, downloads the assets (`gh release download`), **verifies each against the bundle's `checksums.txt`**, and stages them under `bundled/` with canonical names. It hard-fails when any lstk target platform has no matching asset (subject to an explicit not-supported allowlist), so platform gaps surface at pull time, not as an empty-glob failure inside GoReleaser. -The descriptions file (`lstk-extensions.toml`) is hand-authored in LocalStack's private extensions repository — the same source of truth that builds the closed-source binaries — and shipped as-is. The open-source repo does not generate it. A release-time bash script, `scripts/check-descriptions.sh` (consistent with `scripts/test-integration.sh`), extracts the described command names (the bare left-hand identifiers of the flat `name = "…"` table — values are never parsed) and fails the release if any described name has no corresponding `lstk-` binary in the staged dir. A staged binary with no description is allowed (help degrades to name-only). +The credential is a **dedicated fine-grained read-only PAT** (contents: read on the private repo only), stored as a repository/organization secret — not a reuse of the broader `PRO_ACCESS_TOKEN`. Least privilege, independent rotation. -**Validation targets a single, host-native staging dir** — descriptions are os/arch-independent, so the check runs once against one staging dir (the release host's own OS), where binaries are bare `lstk-` with no `.exe`/PATHEXT ambiguity. +**Alternatives considered**: GitHub Actions artifacts from the private repo (rejected: 90-day maximum retention breaks re-running a release build and any rebuild-from-tag; awkward cross-repo API); an S3 bucket or OCI registry (rejected for now: new infrastructure and a second credential lifecycle for no gain at this scale; revisit if the bundle outgrows release-asset limits). Storage lifecycle: the private release assets are the canonical, permanent store; the CI staging dir lives only for the job; the public release archives (and npm packages) are the permanent distribution copies — the extension *binaries* become publicly downloadable there by design, only *source* stays private. + +### Decision 3: Hand-authored descriptions file, release-validated by a shell script + +The descriptions file (`lstk-extensions.toml`) is hand-authored in the private extensions repository and shipped as-is; the open-source repo never generates it. A release-time bash script, `scripts/check-descriptions.sh` (consistent with `scripts/test-integration.sh`), extracts the described command names — the bare left-hand identifiers of the flat `name = "…"` table (`^[[:space:]]*([A-Za-z0-9][A-Za-z0-9_-]*)[[:space:]]*=`); values are never parsed — and fails the release if any described name has no corresponding executable `lstk-` in the staged dir. A staged binary with no description warns but passes (help degrades to name-only, per the `extension-bundling` spec). + +**Validation targets a single, host-native staging dir** — descriptions are os/arch-independent, so the check runs once against the release runner's own platform staging dir (`linux_amd64`), where binaries are bare `lstk-` with no `.exe`/PATHEXT ambiguity. **Rationale**: validating (not generating) keeps one source of truth in the private repo while preserving the version-lock guarantee; key-only parsing keeps the shell check trivially correct. **Alternatives considered**: a Go validator reusing the runtime `scanDir` + go-toml (rejected: extra build entrypoint and against the repo's "domain logic in Go, helpers in shell" grain for a small set-difference); generating the file from an in-repo manifest (rejected: duplicates a list the private repo already owns). -### Decision 3: Cross-channel packaging places bundled files where lstk resolves them +### Decision 4: Set-wise update = stage-then-commit; additive-only on the binary channel + +Only the self-managed binary channel needs Go work — `brew upgrade` and `npm install -g` replace the whole package directory, which replaces the whole set (including removals and renames) by construction. + +`internal/update/extract.go`'s `extractAndReplace` generalizes from "find `lstk` in the temp dir, rename it over the executable" to a stage-then-commit **set** replacement: + +1. **Discover** the set at the extracted archive root: the lstk binary, every executable `lstk-*`, and `lstk-extensions.toml`. An archive with no extensions yields a set of size one — today's behavior, byte for byte. Exact membership depends on Decision 7: under option (b) the set is `lstk`, `bundled-extensions`, and `lstk-extensions.toml`, and `bundled-extensions` must be discovered explicitly because it does not match `lstk-*`. +2. **Clean orphans**: remove any `*.lstk-new` siblings left by a previously crashed update. +3. **Stage**: copy each member into the destination dir (the running executable's directory) as a `.lstk-new` sibling — same directory ⇒ same filesystem ⇒ each upcoming rename is atomic — and set 0755 on binaries. Any failure here removes the staged files and leaves the installation untouched. +4. **Commit**: rename each `.lstk-new` over its final name — extensions and the descriptions file first, `lstk` itself **last**, so the load-bearing swap is the final act and "update reported success" implies the whole set committed. Windows keeps the existing rename-running-exe-to-`.old` dance for `lstk.exe` only; extensions are not running during `lstk update` and rename directly. + +**The honest guarantee** (this is what the spec promises — not "atomic across files", which POSIX cannot deliver): no partially-written file is ever visible under a final name; the mismatch window is a handful of renames; an interrupted update is healed by re-running `lstk update` (the flow is idempotent). Momentary version skew inside that window is benign by contract: `LSTK_EXT_API_VERSION` bumps only on breaking changes. + +**Additive-only**: the binary-channel update replaces and adds members but never deletes an `lstk-*` sibling absent from the new archive. Deleting safely requires knowing lstk *owns* the file — users may place their own extensions next to the binary, and the descriptions file is not an ownership manifest (the spec deliberately permits undescribed bundled binaries). A renamed/dropped extension therefore leaves its old binary behind on this channel only: it keeps working at its old version and shows name-only help (the replaced descriptions file no longer describes it, so help never disagrees with the shipped set). Deletion is deferred to the managed-extensions-directory work. Corollary: if a user manually placed an `lstk-` in the install dir and a release later bundles that same name, the update overwrites it — that directory is lstk's install dir; PATH is the supported home for user-installed extensions. + +**Alternatives considered**: making release validation two-sided so the descriptions file becomes a complete ownership manifest enabling safe deletion (rejected here: contradicts the `extension-bundling` allowance for undescribed binaries; revisit with the managed dir); a separate shipped manifest file (rejected: a third shipped file to solve a rename edge case). + +### Decision 5: Update continuity is a hard requirement — no existing install may be cut off from `lstk update` + +The transition release (the first that ships bundled extensions) must be reachable by every in-the-field updater, and a later extension-free release must be reachable from a bundling one (rollback). Per channel: + +- **Binary**: the *current* in-the-field `extractAndReplace` extracts the whole archive to a temp dir and touches only the `lstk` member; extra `lstk-*`/toml files are ignored. A pre-bundling lstk therefore updates into a bundling release cleanly (it just doesn't install the extensions — the user's next update, running the new code, does). Constraints this imposes: the lstk binary stays at the archive root under the same name, and the archive `name_template`/`checksums.txt` conventions are frozen (`buildAssetName` in `internal/update/github.go` reconstructs them on user machines). +- **npm**: the wrapper's `bin`, the launcher contract, package names, and the wrapper→platform-package `optionalDependencies` pinning are all untouched — the platform packages merely gain payload files the launcher never reads. A pre-bundling install's `npm install -g @localstack/lstk` (what `updateNPM` runs) replaces wrapper and platform package wholesale and picks up the extensions. +- **Homebrew**: `brew upgrade localstack/tap/lstk` (what `updateHomebrew` runs) installs the new version using the **new** cask definition — so the widened quarantine hook takes effect on the very release that first ships extensions; the old Caskroom version dir is removed wholesale. The cask's `binary "lstk"` stanza and tap location are untouched. +- **Both directions**: the new set-wise updater treats an archive without extensions as a set of size one, so downgrades/rollbacks to pre-bundling releases work. + +**Absent from the archive vs. failed to install.** These are different and only the first is tolerated. An archive that ships no extensions is a valid archive — a pre-bundling release or a rollback to one — and updates as a set of size one. But when an archive does carry extensions they are **not optional**: a member that fails to stage or commit fails the whole update, which reports the failing member and leaves the install on its previous version. Reporting success with a partial set would leave the user believing they have extensions they do not have. + +**The transition leaves an incomplete set, and repairing it needs a version-independent trigger.** The pre-bundling updater in the field replaces only `lstk` and ignores the archive's extra members; it cannot be made to fail retroactively, so a user crossing the transition on the binary channel lands on the new lstk with no extensions. The obvious answer — "their next update installs them" — does not hold: `applyUpdate` always jumps to the newest release, so after that update the user *is* current, and `Check` (`internal/update/update.go`) short-circuits on "already up to date" until another release ships. The repair must therefore key off the completeness of the installed set rather than the version comparison alone: `lstk update` on a current binary with a missing member installs the member instead of reporting up to date. What "complete" means depends on Decision 7 — under (b) it is the command list in `lstk-extensions.toml`; under (a) it needs a shipped list, since the directory contents cannot testify to their own completeness. + +### Decision 6: Release gating and local builds + +The `archives.files` entries land commented until the private pull is wired, then pull + payload are enabled **in one PR**: the PR-level `goreleaser check` job only validates config syntax (globs are not evaluated), but `goreleaser release/build` fails on a glob with zero matches, so the entries must never be live without the staging step that populates `bundled/`. After enabling, local snapshot builds require `scripts/fetch-bundled-extensions.sh` first; its `--stub` mode stages placeholder files for local, never-published builds so contributors without the private-repo credential can still run `goreleaser` locally. The staging tree is gitignored (only the pin file is tracked); it deliberately does not live under `dist/`, which `goreleaser --clean` wipes at startup. + +The first bundling release ships with the smallest viable bundle (a single extension) and is verified against the release-candidate checklist in `docs/extensions-bundling.md` — fresh install and upgrade-from-previous on all three channels — before further extensions are added to the bundle. + +### Decision 7: Bundled binary layout — OPEN, needs a call before implementation + +The bundle is a single closed-source binary that dispatches on the name it was invoked as; the `lstk-` entries are aliases of it, not separate programs. Neither the proposal nor Decisions 1 and 4 account for that — both describe the payload as a flat set of `lstk-*` files. How the aliases are materialized on disk is unresolved, and it determines the payload shape (Decision 1), the discovered set (Decision 4), and what a rollback archive contains (Decision 5). + +**Symlink aliases are ruled out.** Runtime resolution would accept them — `isExecutableFile` stats through the link (`internal/extension/resolve.go`) — but the updater destroys them. `extractTarGz` switches only on `tar.TypeDir` and `tar.TypeReg`, so a `tar.TypeSymlink` entry matches no case and is silently dropped; `extractZip` treats every non-directory entry as a regular file, materializing a symlink as an executable text file containing its own target path. Both failures are silent, and the downstream failure is silent too: a dangling alias makes `os.Stat` fail, `isExecutableFile` return false, and the extension simply not exist — no error, no log line. Windows additionally cannot create symlinks without Developer Mode or elevation. Fixing both extractors is possible but only reaches users one release *after* it ships, since the updater running during the transition is the old in-the-field one (Decision 5). + +That leaves two options: + +- **(a) Ship copies under each alias name.** Nothing above changes: the payload stays a flat set of `lstk-*` regular files, identical on every platform. The cost is size — `lstk` itself is ~30 MB, and both tar.gz and zip compress per entry, so identical copies do not dedupe. Three aliases is roughly +60 MB on disk and +20 MB per download, replicated into every npm platform package. +- **(b) Ship the single binary and dispatch on `argv[0]`.** The archive carries `bundled-extensions` and `lstk-extensions.toml`; lstk takes the command list from the descriptions file and execs the one binary with `Args[0]` set to `lstk-` — the busybox/git approach, and a single call site (`internal/extension/exec.go`). No aliases on disk, so no archive-format, extractor, or Windows concern, and the download carries one copy. Costs: bundled extensions become invocable only through lstk rather than directly from a shell (PATH-resolved third-party extensions are unaffected); `lstk-extensions.toml` becomes load-bearing, so its "degrade to an empty map on any failure" contract must become a hard error for the bundled set; and `Resolver.Resolve`/`List` grow a bundled-set branch that no longer derives from directory contents. + +**Recommendation: (b)**, because it removes the layout problem rather than encoding it into three channels, an extractor, and a rollback path. But it changes the runtime resolution contract, so it is a reviewers' call and not a detail to settle during implementation. **(a)** is the low-risk fallback, requires no change to any other decision, and should be chosen explicitly if the per-download size is acceptable. -GoReleaser includes the staged bundled binaries and the descriptions file at each archive root (and in the Homebrew/npm payloads), siblings of `lstk`. The public release workflow pulls the prebuilt closed-source binaries for each `os/arch` from a private artifact location into a `bundled/_/` staging dir, authenticated with a repository/organization secret; only binaries are pulled, never source. The GoReleaser inclusion is gated/commented until the private-CI pull is wired, so the credential-less open-source build never fails on an empty glob. +## Open Questions -**Rationale**: lstk resolves the directory next to its symlink-resolved executable, so every channel must land bundled files there — sibling-at-archive-root for tarballs, libexec for Homebrew, package dir for npm. +- **Decision 7 must be settled before implementation starts.** To close it: confirmation that the bundle really is a single multi-call binary, and its built size per platform — that number is the input to the (a)-versus-(b) trade-off. diff --git a/openspec/changes/add-bundled-extension-distribution/proposal.md b/openspec/changes/add-bundled-extension-distribution/proposal.md index 6cf2c1d9..8e33dd2b 100644 --- a/openspec/changes/add-bundled-extension-distribution/proposal.md +++ b/openspec/changes/add-bundled-extension-distribution/proposal.md @@ -2,18 +2,24 @@ The `add-extension-mechanism` change ships the extension *mechanism* — lstk resolves and runs `lstk-` executables (PATH and a bundled directory next to the binary) and conveys runtime context to them. It deliberately stops short of *distributing* LocalStack's own bundled extensions: the first release is a test bed where a bundled `lstk-` is validated by manual placement. This change closes that loop — it automates packaging LocalStack's (possibly closed-source) bundled extensions into the install artifacts, ships their help descriptions, and keeps the `lstk`/`lstk-*` set version-matched across updates — so bundled extensions like `lstk-deploy` are available immediately after a standard install with no manual step. +The runtime half already exists and is not touched by this change: `extension.BundledDir` resolves the directory of the symlink-resolved lstk executable, `extension.LoadDescriptions` reads `lstk-extensions.toml` from it, and help rendering consumes both. This change is exclusively pipeline (getting the files into the artifacts) and update (keeping them version-matched afterwards). + ## What Changes -- **Package bundled extensions into every install channel** (binary archive, Homebrew, npm) so they land in the directory lstk resolves, with no `PATH` change required by the user. -- **Pull the prebuilt closed-source bundled binaries from private CI** into the release build context, version-pinned to the lstk release, without exposing source in the public repository. -- **Ship a hand-authored descriptions file** (`lstk-extensions.toml`) alongside the bundled binaries, owned by LocalStack's private extensions repository, and **validate it at release time** against the staged binaries so a described-but-missing extension is a release-blocking error. -- **Update the `lstk`/`lstk-*` set atomically** in `internal/update`, so a running `lstk` and its bundled extensions are never left at mismatched versions across any install method. +- **Package bundled extensions into every install channel** so they land in the directory lstk resolves, with no `PATH` change required by the user: + - binary archive: `lstk-*` binaries and `lstk-extensions.toml` as siblings of `lstk` at the archive root; + - Homebrew: automatic via the cask's Caskroom staging of the whole archive (lstk ships as a **cask**, not a formula — no libexec involved); the cask's post-install quarantine hook is widened from the single `lstk` binary to the whole staged directory; + - npm: bundled files are copied into each **platform package** (`@localstack/lstk--`), where the real binary lives — not the wrapper package — via a post-processing step in the release job. +- **Pull the prebuilt closed-source bundled binaries from the private extensions repository's releases** into the release build context, **version-pinned via a pin file in this repo** (`bundled/extensions.version`), checksum-verified against the private release's manifest, authenticated with a dedicated read-only token, without exposing source in the public repository. +- **Ship the hand-authored descriptions file** (`lstk-extensions.toml`), owned by LocalStack's private extensions repository, and **validate it at release time** (`scripts/check-descriptions.sh`) so a described-but-missing extension is a release-blocking error. +- **Update the `lstk`/`lstk-*` set as one unit** in `internal/update` for the self-managed binary channel (stage `.lstk-new` siblings, then rename, lstk last); Homebrew and npm replace the whole package — and therefore the whole set — via their package managers. +- **Guarantee update continuity**: `lstk update` keeps working for every existing install across the transition — a pre-bundling lstk updates cleanly into the first bundling release on all three channels (Homebrew and npm especially, where the updater shells out to the package manager), and a bundling lstk updates cleanly from an archive that carries no extensions (rollback). An archive carrying no extensions is a valid archive and must not fail an update — but when an archive does carry them they are not optional: the update installs the complete set or fails, and never reports success with a partial one. ## Capabilities ### New Capabilities -- `extension-bundling-distribution`: Automated distribution and version-matched co-update of LocalStack's bundled extensions — cross-channel packaging (binary archive, Homebrew, npm), the private-CI binary pull, the release-shipped + release-validated descriptions file, and atomic updates of the `lstk`/`lstk-*` set via `internal/update`. Builds on the bundled-directory *resolution* delivered by `extension-bundling` in `add-extension-mechanism`. +- `extension-bundling-distribution`: Automated distribution and version-matched co-update of LocalStack's bundled extensions — cross-channel packaging (binary archive, Homebrew cask, npm platform packages), the pinned + verified private-release pull, the release-shipped + release-validated descriptions file, set-wise updates of `lstk`/`lstk-*` via `internal/update`, and update continuity across the transition. Builds on the bundled-directory *resolution* delivered by `extension-bundling` in `add-extension-mechanism`. ### Modified Capabilities @@ -21,12 +27,14 @@ The `add-extension-mechanism` change ships the extension *mechanism* — lstk re ## Impact -- **Touched code**: `internal/update` (atomic replacement of the `lstk`/`lstk-*` family + descriptions file), `.goreleaser.yaml` (archive/cask/npm payload inclusion), the public release workflow (private-CI pull, version pinning), and `scripts/check-descriptions.sh` (release-time validation). -- **Packaging/release**: binary archive, Homebrew formula/cask, and npm package lay out bundled extensions where lstk resolves them; the release workflow pulls prebuilt closed-source bundled binaries from private CI, authenticated with a repository/organization secret. -- **Docs**: re-introduce `docs/extensions-bundling.md` (on-disk layout per channel, the release pipeline, atomic update, the descriptions file and its validation). -- **External dependencies/services**: a private artifact location for the prebuilt bundled binaries and a release-time credential to pull them. +- **Touched code**: `internal/update/extract.go` (+ re-introduced `extract_test.go`) — set-wise stage-then-commit replacement; `.goreleaser.yaml` — archive payload entries and the cask quarantine hook; `.github/workflows/ci.yml` release job — private pull step, descriptions validation step, npm platform-package copy step; new `scripts/fetch-bundled-extensions.sh` and `scripts/check-descriptions.sh`; new `bundled/extensions.version` pin file (+ `.gitignore` entries for the staging dirs). +- **Packaging/release**: the binary archives gain `lstk-*` + `lstk-extensions.toml` at the root; the Homebrew cask inherits them via archive staging (hook widened); the npm platform packages gain them via post-processing; the release workflow pulls the pinned private release's binaries with a repository/organization secret (dedicated read-only PAT). +- **Docs**: re-introduce `docs/extensions-bundling.md` (on-disk layout per channel, the release pipeline, pin-bump process, local snapshot builds, update semantics and guarantees, rollback); update the CLAUDE.md Extensions section. +- **External dependencies/services**: the private extensions repository publishing tagged releases (per-platform binaries, `lstk-extensions.toml`, `checksums.txt`) and a release-time read-only credential to download them. +- **Public visibility note**: once embedded in the public release archives, the bundled extension *binaries* are publicly downloadable; only their *source* stays private. Any gating of what an extension does must happen at runtime (e.g. auth via `LSTK_EXT_CONTEXT`), not at distribution. ## Deferred (future work) - User-facing `lstk extension` management commands (`list`/`info`/`install`/`remove`) and a user-mutable managed extensions directory. - Internet-download of third-party extensions and any associated allow-listing / signature verification. +- Deleting stale bundled binaries on the self-managed binary channel when a release renames or drops an extension (requires an ownership manifest; folds into the managed-extensions-directory work). This change is additive-only there; Homebrew and npm remove stale members naturally via whole-package replacement. diff --git a/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md b/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md index 457a6dfb..54dbc84d 100644 --- a/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md +++ b/openspec/changes/add-bundled-extension-distribution/specs/extension-bundling-distribution/spec.md @@ -2,13 +2,13 @@ ## Purpose -Automate shipping LocalStack's own bundled extensions (for example a closed-source `lstk-deploy`) so they are available immediately after a standard install, carry their help descriptions, and stay version-matched with the `lstk` binary across updates. This builds on the bundled-directory *resolution* delivered by the `extension-bundling` capability (which lets lstk run a bundled extension that is present); here we cover how bundled extensions get *there* and stay correct. +Automate shipping LocalStack's own bundled extensions (for example a closed-source `lstk-deploy`) so they are available immediately after a standard install, carry their help descriptions, and stay version-matched with the `lstk` binary across updates. This builds on the bundled-directory *resolution* delivered by the `extension-bundling` capability (which lets lstk run a bundled extension that is present); here we cover how bundled extensions get *there*, how they are updated, and how existing installs keep updating across the transition. ## ADDED Requirements ### Requirement: Bundled extensions are available after a standard install -A set of extensions MAY be designated as bundled and SHALL be installed alongside `lstk` by the same single installation command across supported distribution channels (binary archive, Homebrew, npm), placed in the bundled-extensions directory, and resolvable immediately as `lstk ` with no separate install step. Packaging SHALL place bundled extensions where lstk resolves them without requiring the user to add them to `PATH`. The closed-source bundled binaries SHALL be built in private CI and pulled into the release build context version-pinned to the lstk release, without exposing source in the public repository. +A set of extensions MAY be designated as bundled and SHALL be installed alongside `lstk` by the same single installation command across supported distribution channels (binary archive, Homebrew cask, npm), placed in the bundled-extensions directory — the directory of the symlink-resolved lstk executable — and resolvable immediately as `lstk ` with no separate install step and no `PATH` change by the user. Concretely per channel: siblings of `lstk` at the binary-archive root; the Caskroom staged directory for Homebrew (the cask stages the whole archive; bundled extensions are not symlinked into `bin`); the platform-specific package directory (`@localstack/lstk--`) for npm, where the launcher-executed binary lives. The closed-source bundled binaries SHALL be built privately and pulled into the release build context without exposing source in the public repository. #### Scenario: Bundled extension available immediately @@ -20,15 +20,87 @@ A set of extensions MAY be designated as bundled and SHALL be installed alongsid - **WHEN** a user extracts the binary archive and places only `lstk` on `PATH` - **THEN** a bundled `lstk-deploy` sibling is still resolved by `lstk deploy` because lstk searches the directory alongside its executable -### Requirement: Bundled extensions update atomically with lstk +#### Scenario: npm install places the set where the real binary lives -Updating lstk SHALL update its bundled extensions to the matching version as a single, atomic set, so a running `lstk` and its bundled extensions are never left at mismatched versions. `internal/update` SHALL replace the lstk executable and its bundled extensions together regardless of the install method, or fail without partially updating. +- **WHEN** a user runs `npm install -g @localstack/lstk` +- **THEN** the bundled extensions and descriptions file are present in the platform package directory containing the Go binary the launcher executes +- **AND** the wrapper package, its `bin` entry, and the launcher behavior are unchanged + +#### Scenario: Bundled extension runs on macOS without a Gatekeeper block + +- **WHEN** a user installs via the Homebrew cask on macOS and runs a bundled extension for the first time +- **THEN** the extension executes without a Gatekeeper/quarantine prompt, because the cask's post-install hook de-quarantines the whole staged directory, not only the `lstk` binary + +### Requirement: Bundled extensions update as one set with lstk + +Updating lstk SHALL replace the lstk executable, its bundled extensions, and the descriptions file as one version-matched set on every install method. On Homebrew and npm this is inherited from whole-package replacement by the package manager. On the self-managed binary channel, `internal/update` SHALL stage every member of the new set next to its destination and rename each into place (the lstk binary last), guaranteeing that: no partially-written file is ever visible under a final name; a failure before commit leaves the installation untouched; and an interrupted commit is fully repaired by re-running `lstk update`. The binary channel is additive-only: it SHALL NOT delete an `lstk-*` sibling that is absent from the new archive (ownership of such files cannot be established; see design). #### Scenario: Bundled extensions updated with lstk - **WHEN** lstk is updated to a new version that ships a newer bundled `lstk-deploy` -- **THEN** the bundled `lstk-deploy` is replaced with the matching version as part of the same update -- **AND** an interrupted update does not leave lstk and the bundled extension at mismatched versions +- **THEN** the bundled `lstk-deploy` is replaced with the matching version as part of the same update on every install method + +#### Scenario: Interrupted binary-channel update is safe and recoverable + +- **WHEN** a binary-channel update is interrupted at any point +- **THEN** every file visible under a final name is complete (never truncated or partially written) +- **AND** re-running `lstk update` completes the replacement of the whole set + +#### Scenario: Renamed or dropped extension + +- **WHEN** lstk is updated to a version whose bundle renames or drops an extension +- **THEN** on Homebrew and npm the old binary is gone (whole-package replacement) +- **AND** on the binary channel the old binary MAY remain (additive-only) but appears name-only in help, because the replaced descriptions file no longer describes it + +### Requirement: Existing installs keep updating across the transition + +Introducing bundled-extension distribution SHALL NOT break `lstk update` for any existing install, in either direction. The update entry points per install method (`brew upgrade` for Homebrew, `npm install -g` for npm, archive download-verify-replace for binary) are unchanged. The conventions in-the-field updaters depend on SHALL be preserved: the archive name template and `checksums.txt` manifest, the lstk binary's name and archive-root location, the npm package names / wrapper `bin` / launcher contract, and the cask name, tap, and `binary "lstk"` stanza. + +Bundled extensions are payload rather than a precondition in one sense only: **an archive that carries none is a valid archive**. A release shipping no bundled extensions — a pre-bundling release, or a rollback to one — SHALL update successfully as a set of size one. When an archive does carry bundled extensions they are **not optional**: the update SHALL install the complete set or fail, and a partial set SHALL NOT be reported as a successful update. + +#### Scenario: Pre-bundling lstk updates into the first bundling release (binary) + +- **WHEN** a user on a pre-bundling lstk runs `lstk update` and the latest release bundles extensions +- **THEN** the update succeeds using the in-the-field updater (which replaces only the lstk binary and ignores the archive's extra members) +- **AND** the install is left with an incomplete set, since that updater predates bundling and cannot be made to fail +- **AND** the incomplete set is repaired by the next `lstk update`, which SHALL NOT wait for a newer release to become available + +#### Scenario: Pre-bundling lstk updates via Homebrew or npm + +- **WHEN** a user on a pre-bundling lstk installed via Homebrew or npm runs `lstk update` +- **THEN** the package manager replaces the whole package and the bundled extensions are present immediately after that single update + +#### Scenario: Rollback to an extension-free release + +- **WHEN** the new set-wise updater applies an archive that carries no bundled extensions +- **THEN** the update succeeds, replacing only the lstk binary (a set of size one) + +#### Scenario: Bundled extensions fail to install + +- **WHEN** the set-wise updater applies an archive that carries bundled extensions and any member fails to stage or commit +- **THEN** the update fails with an error naming the member that failed +- **AND** the installation is left on its previous version rather than reporting success with an incomplete set + +#### Scenario: Incomplete bundled set is repaired when lstk is already current + +- **WHEN** `lstk update` runs on an install whose lstk binary is already the latest version but whose bundled set is incomplete +- **THEN** the update SHALL NOT report "already up to date" +- **AND** it installs the missing members of the set +- **AND** on the binary channel, previously installed bundled extensions remain in place and still run + +### Requirement: Bundle provenance is pinned and verified + +Each lstk release SHALL be reproducibly tied to exactly one extensions bundle by a version pin committed to the lstk repository, changed only via ordinary review. The release process SHALL download the pinned bundle's prebuilt binaries and descriptions file from the private extensions repository's release assets, SHALL verify every downloaded asset against the bundle's checksum manifest before staging (hard fail on a missing or mismatching manifest), and SHALL fail when a bundled extension lacks a binary for any lstk target platform not explicitly allow-listed as unsupported. The credential used is read-only and scoped to the private extensions repository. + +#### Scenario: Checksum mismatch blocks the release + +- **WHEN** a downloaded bundled binary does not match the bundle's checksum manifest +- **THEN** the release fails before any artifact is built + +#### Scenario: Missing platform coverage blocks the release + +- **WHEN** the pinned bundle has no `lstk-deploy` binary for a supported lstk platform that is not allow-listed as unsupported +- **THEN** the release fails at the pull step with an error naming the missing platform ### Requirement: Hand-authored descriptions file, validated at release time @@ -48,5 +120,5 @@ A static descriptions file that maps each bundled extension's command name to a #### Scenario: Descriptions update atomically with the bundled set - **WHEN** lstk is updated to a version that bundles a renamed or re-described extension -- **THEN** the descriptions file is updated as part of the same atomic update +- **THEN** the descriptions file is updated as part of the same update - **AND** lstk never shows a description that disagrees with the bundled binaries diff --git a/openspec/changes/add-bundled-extension-distribution/tasks.md b/openspec/changes/add-bundled-extension-distribution/tasks.md index 4215d363..77248c06 100644 --- a/openspec/changes/add-bundled-extension-distribution/tasks.md +++ b/openspec/changes/add-bundled-extension-distribution/tasks.md @@ -1,22 +1,120 @@ -## 1. On-disk layout and packaging +> **How to read this plan.** Sections 1–3 have no dependency on the private extensions repo — they can be built and merged right away (section 1 changes nothing user-visible until archives actually contain extensions). Section 4 needs the private repo's owners and is the long-lead item. Section 5 turns packaging on and MUST merge in the same PR as the CI step from 4.3 — never separately (see design Decision 6 for why a half-enabled state breaks releases). Sections 6–7 are verification and docs. -- [ ] 1.1 Define the bundled-extensions on-disk layout next to the lstk binary (including the descriptions file) and how each channel populates it: binary archive (sibling files at the archive root), Homebrew (libexec, not symlinked to global bin), npm (package dir resolvable via the symlink-resolved exe path) — documented in `docs/extensions-bundling.md` (re-introduce this doc) -- [ ] 1.2 Wire GoReleaser archive/cask/npm payload inclusion of the staged `bundled/_/lstk-*` binaries and `lstk-extensions.toml`; keep it gated/commented until the private-CI pull is wired so the credential-less open-source build does not fail on an empty glob +## 1. Make `lstk update` replace all the files, not just the lstk binary -## 2. Private-CI binary pull +Today `internal/update/extract.go` extracts the downloaded archive and replaces exactly one file: the `lstk` binary. Once archives also contain extension binaries (`lstk-deploy`, …) and the descriptions file (`lstk-extensions.toml`), the updater has to replace all of them — without ever leaving a half-written file if the update is interrupted. The approach: copy the new files into the install directory under temporary names first (`lstk-deploy.lstk-new`), and only when every copy has succeeded, rename each one over the real name. Renames within a directory are instant and atomic, so nobody can ever run a half-copied binary. -- [ ] 2.1 Wire the public release workflow to pull prebuilt closed-source bundled binaries (e.g. `lstk-deploy`) from private CI into a `bundled/_/` staging dir, version-pinned to the lstk release, authenticated with a repository/organization secret, without exposing source +- [ ] 1.1 In `internal/update/extract.go`, build the list of files to replace by looking at the extracted archive root: the lstk binary (`lstk` / `lstk.exe`), every executable file named `lstk-*`, and `lstk-extensions.toml`. If the archive contains only `lstk` (all current releases, and any future rollback), the list has one entry and the updater must behave exactly as it does today. +- [ ] 1.2 Before doing anything else, delete any leftover `*.lstk-new` files in the install directory. These can only exist if a previous update crashed partway through; cleaning them up is what makes "just run `lstk update` again" always repair an interrupted update. +- [ ] 1.3 Copy phase: copy each file from the list into the install directory (the directory of the running executable) under the temporary name `.lstk-new`, and make binaries executable (0755). If any copy fails (disk full, permissions, …), delete the `.lstk-new` files and return an error — the existing installation must be completely untouched. +- [ ] 1.4 Rename phase: once all copies succeeded, rename each `.lstk-new` to ``. Rename the extensions and the toml first and the lstk binary **last**, so if the process dies mid-way the user still has a working lstk and a re-run finishes the job. Keep two existing behaviors as-is: on Windows, the running `lstk.exe` is first moved aside to `lstk.exe.old` (you cannot rename over a running exe there — this applies only to lstk itself, extensions aren't running during an update); and the cross-device copy fallback for installs where rename fails. If a rename fails, stop and return an error naming the file that failed — never report success with only part of the set installed. Renaming lstk last is what makes that safe: any failure before the final rename leaves the user on their previous, complete version. +- [ ] 1.5 Write the resulting guarantee into the package documentation so it survives future refactors: (a) a file visible under its real name is never truncated or half-written, (b) an interrupted update is fixed by re-running `lstk update`, (c) the updater never **deletes** an `lstk-*` file that isn't in the new archive — it can't tell a dropped bundled extension from a file the user put there themselves (design Decision 4 has the full reasoning). +- [ ] 1.6 Re-introduce `internal/update/extract_test.go` with tests that build small tar.gz/zip archives on the fly and cover each behavior above: an archive with lstk + two extensions + toml replaces all of them; an update that introduces a brand-new extension installs it; an archive with only `lstk` reproduces today's behavior; a failed copy leaves the installation untouched; leftover `.lstk-new` files from a fake earlier crash get cleaned up; an `lstk-*` file NOT present in the archive is left alone; a rename failure partway through returns an error and leaves lstk on its previous version; and the Windows zip/`.exe` variant works. +- [ ] 1.7 Repair an incomplete set even when the binary is already current. Anyone crossing the transition on the binary channel gets the new lstk with no extensions, because the updater that ran was their old one which ignores the archive's extra files. They cannot fix that by updating again: `applyUpdate` always jumps straight to the newest release, so they are already on it, and `Check` in `internal/update/update.go` reports "already up to date" until another release ships — leaving them without extensions for up to a week. Make `lstk update` compare the installed set against the set the release is expected to contain, and re-run the install when a member is missing, instead of short-circuiting on the version alone. What the expected set is depends on design Decision 7 — under (b) it is the command list in `lstk-extensions.toml`, under (a) it needs a shipped list, because a directory cannot testify to its own completeness. Cover it in `extract_test.go`/`update_test.go`: a current binary with a missing member installs the member; a current binary with a complete set still reports up to date. -## 3. Descriptions file shipping + validation +## 2. The release-time check that descriptions match binaries -- [ ] 3.1 Re-introduce `scripts/check-descriptions.sh` (bash, consistent with `scripts/test-integration.sh`): extract the described names from the hand-authored `lstk-extensions.toml` and fail the release if any has no corresponding `lstk-` binary in the staged dir (a staged binary with no description is allowed; runs against one host-native staging dir since descriptions are os/arch-independent) -- [ ] 3.2 Wire the release process to pull the hand-authored descriptions file from the private extensions repo into the staging dir and run `scripts/check-descriptions.sh` against the staged binaries +The descriptions file `lstk-extensions.toml` is a flat TOML table (`deploy = "One-line description"`), hand-written in the private extensions repo. If it describes an extension that we didn't actually ship a binary for, users would see help text for a command that doesn't work. This script makes that a release-blocking error. -## 4. Atomic version-matched update +- [ ] 2.1 Re-introduce `scripts/check-descriptions.sh` (plain bash, same style as `scripts/test-integration.sh`). Input: a directory containing the downloaded extension binaries and the toml. Behavior: read the names on the left-hand side of each `name = "…"` line (only the names — never parse the values, so a weird description string can't break the script); for each name, check an executable file `lstk-` exists in that directory; if any is missing, print which ones and exit non-zero (this fails the release). The reverse case — a binary present but not described — only prints a warning, because lstk's help intentionally falls back to showing such extensions name-only. +- [ ] 2.2 Test the script against fixture directories (a small test script or make target creating temp dirs): described-but-missing binary → fails and names it; described-and-present → passes; binary-without-description → warns but passes; empty or absent toml → passes (nothing is described, nothing to check). -- [ ] 4.1 Extend `internal/update` (`extract.go`) to replace lstk, its bundled `lstk-*` extensions, and the descriptions file as one atomic, version-matched set across all install methods (stage `.lstk-new` siblings, then rename); never leave them mismatched on an interrupted update — re-introduce `extract_test.go` coverage (`TestExtractAndReplaceUpdatesLstkSet`, `TestExtractAndReplaceAddsNewBundledExtension`) +Note: the check runs once per release, against the Linux/amd64 download directory only. Descriptions are the same for every OS, and on Linux the binaries have plain names with no `.exe`, so one directory is enough. -## 5. Tests and docs +## 3. Pinning which extensions bundle a release ships, and the script that downloads it -- [ ] 5.1 Integration test: bundled extension available immediately after a simulated standard install; bundled set updates atomically with lstk; descriptions file shipped where lstk reads it -- [ ] 5.2 Update `docs/extensions-bundling.md` and the CLAUDE.md Extensions section to document distribution + atomic update once enabled +The extension binaries are never committed to this repo. Instead, this repo records *which version* of the private extensions bundle each lstk release ships — a one-line "pin" file — and a script downloads exactly that version at release-build time. + +- [ ] 3.1 Add the pin file `bundled/extensions.version` containing a single release tag of the private extensions repo (e.g. `v0.1.0`). Add `.gitignore` rules so ONLY the pin file is tracked: the downloaded binaries land in `bundled/_/` folders and the toml at `bundled/lstk-extensions.toml`, and none of that may ever be committed. (Why the staging folder is `bundled/` at the repo root and not inside `dist/`: the release runs `goreleaser --clean`, which deletes `dist/` before building — it would wipe the downloads.) +- [ ] 3.2 Add `scripts/fetch-bundled-extensions.sh`. What it does, in order: read the pin file; download that tag's release assets from the private extensions repo with `gh release download` (repo name configurable via an env var, with a sensible default); verify every downloaded file against the `checksums.txt` that the private repo publishes in the same release — abort loudly if the manifest is missing or any hash doesn't match; then arrange the files into the layout the rest of the pipeline expects: binaries at `bundled/_/lstk-` (with `.exe` for Windows), executable bit set, and the descriptions file at `bundled/lstk-extensions.toml`. +- [ ] 3.3 Make the script fail — listing exactly what's missing — if any of lstk's six target platforms (`linux`/`darwin`/`windows` × `amd64`/`arm64`) has no binary for a bundled extension. A platform can be exempted by adding it to an `UNSUPPORTED_PLATFORMS` list at the top of the script, so skipping a platform is always a visible, deliberate choice. Without this check, a missing binary would surface later as a confusing "glob matched nothing" error inside GoReleaser. +- [ ] 3.4 Add a `--stub` flag that skips the download entirely and writes placeholder files into the same layout. This exists for contributors without access to the private repo who want to run a local `goreleaser` snapshot build (which fails if `bundled/` is empty once section 5 is merged). Print an unmissable banner that stub output must never be released. +- [ ] 3.5 When run without a token, fail with a message that says which secret/env var is needed and mentions `--stub` as the alternative for local builds. + +## 4. The private repo side, and wiring the download into the release (cross-team) + +- [ ] 4.1 Agree with the owners of the private extensions repo on what their releases must contain, and write it down in `docs/extensions-bundling.md`: each tagged release ships one binary per extension per platform, named `lstk-__` (plus `.exe` for Windows), the hand-written `lstk-extensions.toml`, and a `checksums.txt` covering every asset. They own the descriptions text; we only validate it. +- [ ] 4.2 Create the credential the release uses to download from the private repo: a fine-grained personal access token with **read-only** access to **only** that repo, stored as a repository/organization secret (e.g. `LSTK_EXTENSIONS_READ_TOKEN`). Deliberately not reusing `PRO_ACCESS_TOKEN` — the release should not hold broader access than it needs, and a read-only token can be rotated independently. +- [ ] 4.3 In `.github/workflows/ci.yml`, add two steps to the `release` job before the GoReleaser step: run `scripts/fetch-bundled-extensions.sh` (with the secret), then `scripts/check-descriptions.sh bundled/linux_amd64`. Either failing must fail the release. +- [ ] 4.4 Decide and document how the pin gets bumped when the private repo publishes a new bundle: manual PR to start (name who reviews it), with the option to automate later (the private repo's release workflow opening the bump PR). + +## 5. Turn on packaging in all three install channels (one PR, together with 4.3) + +⚠️ These changes reference the `bundled/` folder that only exists after the fetch script has run. GoReleaser fails a release when a `files:` pattern matches nothing, and the PR-level `goreleaser check` job won't catch it (it only checks config syntax, it never looks at the filesystem). So this section must merge in the same PR as the CI wiring in 4.3 — enabling one without the other breaks every release until reverted. + +- [ ] 5.1 Binary archives — in `.goreleaser.yaml`, add the bundled files to `archives.files` so they end up next to `lstk` at the archive root: + `{ src: "bundled/{{ .Os }}_{{ .Arch }}/lstk-*", strip_parent: true, info: { mode: 0o755 } }` and + `{ src: "bundled/lstk-extensions.toml", strip_parent: true }`. + (`strip_parent` drops the `bundled/linux_amd64/` folder prefix so the files sit at the root; the explicit `mode` keeps them executable regardless of how the download step left them.) +- [ ] 5.2 Homebrew — the cask needs no layout work at all: it stages the whole archive into the Caskroom, and lstk finds the extensions there automatically. Two things to do anyway: change the post-install hook in `.goreleaser.yaml` from de-quarantining only `#{staged_path}/lstk` to the whole staged directory (`xattr -dr com.apple.quarantine "#{staged_path}"`) — without this, macOS Gatekeeper blocks the first run of every bundled extension — and double-check the generated cask still symlinks only `lstk` into `bin` (extensions must stay un-symlinked; they're found via the bundled dir, not PATH). +- [ ] 5.3 npm — the real Go binary lives in the platform package (`@localstack/lstk-darwin-arm64` etc.), not in the `@localstack/lstk` wrapper, so that's where the extensions must go. The npm build tool can't add per-platform files, so add a step to the release job right after the existing "Install signal-forwarding launcher" step: for each `dist/npm/lstk--/` directory, copy in the matching `bundled/_/lstk-*` files and the toml. Node and Go name platforms differently — translate `win32`→`windows` and `x64`→`amd64` (`darwin`, `linux`, `arm64` are the same in both). Don't touch the wrapper package, its `bin` entry, or the launcher. +- [ ] 5.4 Document how to run a local snapshot build after this lands (fetch with a token, or `--stub`) in `docs/extensions-bundling.md`, and leave a one-line comment next to the new `.goreleaser.yaml` entries pointing there. + +## 6. Test plan for the update path + +`lstk update` is the riskiest part of this change, because a mistake there is not something we can fix in the next release — a broken updater cannot deliver its own fix. This plan is written to be reviewed on its own, before implementation starts. + +Two failures matter more than the rest, and every group below exists to rule one of them out: + +1. **Stranding an install.** Someone on today's lstk can no longer update, on any channel, in either direction. +2. **Silently shipping a partial set.** An update reports success but the user is left without extensions, or with extensions that don't match their lstk. + +Groups 6.2–6.6 are automated (unit tests in `internal/update/extract_test.go`, integration tests in `test/integration/`). Groups 6.7–6.8 are partly automated and partly manual. Group 6.10 is manual and runs against a real release candidate. Nothing here can be finished until design Decision 7 is settled — it determines what "the set" is on disk, so every case that names individual `lstk-` files is written against option (a) and needs rephrasing (not re-thinking) under option (b). + +- [ ] 6.1 Build the harness first. Everything below needs: a mock release server (the existing `LSTK_UPDATE_GITHUB_API_ENDPOINT` / `LSTK_UPDATE_GITHUB_DOWNLOAD_ENDPOINT` pattern) serving a `checksums.txt` and an archive; a helper that builds tar.gz and zip archives on the fly with a configurable member list, so every case can be expressed as "an archive containing X"; a copy of a **real previously released lstk binary** for the continuity cases, since the point is to exercise the updater that is actually in the field rather than a reimplementation of it; and `testEnvWithHome` isolation so no test touches the developer's own install. + +- [ ] 6.2 Replacing the set — the happy paths. Run each against both tar.gz and zip. + - lstk + two extensions + toml: all four files replaced, extensions executable (0755). + - An archive that adds an extension the install didn't have: it gets installed. + - An archive containing only lstk: behaves exactly as today, byte for byte (this is the rollback shape and the "no extensions yet" shape). + - An archive with lstk + toml but no extension binaries: succeeds, toml replaced. + - An `lstk-` on disk that the user put there and the archive doesn't contain: left untouched (additive-only). + - An archive that drops an extension the install has: the old binary stays, and help shows it name-only because the new toml no longer describes it. + - An install directory on a different filesystem from the temp dir: the cross-device copy fallback is exercised rather than skipped. + +- [ ] 6.3 Failure and interruption — the cases that decide whether a partial set can ever be reported as success. + - A copy fails partway (make one member unwritable): the update fails, the error names the file, no `.lstk-new` files are left, and the whole installation is untouched. + - A rename fails partway: the update fails, the error names the file, and lstk is still the previous version — this is the case that justifies renaming lstk last. + - Leftover `.lstk-new` files from an earlier crash: cleaned up before staging, so re-running always repairs. + - Killed during staging: nothing is visible under a real name, and a re-run completes. + - Killed during the rename phase before lstk itself is renamed: the old lstk still runs, and a re-run completes the set. + - The downloaded archive's checksum doesn't match `checksums.txt`: refused before any file is touched. + - No file visible under its final name is ever truncated or half-written, in any of the above. + +- [ ] 6.4 Crossing the transition — the scenario Peter raised, and the one most likely to bite real users. + - Forward: a **real pre-bundling lstk binary** updates against a bundling archive. It succeeds, ignores the extra members, and does not error. Guard this deliberately, because the invariant it depends on is easy to break later by renaming the binary inside the archive or moving it out of the archive root. + - The install is now incomplete: new lstk, no extensions. Assert that state explicitly rather than inferring it. + - Repair: running `lstk update` again — with no newer release available — installs the missing members instead of reporting "already up to date". + - Once complete, a further `lstk update` does report up to date and changes nothing. + - Version skip: the same forward path where the installed version is several releases behind the latest (the "back from holiday" case, N-1 → N+3). It must behave identically to the single-step case; the updater jumps straight to newest either way, so this is about proving there is no stepwise assumption anywhere. + - The same forward path on Homebrew and npm, where it should take **one** update rather than two, because the package manager replaces everything at once. + +- [ ] 6.5 Rollback and re-update. + - A bundling lstk updates from an archive with no extensions: succeeds, replaces only lstk, and leaves the previously installed extensions in place and still runnable. + - Updating forward again from that state restores the full matched set. + +- [ ] 6.6 Archive layout invariants that in-the-field updaters depend on. These are cheap tests that exist to fail loudly if someone changes packaging later: the lstk binary keeps its name and stays at the archive root; the archive name template and the `checksums.txt` name and format are unchanged (`buildAssetName` in `internal/update/github.go` reconstructs both on user machines, so a change here breaks updates for everyone already installed). + +- [ ] 6.7 Per-channel end-to-end. For each of the three channels: the bundled files land in the directory lstk resolves, `lstk ` runs the extension, and `lstk --help` shows its description. + - Binary archive: extract and run in place. + - npm: files are inside the **platform** package (not the wrapper), and the bundled dir resolves correctly through the `.bin` symlink the launcher is invoked by. + - Homebrew: files are in the staged Caskroom directory, the bundled dir resolves through the PATH symlink, and brew has symlinked **only** `lstk` — no extension may get its own `binary` stanza. + - For npm and Homebrew, also verify that `lstk update` (which shells out to the package manager) lands the matched set. + +- [ ] 6.8 Platform-specific. + - Windows: zip archives, `.exe` suffixes, resolution through PATHEXT, and the existing move-the-running-exe-aside dance — which applies to `lstk.exe` only, since extensions are not running during an update. + - macOS: the widened quarantine hook strips the attribute from the whole staged directory, and a bundled extension runs with no Gatekeeper prompt on a genuinely downloaded (not locally built) archive. + - Linux: baseline, and the case where the install directory is not writable by the current user — the update must fail cleanly with a useful message rather than half-applying. + - An install with more than one lstk on PATH: the existing multiple-install warning still reports correctly after an update. + +- [ ] 6.9 What must still work after an update, checked by running the real binary rather than inspecting files: the extension executes and reports its **new** version; help lists it with the description from the **new** toml; a bundled extension still wins over a same-named executable on PATH; and `LSTK_EXT_API_VERSION` / `LSTK_EXT_CONTEXT` are conveyed as before, so an extension that worked before the update still works after it. + +- [ ] 6.10 Manual release-candidate checklist, written into `docs/extensions-bundling.md` and executed on the first bundling release and after any packaging change. On each of the three channels: fresh install (`curl`+tar, `brew install localstack/tap/lstk`, `npm install -g @localstack/lstk`) and verify `lstk ` works immediately with its description in help; then, starting from the **previously released** version on each channel, run `lstk update` and verify it succeeds and lands the matched set — brew and npm are the ones to watch, since there `lstk update` shells out to the package manager. Also: on macOS verify a bundled extension runs with no Gatekeeper prompt; verify brew created a PATH symlink for `lstk` only; and verify the binary-channel transition case by hand, since it is the one path where the code doing the update is code we are not shipping. + +- [ ] 6.11 Exit criteria — the first bundling release does not go out until: every case in 6.2–6.6 passes on Linux, macOS and Windows in CI; 6.7 passes on all three channels; 6.10 has been executed by hand against the real release candidate and signed off; and the transition case (6.4) has been verified against a genuinely published previous release rather than a locally built stand-in. + +## 7. Documentation + +- [ ] 7.1 Re-introduce `docs/extensions-bundling.md` covering, for a reader who knows none of the context: where the bundled files live on disk for each install method; the release pipeline (pin file → download + checksum verify → descriptions check → packaging); the private repo's release-asset convention (4.1); how the pin gets bumped (4.4); how to do local snapshot builds (`--stub`); what an update does on each channel, the exact safety guarantee from 1.5, and why the binary channel never deletes extensions; how to roll back; and the release-candidate checklist (6.10). +- [ ] 7.2 Update the Extensions section of CLAUDE.md: distribution and co-update are now automated (remove the sentence deferring them to this change), link `docs/extensions-bundling.md`. +- [ ] 7.3 Run `make lint`, `make test`, `make test-integration`, and `goreleaser check` after the config edits; all green.