[experiment] rshell in Rust

[AlexandreYang]

No description provided.

[AlexandreYang]

⏺ Got it — that's enough to sketch a plan. A few I'm picking defaults on rather than asking; push back if any feel wrong.

  Workspace layout (rust/ subdir)

  rust/
  ├── Cargo.toml                 # virtual workspace
  ├── crates/
  │   ├── rshell-cli/            # binary crate; clap-derive; entry point
  │   ├── rshell-interp/         # runner, redirections, expansion driver, var scope
  │   ├── rshell-parser/         # syntax → AST (replaces mvdan/sh/v3/syntax)
  │   ├── rshell-expand/         # word expansion, globbing, splitting (replaces mvdan/sh/v3/expand)
  │   ├── rshell-builtins/       # all 30 builtins; one module per command
  │   ├── rshell-sandbox/        # AllowedPaths over cap-std
  │   ├── rshell-analysis/       # 1:1 port of analysis/ (symbol allowlist verifier)
  │   ├── rshell-telemetry/      # 1:1 port of pkg/fleet/installer/telemetry
  │   └── rshell-test-runner/    # YAML scenario harness, bash comparison
  └── README.md

  Why split this way: lets us land crates incrementally, keeps rshell-parser/rshell-expand reusable, and isolates the analyzer (which has nothing to do with the
  runtime).

  Foundational picks (please confirm)

  - String model: bytes, not UTF-8. Shell variables, file paths, and command output are byte streams in bash. Use bstr::BString/BStr everywhere a shell value
  flows; only convert to &str at output boundaries when we know it's UTF-8. Path arguments to syscalls go through OsStr/OsString. This is the single biggest
  correctness decision and it ripples through every crate.
  - CLI parser: clap (derive). Direct cobra analogue. For the internal pflag use inside builtins, also clap/pflag-style parsing via clap builder API per builtin
  (lighter than derive for repetitive use).
  - Regex: regex crate (RE2, linear time — matches your existing rule). For glob, use globset.
  - Logging: tracing for app logs, while still porting the DataDog telemetry span API 1:1 as a separate crate (since you specified that). The two coexist;
  telemetry is for the embedded host's tracer, tracing is for our diagnostics.
  - MSRV: pin to current stable (1.85+ for edition 2024). Document in rust-toolchain.toml.
  - Error model: thiserror for typed errors at crate boundaries, anyhow only in the CLI binary.

  Phased migration

  1. Phase 0 — scaffolding. Workspace, CI matrix (lint + test on linux/mac/windows), rust-toolchain.toml, Makefile targets that proxy to cargo.
  2. Phase 1 — test runner. Port the YAML scenario runner to Rust first, but have it shell out to the existing Go rshell binary. This proves the harness works
  and gives us our oracle for everything that follows.
  3. Phase 2 — parser. rshell-parser to the point it round-trips every script in tests/scenarios/. No interp yet.
  4. Phase 3 — expansion + sandbox + minimal interp. Enough to run pipelines, redirects, simple commands, with cap-std-backed AllowedPaths.
  5. Phase 4 — builtins. Port in dependency order: echo/true/false/pwd/exit first; then file-readers (cat/head/tail/wc/grep/cut/sort/uniq/tr/sed/find/ls/du);
  then exotics (ip/ss/ps/ping/uname/strings_cmd/printf/help).
  6. Phase 5 — analysis port. Last because it doesn't block runtime parity.
  7. Phase 6 — bake-off. Run the YAML harness against both binaries until output matches; benchmark binary size, startup, RSS.

  Open questions before I write code

  1. pkg/fleet/installer/telemetry source access. Is it OSS (datadog-agent repo) so I can read it to do the 1:1 port? If not, do you have an internal pointer /
  docs?
  2. Binary name. Same rshell, or rshell-rs during the cohabitation period to avoid $PATH collisions while testing?
  3. Should I start by writing a rust/DESIGN.md doc capturing the above, or jump to scaffolding the workspace?

  For #3, my lean is doc first so we can iterate on text rather than code, but I'll do whichever you want.
  │   ├── rshell-analysis/       # 1:1 port of analysis/ (symbol allowlist verifier)
  │   ├── rshell-telemetry/      # 1:1 port of pkg/fleet/installer/telemetry
  │   └── rshell-test-runner/    # YAML scenario harness, bash comparison
  └── README.md

  Why split this way: lets us land crates incrementally, keeps rshell-parser/rshell-expand reusable, and isolates the analyzer (which has nothing to do with the
  runtime).

  Foundational picks (please confirm)

  - String model: bytes, not UTF-8. Shell variables, file paths, and command output are byte streams in bash. Use bstr::BString/BStr everywhere a shell value
  flows; only convert to &str at output boundaries when we know it's UTF-8. Path arguments to syscalls go through OsStr/OsString. This is the single biggest
  correctness decision and it ripples through every crate.
  - CLI parser: clap (derive). Direct cobra analogue. For the internal pflag use inside builtins, also clap/pflag-style parsing via clap builder API per builtin
  (lighter than derive for repetitive use).
  - Regex: regex crate (RE2, linear time — matches your existing rule). For glob, use globset.
  - Logging: tracing for app logs, while still porting the DataDog telemetry span API 1:1 as a separate crate (since you specified that). The two coexist;
  telemetry is for the embedded host's tracer, tracing is for our diagnostics.
  - MSRV: pin to current stable (1.85+ for edition 2024). Document in rust-toolchain.toml.
  - Error model: thiserror for typed errors at crate boundaries, anyhow only in the CLI binary.

  Phased migration

  1. Phase 0 — scaffolding. Workspace, CI matrix (lint + test on linux/mac/windows), rust-toolchain.toml, Makefile targets that proxy to cargo.
  2. Phase 1 — test runner. Port the YAML scenario runner to Rust first, but have it shell out to the existing Go rshell binary. This proves the harness works
  and gives us our oracle for everything that follows.
  3. Phase 2 — parser. rshell-parser to the point it round-trips every script in tests/scenarios/. No interp yet.
  4. Phase 3 — expansion + sandbox + minimal interp. Enough to run pipelines, redirects, simple commands, with cap-std-backed AllowedPaths.
  5. Phase 4 — builtins. Port in dependency order: echo/true/false/pwd/exit first; then file-readers (cat/head/tail/wc/grep/cut/sort/uniq/tr/sed/find/ls/du);
  then exotics (ip/ss/ps/ping/uname/strings_cmd/printf/help).
  6. Phase 5 — analysis port. Last because it doesn't block runtime parity.
  7. Phase 6 — bake-off. Run the YAML harness against both binaries until output matches; benchmark binary size, startup, RSS.

  Open questions before I write code

  1. pkg/fleet/installer/telemetry source access. Is it OSS (datadog-agent repo) so I can read it to do the 1:1 port? If not, do you have an internal pointer /
  docs?
  2. Binary name. Same rshell, or rshell-rs during the cohabitation period to avoid $PATH collisions while testing?
  3. Should I start by writing a rust/DESIGN.md doc capturing the above, or jump to scaffolding the workspace?

  For #3, my lean is doc first so we can iterate on text rather than code, but I'll do whichever you want.