diff --git a/rfc/0011-multi-player-design/README.md b/rfc/0011-multi-player-design/README.md new file mode 100644 index 000000000..8da217101 --- /dev/null +++ b/rfc/0011-multi-player-design/README.md @@ -0,0 +1,561 @@ +--- +authors: + - "@derekwaynecarr" +state: draft +links: + - https://github.com/NVIDIA/OpenShell/issues/1977 +--- + +# RFC 0011 - Multi-Player Support + +## Summary + +This RFC proposes adding multi-user support to OpenShell. Today, sandboxes and +providers are gateway-global with no ownership tracking or isolation between +users. This proposal introduces workspaces as hard isolation boundaries, an +expanded role model (Platform Admin, Workspace Admin, Auditor, User, Service +Account), owner-scoped access guards, per-workspace and per-user quota +enforcement, and audit trail enhancements. A `default` workspace preserves +backwards compatibility for single-player deployments. + +## Motivation + +OpenShell is currently a single-player experience. Every authenticated user sees +every sandbox and every provider. There is no concept of resource ownership, +tenant isolation, or delegated administration. This blocks several adoption +scenarios: + +- **Enterprise teams** cannot share a gateway without seeing each other's + sandboxes, credentials, and activity. There is no way to scope visibility + or enforce per-team resource limits. + +- **CI/CD and agent orchestration** workflows need machine identities (service + accounts) with scoped, rotatable credentials. Today the only option is + full-privilege OIDC tokens or mTLS certs with no role granularity beyond + admin/user. Workspaces provide the scoping boundary for these identities. + +- **Compliance and incident response** teams need audit trails that attribute + every sandbox and control-plane action to a specific principal. The existing + OCSF infrastructure logs sandbox-level events but does not consistently tag + them with the creating principal. + +- **Cost attribution** is impossible without ownership metadata. Operators cannot + answer "which team consumed how many GPU-hours last month." + +The existing codebase provides a foundation: OIDC authentication, a principal +model (User/Sandbox/Anon), two-tier RBAC, OCSF event infrastructure, and labels +on `ObjectMeta`. The gap is the isolation, ownership, and governance layer on +top. + +Leaving the current design unchanged limits OpenShell to single-operator, +single-team deployments, which constrains adoption and forces organizations +to run one gateway per team. + +## Non-goals + +- **Cross-gateway federation.** This RFC scopes multi-player to a single gateway. + Multi-gateway federation (e.g., routing users to regional gateways) is a + separate concern. +- **Fine-grained ABAC or policy language.** The role model uses coarse-grained + roles with workspace scoping, not attribute-based access control or a policy + DSL like OPA/Rego for authorization decisions. +- **UI/dashboard for user management.** This RFC covers the API and data model. + Administrative UIs are a follow-on. +- **Billing integration.** Principal attribution on resources enables cost + attribution; integration with billing systems is out of scope. +- **Sandbox-to-sandbox networking isolation.** Network isolation between + workspaces at the container/pod level is out of scope; this RFC addresses + control-plane isolation only. + +## Proposal + +### System Roles + +The role model expands from the current two-tier (admin/user) to five roles: + +| Role | Description | +|------|-------------| +| **Platform Admin** | Manages gateway configuration, auth providers, compute drivers, and quotas. Full visibility across all workspaces. | +| **Workspace Admin** | Manages users, providers, policies, and quotas within a single workspace. Cannot change gateway infra or access other workspaces. | +| **Auditor** | Read-only view of all sandboxes and audit logs across workspaces for monitoring, incident response, and compliance. Cannot create or modify sandboxes. Cannot access raw credential material. Enforces compliance through audit trail visibility, not direct action. | +| **User** | Creates and manages their own sandboxes within assigned workspaces. Uses credentials available in those workspaces. Default role for OIDC-authenticated humans. Covers both interactive users and app developers building agentic applications. | +| **Service Account** | Machine identity for CI/CD, automation, or agent-to-agent orchestration. Scoped to a workspace with explicit grants. | + +### Workspaces + +A workspace is a hard isolation boundary. Sandboxes, providers, and policies +within a workspace are invisible to other workspaces. Every resource belongs to +exactly one workspace. A `default` workspace exists for single-player backwards +compatibility. + +This is a first-class field in `ObjectMeta` alongside `created_by`. Within a +workspace, organizational grouping (teams, projects, cost centers) uses the +existing label system with well-known key conventions (e.g., +`openshell.dev/team=infra`, `openshell.dev/project=alpha`) rather than +additional dedicated fields. This: + +- Gives a clear security boundary (workspace) without over-modeling + organizational hierarchy. +- Allows multiple overlapping groupings within a workspace via labels. +- Keeps the proto surface minimal: `workspace` and `created_by` are the only + new fields. + +#### Workspace use cases + +- **Credential segmentation within a team.** Team members sharing a gateway each + bring their own API keys (e.g., per-user Claude or Codex keys). The workspace + lets each user reference their own provider credentials without exposing them + to teammates, eliminating the need for a separate gateway per user. + +- **Shared coding sessions.** Users within a workspace can share sandbox access + for pair programming or collaborative debugging without making sandboxes + globally visible. + +- **CI/CD and automation.** A workspace scopes service account credentials and + sandbox lifecycle to a specific pipeline or project, with limited lifetime + and no interactive access. + +- **Agent harness integration.** A single OpenShell gateway can be partitioned + into discrete workspaces so that multiple agent harness instances (e.g., + OpenClaw) can procure sandboxes from the same gateway with proper isolation. + This removes the requirement for a one-to-one association between an agent + harness instance and an OpenShell gateway instance. + +### Ownership and Access Control + +Every sandbox gets a `created_by` field in `ObjectMeta` populated from the +authenticated principal's subject. Access control is owner-scoped within +workspaces: + +- **Users** can only create, exec into, delete, or view sandboxes they own + within their workspace. +- **Workspace Admins** can manage any sandbox in their workspace. +- **Platform Admins** can manage everything. +- **Auditors** can view everything (read-only). +- `ListSandboxes` filters to owned sandboxes within the caller's workspace by + default; Workspace Admins see all sandboxes in their workspace; Platform + Admins and Auditors can list across workspaces. + +A User can share a sandbox with another user within the same workspace via an +explicit `ShareSandbox` RPC. The sandbox maintains a per-sandbox ACL of +principals with an access level (read-only or exec). Shared sandboxes appear in +a `shared-with-me` listing filter without making them globally visible. +Transitive provider access is not granted — the sharee uses the sandbox but does +not gain direct access to the workspace's provider credentials. Platform Admins +can grant targeted cross-workspace access for specific use cases (e.g., a shared +services workspace). + +### Policy Assignment + +Platform Admins set gateway-wide default policies. Workspace Admins can tighten +(but not loosen) policies for their workspace. Users and application workloads +cannot modify or override policies. This ensures a minimum security baseline +that holds even for agentic applications — app developers operate within +policy constraints set by their Workspace Admin, and compromised workloads +cannot escalate beyond them. + +### Authorization Boundaries (Kubernetes Deployments) + +In Kubernetes deployments, authorization operates at two layers. Kubernetes +RBAC governs control-plane operations: who can create, delete, or manage +sandbox custom resources and other objects in a Kubernetes namespace. The +gateway's five-role model governs data-plane operations: who can exec into a +running sandbox, stream relay output, share a session, or view audit logs. +These are runtime authorization decisions through the gateway's gRPC endpoints +where Kubernetes RBAC has no reach. + +Both layers are needed in Kubernetes deployments with clear boundaries between +them. For non-Kubernetes drivers (Docker, Podman, VM), the gateway's role model +is the sole authorization mechanism. + +### Provider Credential Scoping + +Providers belong to a workspace. A User can only attach providers available in +their workspace when creating a sandbox. No role — including service accounts — +sees raw credential material through the API; all roles reference providers by +name. The sandbox supervisor resolves credentials at runtime through an internal +trusted path, not through role-level permissions. Workspace Admins grant +specific provider credentials to users or service accounts within their +workspace. + +### Authentication + +- **Multi-provider OIDC.** Support multiple OIDC providers (corporate SSO, + GitHub, Google) mapped to internal identities. The authenticator chain already + supports this; the gap is identity federation and mapping to the expanded role + model. +- **API key authentication.** For service accounts and CI/CD. Long-lived keys + scoped to a workspace and role, stored hashed, rotatable. +- **mTLS for service-to-service.** Already partially supported via + `MtlsAuthConfig`. Extend to map cert OU/CN to roles. + +### Audit Trail + +- **Control-plane audit log.** Every mutating gRPC call (`CreateSandbox`, + `DeleteSandbox`, `CreateProvider`, `UpdatePolicy`) emits an OCSF + `ConfigStateChange` or `ApiActivity` event with the authenticated principal, + action, target resource, and timestamp. Builds on the existing OCSF + infrastructure. +- **Session attribution.** Sandbox activity (network, process, SSH events) + tagged with the creating principal's subject, so security teams can trace + sandbox behavior back to a human or service account. +- **Audit log export.** Structured OCSF JSONL shipped to SIEM/log aggregation. + Auditors can query "who created sandbox X" or "what did user Y do between T1 + and T2." + +### Resource Governance + +- **Per-workspace quotas.** Max concurrent sandboxes, max GPU allocations, max + sandbox lifetime per workspace. Enforced at the gateway before sandbox + creation. Quotas are framed as DoS and abuse protection for the control and + data plane, not as a chargeback mechanism. Quota limits are properties on the + workspace data model; detailed schema design is deferred to implementation. + +### Kubernetes Compute Driver: Workspace Mapping + +OpenShell workspaces are a logical concept. When the Kubernetes compute driver +renders sandboxes onto a cluster, it must map each OpenShell workspace to a +Kubernetes namespace. The driver supports two modes, configured per deployment: + +**Managed mode** (default) — the driver creates and deletes Kubernetes +namespaces on demand. The Kubernetes namespace name is derived from the gateway +identifier and the OpenShell workspace: +`openshell-{gateway-id}-{workspace-name}`. For example, if the gateway +identifier is `prod` and the OpenShell workspace is `team-ml`, the Kubernetes +namespace is `openshell-prod-team-ml`. + +The gateway identifier prefix ensures that multiple gateways can operate on a +common Kubernetes cluster without namespace collisions. Each gateway owns its +own set of Kubernetes namespaces and can independently create, watch, and delete +them. The gateway identifier is already part of the gateway's bootstrap +configuration. + +When an OpenShell workspace is deleted, the driver deletes the corresponding +Kubernetes namespace after draining all sandboxes. + +Managed mode requires a `ClusterRole` with namespace create/delete permissions. +The Helm chart includes conditional `ClusterRole` and `ClusterRoleBinding` +templates that are enabled by default. + +**Operator mode** — an alternative for environments where the gateway should not +create Kubernetes namespaces. The OpenShell workspace name maps one-to-one to a +Kubernetes namespace of the same name. If a sandbox belongs to OpenShell +workspace `team-ml`, the driver renders it into the Kubernetes namespace +`team-ml`. No mapping configuration is required. The Kubernetes namespaces must +be pre-provisioned — the driver has no permission to create or delete them. + +This direct identity mapping enables the OpenShell gateway to operate as a +natural Kubernetes-style operator: it receives a desired state (sandbox in +workspace X) and renders it into the corresponding cluster namespace. Platform +teams manage Kubernetes namespaces through their existing tooling (kubectl, +GitOps, Terraform) and OpenShell follows. + +```toml +[openshell.drivers.kubernetes] +workspace_mode = "operator" # opt-in; default is "managed" +``` + +**Watcher strategy.** Today the Kubernetes driver watches a single Kubernetes +namespace via `Api::namespaced_with()`. With multiple workspaces, the driver +shifts to a cluster-wide list/watch filtered by OpenShell labels (e.g., +`openshell.dev/managed-by=gateway`). This follows the standard Kubernetes +operator pattern for multi-namespace controllers. A per-namespace watcher +approach does not scale — it requires O(n) API connections and complicates +dynamic workspace addition/removal. The cluster-wide watch requires a +`ClusterRole` granting list/watch across Kubernetes namespaces (applicable to +both operator and managed modes). + +**Docker and Podman drivers.** The Docker driver's existing `sandbox_namespace` +label naturally maps to the OpenShell workspace value with no additional API +concepts to manage. The same applies to Podman and other local drivers — +workspace isolation is enforced at the gateway level and does not require +Kubernetes. This enables lightweight multi-workspace deployments on a single +host (e.g., rootless Podman with one gateway serving multiple workspaces). + +### Enterprise Deployment: Multi-Consumer Gateway + +A common enterprise deployment pattern — particularly in regulated industries +like financial services and defense — involves one or two data centers, each +running a handful of Kubernetes clusters. In this environment, organizations +want to minimize the number of OpenShell gateways they need to reason about. +Workspaces enable a single gateway per compute region to serve multiple +independent sandbox consumers with proper isolation between them. + +**Multiple consumers, one gateway.** In a single enterprise OpenShell +deployment, many independent consumers procure sandboxes on demand — agent +harnesses, CI/CD pipelines, internal tooling, and interactive users. Using +OpenClaw as an example agent-harness consumer that needs to procure sandboxes +on demand, an OpenClaw instance adds a single `workspace` field to its +OpenShell plugin config: + +```json5 +// OpenClaw plugin config — workspace scopes all sandbox operations +{ + plugins: { + entries: { + openshell: { + enabled: true, + config: { + from: "openclaw", + mode: "remote", + gateway: "prod", + gatewayEndpoint: "https://openshell.internal:8443", + workspace: "team-capital-markets", + }, + }, + }, + }, +} +``` + +The plugin passes `--workspace` to every `openshell` CLI invocation (`sandbox +get`, `sandbox create`, `sandbox list`). The rest of the OpenClaw integration +— sandbox lifecycle, SSH transport, workspace sync — is unchanged. + +OpenClaw sandboxes for tool execution and agent sessions are provisioned within +the assigned workspace. Other consumers — a different OpenClaw instance for +another department, a separate agent harness, or a CI pipeline — each operate +in their own workspace on the same gateway. Sandbox list operations are +workspace-scoped: a consumer sees only its own sandboxes, never sandboxes +belonging to other consumers. + +This is not OpenShell absorbing multiplayer concerns from its consumers. +OpenClaw and other agent harnesses own their own multi-user models. The +requirement on the OpenShell side is narrower: a single gateway must support +1:N partitioning so that each consumer's sandboxes are properly isolated from +every other consumer's sandboxes, without requiring a dedicated gateway per +consumer. + +**Kubernetes-level isolation chain.** When the gateway renders sandboxes onto +the cluster, each workspace maps to a discrete Kubernetes namespace. This +activates the full Kubernetes isolation stack: + +- **Network policy** partitions traffic between Kubernetes namespaces, preventing + cross-workspace network access between sandboxes. +- **UID/GID range allocation** (as enforced on platforms like OpenShift) assigns + each Kubernetes namespace a unique UID/GID range. Every sandbox process runs + under a UID that is unique to its workspace's namespace. +- **SELinux labeling** (on OpenShift and similarly configured platforms) assigns + each Kubernetes namespace a unique SELinux label (MCS category). Kernel-level + mandatory access control prevents processes in one namespace from reading + files, secrets, or process state belonging to another. + +**Sandbox escape threat model.** Container breakout is the dominant concern in +regulated environments. The workspace-to-Kubernetes-namespace mapping means +that even in the event of a sandbox escape, the attacker's process carries the +UID/GID and SELinux label of the originating workspace's Kubernetes namespace. +On platforms that enforce these boundaries at the node level, a compromised +process cannot: + +- Read Kubernetes secrets belonging to other workspaces. +- Enter or attach to sandboxes belonging to other consumers. +- Access files or process state outside its own UID/GID and SELinux domain. + +The combination of gateway-level workspace isolation (control plane) and +Kubernetes namespace isolation (data plane) produces defense in depth: even if +one layer is compromised, the other constrains the blast radius to a single +workspace. + +### Service Account Workflows + +- **CI/CD sandbox creation.** A service account creates sandboxes on behalf of a + pipeline within its workspace, labeled for the target project, with limited + lifetime and no interactive access. +- **Agent orchestration.** One agent's service account creates sandboxes for + sub-agents, each getting their own sandbox principal. The parent service + account retains visibility. + +## Implementation plan + +The implementation builds on the existing authentication, RBAC, and OCSF +foundations. The work can be phased to deliver value incrementally: + +- **Phase 1: Workspace and ownership model.** Add `workspace` and `created_by` + fields to `ObjectMeta` in the proto. Implement workspace-scoped storage and + filtering in gRPC handlers. Create the `default` workspace for backwards + compatibility. Sandbox name uniqueness shifts from globally unique to + unique-within-workspace. Existing sandboxes are backfilled into the `default` + workspace. Backward compatibility is desirable but not a hard requirement + at this stage — existing users may need to recreate resources when upgrading. + +- **Phase 2: Kubernetes driver — managed mode (default).** The driver creates + Kubernetes namespaces on demand using the naming convention + `openshell-{gateway-id}-{workspace-name}`. The watcher shifts from + single-namespace `Api::namespaced_with()` to cluster-wide list/watch with + OpenShell label filtering. Workspace cascade delete drains sandboxes before + removing the Kubernetes namespace. Helm chart adds `ClusterRole` and + `ClusterRoleBinding` for namespace create/delete and multi-namespace + list/watch permissions (enabled by default). Includes idempotent create with + retry to handle races. + +- **Phase 3: Kubernetes driver — operator mode.** Alternative mode where the + OpenShell workspace name maps one-to-one to a pre-existing Kubernetes + namespace. The driver accepts per-sandbox workspaces from the gateway + (populated via `driver_sandbox_from_public()`) and renders sandboxes into the + corresponding Kubernetes namespace. No namespace create/delete permissions + required. Opt-in via `workspace_mode = "operator"` in the driver config. + +- **Phase 4: Expanded role model.** Extend the RBAC system from two-tier + (admin/user) to five roles. Implement owner-scoped access guards in gRPC + handlers. Add Workspace Admin role with per-workspace management capabilities. + Add Auditor role with read-only cross-workspace access. + +- **Phase 5: Provider credential scoping.** Scope provider resources to + workspaces. Add credential delegation from Workspace Admins to users and + service accounts. Enforce provider visibility restrictions in sandbox creation. + +- **Phase 6: API key authentication and service accounts.** Implement API key + authenticator with hashed key storage, workspace-scoped keys, and rotation. + Add service account principal type with explicit grants. + +- **Phase 7: Audit trail enhancements.** Add `ApiActivity` OCSF event type for + control-plane mutations. Tag all sandbox activity events with the creating + principal's subject. Extend OCSF JSONL export with attribution fields. + Depends only on Phase 1 (`created_by` attribution) and can be pursued as a + separate effort — meaningful audit trails are only achievable once every + resource carries principal attribution. + +- **Phase 8: Collaboration.** Implement sandbox sharing within workspaces. + Add cross-workspace access grants for Platform Admins. Implement label-scoped + sandbox listing. + +- **Phase 9: Quota enforcement.** Implement per-workspace quota checks at the + gateway. Add quota configuration surface for Platform Admins and Workspace + Admins. + +Each phase is independently shippable and testable. Phase 1 is the prerequisite +for all subsequent phases but does not require any of them. Phases 2-9 can be +reordered based on priority. + +## Risks + +- **Migration complexity.** Existing deployments have no workspace concept. The + `default` workspace provides backwards compatibility, but platform teams with + established workflows may need to re-organize resources when adopting + workspaces. Migration tooling and documentation will be needed. + +- **Proto surface growth.** Adding `workspace`, `created_by`, and role-related + fields to the proto increases the API surface that must be maintained across + versions. The design intentionally keeps the new proto fields minimal + (workspace + created_by) and uses labels for soft grouping to limit this. + +- **RBAC complexity.** Five roles with workspace scoping is significantly more + complex than the current two-tier model. Misconfiguration could lead to + privilege escalation or overly restrictive access. Clear defaults, validation, + and documentation are essential. + +- **Performance at scale.** Workspace-scoped filtering and quota enforcement add + per-request overhead. For deployments with many workspaces and users, the + filtering and quota checks must be efficient. Indexing strategies need + consideration during implementation. + +- **Quota enforcement races.** Concurrent sandbox creation within a workspace + could race against quota limits. The quota check and sandbox creation must be + atomic or use optimistic concurrency control with retry. + +- **Kubernetes ClusterRole requirements.** Both operator and managed modes require + a `ClusterRole` for cluster-wide list/watch. Managed mode additionally + requires namespace create/delete permissions. Some clusters restrict these + grants. The Helm chart must make these conditional and clearly documented. + +- **Managed mode race conditions.** Kubernetes namespace creation is async. + Sandbox creation may race against it. The naming convention + (`openshell-{gateway-id}-{workspace-name}`) is deterministic, so concurrent + creates from the same gateway are idempotent. + +- **In-flight sandboxes during workspace deletion.** Deleting a workspace with + active sandboxes requires a decision: block deletion until drained (safe, + explicit), cascade delete all sandboxes, or mark as deleting and drain + gracefully. The initial implementation should block deletion if active + sandboxes exist. + +- **Multi-gateway coordination.** The `openshell-{gateway-id}-{workspace-name}` + naming convention partitions Kubernetes namespaces by gateway, so multiple + gateways can share a cluster without collisions. However, this means each + gateway manages its own workspace set independently — cross-gateway workspace + visibility requires external coordination. + +## Alternatives + +### Flat label-based tenancy (no workspaces) + +Use labels alone for all isolation, without a first-class workspace concept. +Users would filter by label, and access control would use label selectors. + +This was rejected because labels are a soft grouping mechanism with no +enforcement guarantee. A mislabeled resource would be visible across tenant +boundaries. Hard isolation requires a first-class field that the system enforces +at every access point, not a convention that depends on correct labeling. + +### One gateway per team + +Instead of multi-tenancy, deploy separate gateways per team. This provides +complete isolation by default. + +This was rejected because it creates operational overhead (N gateways to +manage), prevents resource sharing across teams, and makes cross-team +collaboration impossible. It also pushes the multi-tenancy problem to the +infrastructure layer without solving it. In practice, even within a single +team, individual members typically have private per-user API keys for services +like Claude or Codex that they cannot share with teammates. This pushes +team-level deployments toward per-user gateways, compounding the operational +cost. The multi-player proposal mitigates this by letting team members in a +shared department use per-user credentials within a common gateway deployment, +collaborate on sandboxes where appropriate, and avoid multiplying platform +infrastructure. + +### OPA/Rego for authorization + +Use a policy language like OPA/Rego for fine-grained authorization decisions +instead of role-based access control. + +This was considered but deferred. The current need is coarse-grained role-based +isolation, not attribute-based policy evaluation. OPA/Rego authorization could +be layered on top of the workspace and role model in a future RFC if +fine-grained policies are needed. + +## Prior art + +- **Kubernetes namespaces and RBAC.** The workspace model draws from Kubernetes + conventions: hard isolation boundaries, labels for soft grouping, and RBAC + with role bindings scoped to boundaries. The term "workspace" is intentionally + distinct from "Kubernetes namespace" to avoid conflation — an OpenShell + workspace may map to a Kubernetes namespace, but the concepts are independent. + +- **GitHub organizations and teams.** GitHub's model of organizations + (workspaces) with teams (label-based grouping) and per-repo role assignments + informed the separation between hard boundaries and soft grouping. + +- **AWS IAM.** AWS's account-level isolation with IAM roles and policies within + accounts informed the quota and credential scoping model. The lesson is that + hard account boundaries with delegated administration scales better than + flat permission models. + +## Open questions + +- Should workspace creation be self-service (any authenticated user can create + a workspace) or admin-only (Platform Admins create workspaces and assign + Workspace Admins)? + +- What is the identity mapping strategy for multi-provider OIDC? If a user + authenticates via both corporate SSO and GitHub, how are those identities + linked to a single internal principal? For the initial implementation, each + OIDC identity is treated as a separate principal — identity federation can + be added later. + +- Should per-workspace quota limits be hard (reject sandbox creation) or soft + (warn but allow, with alerting)? + +- How should sandbox sharing permissions interact with policy tightening? If a + Workspace Admin tightens a policy, does it retroactively affect shared + sandboxes that were created under the looser policy? + +- What is the storage backend for API keys and quota state? The provider + credential storage drivers (#1986) introduce gateway-owned credential + persistence with encrypted database, Kubernetes Secrets, and Vault backends. + API key and quota state could build on this infrastructure — the remaining + question is what those records should look like. + +- In operator mode, should the driver validate that the target Kubernetes + namespace exists before accepting a sandbox create, or should it let the + Kubernetes API reject the request and surface the error?