Services Control Plane

Control plane for managing hosted StreamingFast services on Kubernetes.

Project Structure

├── k8s/api/v1alpha1/      # CRD type definitions (HostedServiceDeployment)
├── cli/                   # TypeScript CLI for interacting with control plane
├── cmd/control-freak/     # Go gRPC server & CLI commands
├── store/deployment/      # Deployment data persistence (PostgreSQL)
├── dsn/                   # Database connection string builder
├── store/event/           # Deployment event tracking (PostgreSQL)
├── grpc/                  # gRPC service implementations
├── k8s/
│   ├── deployer/          # StatefulSet builders (sink-sql, foundational-store)
│   ├── operator/          # CRD operator/controller (reconcile loop)
│   ├── setup/             # Local development environment (START HERE!)
│   └── tests/             # Integration tests
├── pb/                    # Generated protobuf code
├── proto-internal/        # Internal proto definitions
├── registry/              # Network & foundation store registries (PostgreSQL)
└── vault/                 # Hashicorp Vault integration (secrets)

Quick Start (Local Development)

Want to test without touching production? Start here!

# One-time setup (creates local K8s cluster with all dependencies)
go run ./cmd/control-freak local setup

# Start port forwards (runs in foreground, Ctrl+C to stop)
go run ./cmd/control-freak local port-forward

# Run CLI (in another terminal)
cd cli
NODE_ENV=local bun run index.ts

You now have a complete local environment with:

• Local Kubernetes cluster (k3d/k3s)
• gRPC server running with CRD operator
• PostgreSQL for deployment data, events, and registries
• Hashicorp Vault for secret management

Full guide: k8s/setup/QUICKSTART.md Detailed docs: k8s/setup/README.md

Features

• Deploy: Create service deployments on Kubernetes via CRDs
• Undeploy: Remove deployments and clean up resources
• SetReplica: Scale deployments up or down
• UpdateDeploymentConfig: Update spkg, execution config, output config
• ResetDeployment: Drop schema/truncate and restart from scratch
• Logs: Stream logs from deployment pods
• Deployment State: Real-time monitoring via CRD operator status
• Deployment Events: Lifecycle event tracking in PostgreSQL
• Deployment Persistence: All deployment data stored in PostgreSQL deployments table
• Secret Management: Database passwords stored in Hashicorp Vault
• Organization Filtering: Multi-tenant access control via organization_id
• Admin Service: Separate gRPC service for CLI/admin portal operations
• Network Registry: Manage substreams network endpoints
• Foundation Store Registry: Manage foundational store deployments
• TUI: Terminal interface for deployment management

Architecture

┌─────────────┐      gRPC        ┌───────────────────┐
│     CLI     │◄────────────────►│  control-freak    │
│ (TypeScript)│                  │  (Go gRPC server) │
└─────────────┘                  └────────┬──────────┘
                                          │
                          ┌───────────────┼───────────────┐
                          │               │               │
                          ▼               ▼               ▼
                   ┌────────────┐  ┌────────────┐  ┌──────────┐
                   │ Kubernetes │  │ PostgreSQL │  │  Vault   │
                   │   CRDs     │  │            │  │          │
                   └─────┬──────┘  └────────────┘  └──────────┘
                         │           deployments     secrets
                         │           events
                    ┌────┴──────┐    registries
                    │ Operator  │
                    │(reconcile)│
                    └────┬──────┘
                         │
                         ▼
                  ┌──────────────┐
                  │ StatefulSets │
                  │    + Pods    │
                  └──────────────┘

Data flow:

1. CLI sends gRPC request to control-freak
2. control-freak creates/updates HostedServiceDeployment CRD
3. control-freak persists deployment data to PostgreSQL deployments table (fire-and-forget)
4. CRD operator reconciles CRD → StatefulSet + Service
5. Operator updates CRD status with live pod state
6. Secrets stored in Vault, injected via Vault Agent sidecar

Development

Make server changes

# Edit Go code, then rebuild and restart in cluster:
go run ./cmd/control-freak local restart

Make CLI changes

# Edit TypeScript code, then:
cd cli
NODE_ENV=local bun run index.ts

Run tests

# Unit tests only
go test $(go list ./... | grep -v k8s/tests) -v -count 1

# Integration tests (requires k3d cluster + SUBSTREAMS_API_KEY)
KEEP_CLUSTER=true SUBSTREAMS_API_KEY=<key> go test ./k8s/tests/... -v -count 1

# All tests
KEEP_CLUSTER=true SUBSTREAMS_API_KEY=<key> go test ./... -v -count 1

# Specific test
go test ./k8s/tests/... -run TestDeploymentStore -v

Teardown local environment

go run ./cmd/control-freak local teardown

Production Deployment

# Build and run gRPC server
go build -o control-freak ./cmd/control-freak
./control-freak serve --postgres-dsn "postgres://..." --vault-addr "http://vault:8200"

# Run CLI
cd cli
bun run index.ts

Environment Variables

Variable	Default	Description
NODE_ENV	production	local for dev cluster, production for prod
GRPC_HOST	Varies by env	gRPC server address
POSTGRES_DSN	Required	PostgreSQL connection string
VAULT_ADDR	Optional	Hashicorp Vault address
VAULT_TOKEN	Optional	Vault authentication token
NAMESPACE	default	Kubernetes namespace for deployments
ORGANIZATION_ID	Required (CLI)	Organization ID for multi-tenant filtering
SUBSTREAMS_API_KEY	Required (tests)	API key for substreams integration tests

PostgreSQL Tables

Table	Purpose
deployments	All deployment configuration and metadata
deployment_events	Lifecycle events (created, deleted, scaled, etc.)
foundation_store_registry	Foundational store deployment registry
network_endpoints	Network endpoint registry
network_aliases	Network name → endpoint mappings

License

See LICENSE file.