Add pluggable storage backends (SQLite/Redis/Postgres+Redis hybrid) (ar-r82f.3)

[atc964]

Summary

Adds a pluggable StorageBackend abstraction with four concrete implementations — SQLite (default), Redis, PostgreSQL, and a Hybrid mode that routes durable data to Postgres and ephemeral data to Redis. This mirrors the seller-agent's storage architecture and gives the buyer agent a path off the single-writer SQLite constraint that today caps deployments at one ECS task.

The legacy DealStore is untouched and remains the synchronous SQLite path used by CrewAI agents. Both APIs coexist; callers can migrate to the pluggable backend incrementally.

Why

• Horizontal scaling: SQLite serializes writes, so production deployments are pinned to DesiredCount: 1. Hybrid mode lifts that.
• Native TTL, pub/sub, distributed locks: Sessions, caches, locks, and rate limits benefit from Redis primitives that SQLite can only emulate.
• Structural consistency with seller-agent: Same ABC shape (connect/disconnect/get/set/delete/exists/keys + domain helpers) and same factory pattern.

What's added

Storage code (src/ad_buyer/storage/):

• base.py — StorageBackend ABC and async domain helpers (deal, campaign, order, session, conversion, optimization decision, experiment, supply path score, quote, negotiation, model artifact, pacing snapshot).
• factory.py — get_storage_backend() selects SQLite / Redis / Hybrid from settings or explicit args; lazy-imports Redis/asyncpg so the base install stays lean.
• sqlite_backend.py — aiosqlite KV implementation with optional TTL.
• redis_backend.py — redis.asyncio KV with namespaced keys, plus publish / incr / get_stats helpers. Importable without Redis installed (raises a clear ImportError only on construction).
• postgres_backend.py — asyncpg JSONB KV with connection pooling (pool_min / pool_max).
• hybrid_backend.py — Prefix-based routing: session:, session_index:, cache:, lock:, pubsub:, rate_limit: go to Redis; everything else to Postgres.
• __init__.py — Merged exports: legacy DealStore + new StorageBackend / get_storage_backend.

Tests (tests/unit/test_storage_backend.py):

• 27 tests covering SQLite KV, domain helpers, factory selection (sqlite default + invalid type + missing-URL paths for redis/hybrid), Postgres backend init + URL normalization + not-connected error, hybrid routing logic, and Redis backend availability gating.

Config:

• New settings: storage_type, postgres_pool_min, postgres_pool_max. All existing settings preserved; defaults keep SQLite behavior unchanged.
• .env.example gains STORAGE_TYPE, POSTGRES_POOL_*, and hybrid usage examples.
• pyproject.toml adds optional extras [redis], [postgres], [production]. Base dependencies and the crewai==1.10.1 pin are preserved.

Docs:

• New docs/architecture/storage-backends.md with backend selection table, three-backend reference, routing rules, optional-extras install commands, configuration reference, and migration sketch.
• Cross-links added from deal-store.md, overview.md, configuration.md, deployment.md, deployment-ops-guide.md, and mkdocs.yml so the page is discoverable.

Optional extras pattern

pip install -e .                  # SQLite only (default)
pip install -e ".[redis]"         # Add Redis
pip install -e ".[postgres]"      # Add PostgreSQL
pip install -e ".[production]"    # Both

The factory lazy-imports redis and asyncpg, so they're never loaded unless the corresponding backend is selected.

Backwards compatibility

• No upstream behavior changes. Existing DealStore callers are unaffected.
• The factory is opt-in: leave STORAGE_TYPE unset (or set it to sqlite) to keep today's behavior.
• New optional extras don't affect the base install.

What's left for follow-on

Callers still use DealStore directly. Migrating other domain stores (campaigns, orders, sessions, conversions, optimization, experiments, pacing) to the pluggable backend is incremental and out of scope here. The ABC includes the helper methods those migrations will need.

Context

• Bead: ar-r82f.3 (parent epic ar-r82f).
• This is the pluggable-storage piece of a larger upstreaming track. Optimization stores, engines/services/constants scaffolding, and other fork-side divergences (settings removals, ML deps, MCP path changes, doc renames) are intentionally NOT in this PR.

Test plan

• CI green: full test suite passes (new test_storage_backend.py: 26 passed + 1 skipped locally; Redis test skips when redis package not installed).
• Import smoke test: python -c "from ad_buyer.storage import StorageBackend, get_storage_backend, DealStore" works in a fresh uv sync --extra dev env.
• Settings defaults preserved: existing settings.database_url, settings.redis_url, etc. unchanged; new settings.storage_type = "sqlite" by default.
• Optional extras: pip install -e ".[redis]" makes RedisBackend instantiable; pip install -e ".[postgres]" makes PostgresBackend connectable.
• Hybrid routing: durable keys (e.g. deal:X) land in Postgres; ephemeral keys (e.g. session:X) land in Redis (verified in test via dual-SQLite stand-ins).

🤖 Generated with Claude Code