llmtape

Record a real LLM call once. Replay it forever. Assert your agent called the right tool.

The problem

Your agent's tests hit the real LLM API. That's slow (seconds per test), costly (every CI run burns money), and flaky (the model doesn't always answer the same way twice). So you either keep paying for it, or you hand-roll a mock that drifts from what the real API actually returns the moment the response shape changes.

Recording and replaying the HTTP call solves the speed and cost problem. But it doesn't solve the problem that's actually under test: did the agent call search_creators with { niche: "beleza" }, or did it hallucinate a different tool, or call the right one with the wrong argument? A cassette full of raw JSON doesn't answer that on its own, you still end up hand-parsing response.content or response.choices[0].message.tool_calls in every test.

The fix

record() and replay() patch fetch, not a specific SDK, so it works with Anthropic, OpenAI, or anything else that talks HTTP, no per-provider adapter required.

import { record, replay, tape } from "llmtape";

// first run: hits the real API, saves __llmtape__/search-creators.json
await record("search-creators", () => runAgent("busca creators de beleza"));

// every run after: reads the fixture, network never touched
const result = await replay("search-creators", () => runAgent("busca creators de beleza"));

// or let it decide: record if the fixture doesn't exist yet, replay if it does
const result = await tape("search-creators", () => runAgent("busca creators de beleza"));

expectToolCall reads the tape and asserts on the thing your test actually cares about:

import { expectToolCall } from "llmtape";

const call = expectToolCall(result, "search_creators", { niche: "beleza" });
// throws with the full list of calls the agent actually made if it doesn't match

It plugs into whatever test runner you already use, Vitest, Bun test, Jest, there's no runner of its own:

import { test } from "vitest";
import { replay, expectToolCall } from "llmtape";

test("busca creator por nicho", async () => {
  const result = await replay("search-creators", () => runAgent("busca creators de beleza"));
  expectToolCall(result, "search_creators", { niche: "beleza" });
});

If the prompt or tool schema changes since the fixture was recorded, replay() throws instead of silently returning a stale response, telling you exactly which request stopped matching so you know to re-record.

See it work

bun install
bun run example   # records a real call, replays it with the server stopped, asserts the tool call
bun run bench      # how much faster a replay is than the real round-trip

Recording a real call...
Saved to __llmtape__/search-creators.json (api key redacted)

Server stopped. Replaying from the fixture, no network involved...
Got: { content: [ { type: "tool_use", name: "search_creators", input: { niche: "beleza" } } ] }

Agent called search_creators({"niche":"beleza"}) ✓

real API call: ~300ms
200 replays in 68.3ms (0.34ms/replay avg)
~879x faster than hitting the real API

Security

Authorization and x-api-key headers are always redacted before a fixture is written, an API key never ends up committed to git by accident. In CI (process.env.CI set), tape() refuses to fall back to record() when a fixture is missing, so a forgotten fixture fails loudly instead of quietly billing the real API.

Why this and not llm-vcr

llm-vcr solves the same record/replay problem and does more of the polish today, streaming/SSE, fuzzy matching for dynamic prompts, custom redaction patterns. What it doesn't do is help you assert on the agent's actual decision, the tool it called and the arguments it used, that's the part expectToolCall exists for. If you just need cheap, deterministic replay of an LLM call, llm-vcr already does that well. If the thing you're testing is "did my agent call the right tool," that's the gap this fills.

Scope today

• record(name, fn) / replay(name, fn) / tape(name, fn), HTTP-level, provider-agnostic, one fixture per name in __llmtape__/.
• expectToolCall(result, name, input?), parses Anthropic tool_use blocks, OpenAI tool_calls, and Gemini functionCall parts out of the tape.
• Drift detection: replay throws if the live request no longer hashes to what was recorded.

Not built yet: streaming/SSE responses, fuzzy matching for dynamic prompt content (timestamps, ids), providers beyond the Anthropic/OpenAI/Gemini response shapes. Add them if you hit the wall, today's shape covers a single-turn or multi-turn tool-calling test straight through.

Install

bun add llmtape