docs: add Braintrust observability guides

[jstar0]

Summary

• Add Braintrust integration guides for standalone and Kubernetes deployments in the main and latest documentation trees.
• Document US, EU, and self-hosted OTLP/HTTP endpoints, x-bt-parent routing, TLS, and Secret-backed authentication.
• Include optional prompt/response capture with data-handling guidance, verification steps, troubleshooting, and navigation updates.
• Use the shared documentation snippets for Kubernetes resource names and namespace values.

Part of #3230.

Verification

• Rebased onto current main at 84943b3a.
• git diff --check
• uv run python scripts/list_untested_docs.py --docs-dir content/docs (all four Braintrust pages appear in the untested-doc inventory)
• uv run python scripts/check_generated_asset_pins.py
• gitleaks scan over the contribution commits

The local checkout does not include the repository Hugo binary; the hosted documentation workflow will provide the Hugo build and rendered-page checks.

[kristin-kronstain-brown]

Thanks for this — and welcome! This is a strong first contribution. The tricky part of a tracing integration is getting the API shape right, and you did.

I validated the Kubernetes config on a live cluster (kind, agentgateway v1.5.0 via Helm, Gateway API 1.6.0) rather than just reading it. Results:

Check	Result
AgentgatewayBackend accepted	Accepted=True
AgentgatewayPolicy accepted + attached	Accepted=True, Attached=True
Credential headers injected on the OTLP export	Yes — authorization: Bearer … and x-bt-parent: … both arrive
Export path	POST /otel/v1/traces, content-type: application/x-protobuf
flattenRecursive(…) attributes block	Accepted by the controller's CEL validation
resources: service.name	Lands in the exported span payload
API key in the proxy admin /config_dump	Not exposed — serialized as "key": "<redacted>"

I pointed the tracing backendRef at an in-cluster OTLP sink that logs request headers, and confirmed the Secret-backed credentials really do get injected on the export request. So the core approach in this PR works.

The security-relevant choices here are also the right ones: tls: {} and backendTLS: {} verify against the system CAs with no insecureSkipVerify anywhere, the Kubernetes path keeps the key in a Secret rather than inline, and the prompt/response capture is opt-in behind an explicit data-handling warning. That last one is the highest-risk thing in the guide and you handled it well.

Worth noting: our existing Honeycomb page still claims "the agentgateway tracing policy does not support custom HTTP headers, you must route traces through an OTel Collector." Your PR demonstrates that's now stale. That change could be added to this PR.

---

Required changes

1. The "Kubernetes tracing" link points at the wrong page.

{{< link-hextra path="/observability/tracing/" >}} is now only an alias that redirects to the Observability section index — the observability/traces/ restructure landed on main after you branched. It won't 404, so it's easy to miss, but the reader loses the destination. Please use:

{{< link-hextra path="/observability/traces/setup/" >}}

You'll want to rebase on main first; the PR currently shows as behind. And FYI - we have some big organizational changes coming soon, so depending on how quickly we get this merge, don't be surprised if we need to do some more conflict handling before getting it merged.

2. "reduce the exporter batch size" isn't an available setting.

Neither the Kubernetes frontend.tracing API nor the standalone frontendPolicies.tracing schema has a batch-size field. Suggested rewrite:

Braintrust limits a single OTLP trace request to 10 MB. The tracing policy has no batch size setting, so if the exporter reports HTTP 413, drop the message content attributes or lower randomSampling so that fewer spans are exported.

3. Port-forward and listener port contradict each other.

The Kubernetes page port-forwards 8080:80, then says the example "assumes … a listener on port 8080." The listener is on port 80; 8080 is the local forwarded port. Suggested:

For local testing, port-forward the agentgateway proxy. The proxy listens on port 80 in the cluster and is reachable at localhost:8080 while the port-forward runs.

4. Use the shared snippets instead of hardcoded values.

The Kubernetes pages hardcode agentgateway-system, AgentgatewayPolicy, AgentgatewayBackend, and agentgateway.dev/v1alpha1. These exist as reuse snippets so they can be swapped in one place:

{{< reuse "agw-docs/snippets/namespace.md" >}}
{{< reuse "agw-docs/snippets/policy.md" >}}
{{< reuse "agw-docs/snippets/backend.md" >}}
{{< reuse "agw-docs/snippets/api-version.md" >}}

They work inside fenced code blocks — see observability/traces/configs/honeycomb.md for the pattern.

5. Drop test: skip from the four new pages.

This one is counterintuitive, so it's worth explaining. Per scripts/list_untested_docs.py, a page counts as covered if it has a test: key — including test: skip. So test: skip doesn't just mean "no tests," it removes the page from the untested-docs backlog entirely. Since these guides genuinely aren't tested yet, we'd rather they show up as candidates for coverage.

Removing the key changes nothing about what runs (zero test cases either way). It also matches the neighbors: langfuse.md, langsmith.md, and phoenix.md in this same directory all have no test: key.

---

Suggestions

• service.name: '"agentgateway"' is a no-op — that's already the default. Either use a distinguishing value like '"agentgateway-llm"', or drop the resources block.

• The root-span paragraph doesn't give the reader an action. "Export the root span for each request; sending only child spans does not create a row in the Logs view" reads like Braintrust SDK guidance — there's no agentgateway setting that controls it. The caveat that does apply here is that custom attributes don't reach policy-call child spans:

  [!NOTE]
  Custom span attributes are not applied to policy call child spans. For more information, see [Policy call child spans]({{< link-hextra path="/observability/traces/attribute-reference/#policy-child-spans" >}}).

• Consider a field-description table after each config block, matching observability/traces/setup.md. Most valuable for protocol/path (Braintrust is HTTP-only, and path is only valid with protocol: HTTP) and for randomSampling.

• A ## Cleanup section would match the other Kubernetes tracing pages — {{< reuse "agw-docs/snippets/cleanup.md" >}} plus the delete commands.

• Worth adding to standalone troubleshooting: the whole config file is shell-expanded before parsing, so an unset BRAINTRUST_API_KEY or BRAINTRUST_PARENT stops the parse rather than producing an empty header. That's a sharp edge readers will hit.

• A note on what leaves the proxy. The page carefully offers a US / EU / self-hosted choice, which tells me you were thinking about data residency — so it's worth naming what goes out by default. Spans include src.addr (client IP), http.path (which often carries IDs or tokens in query strings), and, when JWT authentication is enabled, jwt.sub. The remove field drops them, and readers subject to a data residency policy will want to know it exists:

  [!NOTE]
  Spans include src.addr (client IP) and, when JWT authentication is enabled, jwt.sub. Use the remove field to drop attributes before export if your data handling policy restricts sending identifiers to a third party.

• Optional, on the Bearer prefix: baking it into the Secret value works. You could also store the bare key and set location.header.prefix on the credential — either is fine, but a one-line note on why the prefix is in the Secret would help.

• standalone/.../llm-observability/_index.md: the rewrite drops the "prompt logging, cost tracking, and audit trail" framing and omits Arize Phoenix, which is in that section. Consider just adding Braintrust to the existing sentence, the way you did for the Kubernetes one.

---

Not blocking

The x-bt-parent value, the /otel/v1/traces path, and the US/EU host split are all Braintrust-side details I couldn't verify without an account — I checked them against Braintrust's OpenTelemetry docs and they look right, but you're closer to that than I am.

Thanks again for picking this up!

[jstar0]

Addressed the requested changes in d09d891:

• merged current main, including the latest observability docs restructure;
• updated both Kubernetes pages to link to /observability/traces/setup/, use the shared namespace/API/kind snippets, and describe the 8080:80 port-forward correctly;
• removed test: skip from the four Braintrust pages so they remain in the untested-doc inventory;
• replaced the nonexistent batch-size advice with the supported 10 MB guidance (omit message attributes or lower randomSampling);
• kept the Axiom and Braintrust entries together in the shared index descriptions.

git diff --check passes, the branch is mergeable, and DCO is green. The remaining hosted workflows are awaiting repository approval. Please re-review when convenient.

[kristin-kronstain-brown]

Looking better!

One thing left. The snippets are used in prose as well as in code blocks. Line 54 on both Kubernetes pages still has the literals:

Create an {{< reuse "agw-docs/snippets/backend.md" >}} that enables TLS and reads the Braintrust headers from the Secret. Then attach an {{< reuse "agw-docs/snippets/policy.md" >}} to the Gateway that serves your LLM route.