fix(otel): preserve all exemplars per data point (#1662)

[prabhaks]

What

Fixes #1662.

insert_exemplars in src/otel/metrics.rs wrote every exemplar of a data point to the same static columns (exemplar_time_unix_nano, exemplar_span_id, exemplar_trace_id, exemplar_value). Since the keys were shared, each exemplar overwrote the previous one and only the last survived. This affected Sum, Histogram and ExponentialHistogram (and Gauge, which shares the same NumberDataPoint path).

Approach

Following the recommended option 1 in the issue, all exemplars for a data point are now serialized into a single nested exemplars array column. Each element is an object with exemplar_time_unix_nano, exemplar_span_id, exemplar_trace_id, exemplar_value and any filtered attributes. This matches the OTel data model (exemplars are a repeated field per data point) and keeps the schema stable regardless of exemplar count.

Backward compatibility

Added a P_OTEL_FLATTEN_EXEMPLARS flag (default false). When set to true the previous flat exemplar_* columns are emitted (last exemplar wins, same as before) for anyone already querying those columns. Per the discussion on the issue, the default is the new array behavior.

Bonus fix

Previously exemplar attributes were written straight into the data point map, so they leaked into compute_series_hash and fragmented physical-series identity. exemplars is now part of OTEL_METRICS_KNOWN_FIELD_LIST, so exemplar contents no longer affect the series hash.

Query validation (PoC)

Before settling on the schema I ran a PoC against the same library versions used here (datafusion 53, arrow/arrow-json 58) to confirm the query layer handles the array-of-objects column. arrow-json infers it as List<Struct<...>>, and DataFusion handles reading the column, array_length, unnest, struct field access, and filtering on nested fields. Details are in the issue thread.

Testing

• cargo test: 311 passed, 0 failed.
• New unit tests cover: all exemplars preserved in the array, legacy flat mode keeps last, empty exemplars insert nothing, end to end through a Sum, exemplar contents do not affect the series hash, and the known-field list update.

Flag reference

P_OTEL_FLATTEN_EXEMPLARS env var / --otel-flatten-exemplars flag, default false (new nested exemplars array). Set to true for the legacy flat exemplar_* columns.

Summary by CodeRabbit

• New Features
  • Added configurable OTEL exemplar formatting via P_OTEL_FLATTEN_EXEMPLARS (--otel-flatten-exemplars).
  • Default behavior preserves exemplars in a nested exemplars array.
  • Option enables legacy flat exemplar columns (“last exemplar wins”).
• Bug Fixes
  • Preserves and correctly flattens exemplar data across metric flattening and end-to-end processing.
  • Exemplar variations no longer affect metric series identity/hashing.
• Tests
  • Added exemplar regression coverage for nested vs. legacy behavior, empty exemplars, and series-hash invariance.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 3865145e-46da-4c40-abc2-bf68bf0163ac

📥 Commits

Reviewing files that changed from the base of the PR and between 1e9eb32 and bf49759.

📒 Files selected for processing (2)

• src/cli.rs
• src/otel/metrics.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• src/cli.rs
• src/otel/metrics.rs

---

Walkthrough

OTEL metrics now support configurable exemplar storage: nested arrays preserve all exemplars by default, while P_OTEL_FLATTEN_EXEMPLARS enables legacy flattened columns. The mode flows through JSON and protobuf processing, with updated series identity handling and regression tests.

Changes

OTEL exemplar handling

Layer / File(s)	Summary
Configuration and series identity contract src/cli.rs, src/otel/metrics.rs	Adds the otel_flatten_exemplars option and registers exemplars as a known OTEL metric field.
Exemplar serialization and flattening src/otel/metrics.rs	Adds JSON conversion helpers and applies nested-array or legacy flattened exemplar output across gauge, sum, histogram, and exponential-histogram records.
Pipeline wiring and validation src/otel/metrics.rs	Propagates the option through JSON and protobuf processing and tests preservation, legacy overwrite behavior, empty exemplars, roundtripping, and series hashes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OTELClient
  participant flatten_otel_metrics
  participant process_resource_metrics
  participant flatten_metrics_record
  participant insert_exemplars

  OTELClient->>flatten_otel_metrics: OTEL metrics payload
  flatten_otel_metrics->>process_resource_metrics: configured exemplar mode
  process_resource_metrics->>flatten_metrics_record: metric record and mode
  flatten_metrics_record->>insert_exemplars: exemplar list and mode
  insert_exemplars-->>flatten_metrics_record: nested array or flattened columns
  flatten_metrics_record-->>process_resource_metrics: flattened records

Loading

Poem

I’m a rabbit with metrics to spare,
Nesting exemplars with careful care.
Or flatten them wide, old-style in a row,
While every little trace gets a place to show.
Hop, hop—the data stays whole!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the exemplar-preservation fix and matches the implemented OTEL behavior.
Description check	✅ Passed	The description covers the issue link, problem, chosen approach, compatibility, testing, and flag reference, so it is mostly complete.
Linked Issues check	✅ Passed	The code preserves all exemplars in a nested array by default and keeps legacy flattening behind a flag, satisfying #1662.
Out of Scope Changes check	✅ Passed	The new flag and series-hash adjustment are directly tied to exemplar handling, with no clearly unrelated changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[prabhaks]

Note on the failing coverage check: it fails at the clippy step because of pre-existing lints on main from the newer Rust/clippy 1.97, not because of this PR (none of the flagged files are touched here). I opened #1721 to fix those lints and unblock CI. Once #1721 is merged I will rebase this branch on top so its checks go green.

[nikhilsinhaparseable]

@prabhaks thank you for raising the lint PR, i have verified and merged it, you can rebase this PR now

[prabhaks]

@prabhaks thank you for raising the lint PR, i have verified and merged it, you can rebase this PR now

Done, thank you for reviewing and merging my other PR!

[prabhaks]

@nikhilsinhaparseable @parmesant please take a look at this PR whenever you get a chance!