Add OpenTelemetry tracing across the backbeat pipeline

[delthas]

Summary

Add OpenTelemetry tracing across the backbeat pipeline, gated behind ENABLE_OTEL=true. When the flag is unset, no @opentelemetry/* package is loaded — zero overhead off the OTEL path.

The SDK bootstrap, trust-boundary host filter, and Kafka trace-context helpers now live in arsenal's shared lib/tracing module (scality/Arsenal#2632, ARSN-586); backbeat consumes it through a thin shim instead of carrying its own copy. Companion to the cloudserver (#6140, CLDSRV-884) and vault (#203, VAULT-708) PRs, so all four services share one implementation.

Commits

1. chore: depend on arsenal OTEL tracing module — pin arsenal at the 8.4.6 release tag (shared tracing module + the W3C trace-context stamping on MongoDB metadata writes, ARSN-572, which the Kafka pipeline relies on to continue traces across the oplog boundary). Drop the SDK-core packages now that arsenal carries them as optionalDependencies, and keep the four instrumentation packages backbeat configures itself: instrumentation-http / -ioredis / -mongodb / -aws-sdk.

2. feat: replace in-tree tracing with arsenal shim — lib/tracing/index.js becomes a thin shim over require('arsenal/build/lib/tracing') carrying backbeat's config in one place: serviceName: 'backbeat', the four instrumentations, and outbound-only HTTP (...makeHttpInstrumentationConfig() for the trust-boundary requestHook, plus disableIncomingRequestInstrumentation: true since backbeat pods serve no application HTTP). lib/tracing/kafkaTraceContext.js re-exports arsenal's kafka helpers so the existing require sites are unchanged. The trust-boundary filter and SDK bootstrap that used to live here are now arsenal's.

3. feat: instrument backbeat pods and the Kafka pipeline — wire tracing into the replication, lifecycle, GC, notification, and oplog-populator pods: init() at each of the 8 entry points, per-pod spans, and trace-context propagation across the Kafka pipeline. Producers stamp traceparent onto message headers via the kafka helpers; consumers start a span linked to (not a child of) the upstream span — out-of-process Kafka hops can fire long after the original request, so links keep traces bounded.

Incidental, intentional: the replicationStatusProcessor SIGTERM handler previously had no process.exit(0) on the success path — an inconsistency with the other 7 entry points. Since this PR adds tracing.close() to that handler, it also adds the missing process.exit(0) so the pod exits cleanly on SIGTERM like the rest (rather than potentially hanging on a non-empty event loop).

Why a shim (vs cloudserver/vault's direct calls)

backbeat has 8 init() entry points and 6 Kafka-helper require sites. The shim keeps all 14 call sites untouched and the backbeat-specific config in one file; cloudserver and vault have a single entry point each, so they deep-require arsenal directly.

Configuration

OpenTelemetry environment variables are documented in the arsenal module.

Out of scope (follow-ups)

The trust-boundary enforcement (read OTEL_TRUSTED_HOSTS, strip traceparent on untrusted outbound) lives entirely in arsenal and is wired automatically. Populating that env var per deployment is routine operator config — no backbeat-side work.

Related tickets

• BB-764 — this ticket
• OS-1072 — parent investigation (OTEL across Scality services)
• ARSN-586 — shared arsenal tracing module (scality/Arsenal#2632)
• CLDSRV-884 / VAULT-708 — sibling cloudserver / vault OTEL PRs

Issue: BB-764

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

[codecov]

Codecov Report

❌ Patch coverage is 54.02299% with 40 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.65%. Comparing base (c52fbcc) to head (f172c36).
⚠️ Report is 29 commits behind head on development/9.4.

Files with missing lines	Patch %	Lines
...tensions/lifecycle/conductor/LifecycleConductor.js	41.17%	10 Missing ⚠️
lib/tracing/index.js	33.33%	6 Missing ⚠️
bin/queuePopulator.js	0.00%	3 Missing ⚠️
extensions/gc/service.js	0.00%	3 Missing ⚠️
extensions/lifecycle/bucketProcessor/task.js	0.00%	3 Missing ⚠️
extensions/lifecycle/conductor/service.js	0.00%	3 Missing ⚠️
extensions/lifecycle/objectProcessor/task.js	0.00%	3 Missing ⚠️
extensions/notification/queueProcessor/task.js	0.00%	3 Missing ⚠️
extensions/replication/queueProcessor/task.js	0.00%	3 Missing ⚠️
...ons/replication/replicationStatusProcessor/task.js	0.00%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
extensions/lifecycle/tasks/LifecycleTask.js	91.56% <100.00%> (+0.01%)	⬆️
...ensions/notification/NotificationQueuePopulator.js	98.21% <100.00%> (+0.03%)	⬆️
...cation/destination/KafkaNotificationDestination.js	81.63% <100.00%> (+2.56%)	⬆️
extensions/replication/ReplicationAPI.js	86.95% <100.00%> (+0.28%)	⬆️
...xtensions/replication/ReplicationQueuePopulator.js	91.93% <100.00%> (-1.40%)	⬇️
lib/BackbeatConsumer.js	93.39% <100.00%> (-1.49%)	⬇️
lib/BackbeatProducer.js	89.28% <ø> (-0.90%)	⬇️
lib/queuePopulator/QueuePopulatorExtension.js	100.00% <100.00%> (+8.82%)	⬆️
bin/queuePopulator.js	0.00% <0.00%> (ø)
extensions/gc/service.js	0.00% <0.00%> (ø)
... and 8 more

... and 6 files with indirect coverage changes

Components	Coverage Δ
Bucket Notification	80.22% <75.00%> (-0.01%)	⬇️
Core Library	81.22% <80.00%> (+0.23%)	⬆️
Ingestion	70.63% <ø> (-0.61%)	⬇️
Lifecycle	78.63% <31.25%> (-0.44%)	⬇️
Oplog Populator	85.83% <ø> (ø)
Replication	59.60% <40.00%> (-0.19%)	⬇️
Bucket Scanner	85.76% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/9.4    #2733      +/-   ##
===================================================
- Coverage            74.73%   74.65%   -0.08%     
===================================================
  Files                  199      200       +1     
  Lines                13650    13724      +74     
===================================================
+ Hits                 10201    10246      +45     
- Misses                3439     3468      +29     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.07% <0.00%> (-0.06%)	⬇️
api:routes	8.90% <0.00%> (-0.05%)	⬇️
bucket-scanner	85.76% <ø> (ø)
ft_test:queuepopulator	10.92% <10.34%> (+0.79%)	⬆️
ingestion	12.50% <9.19%> (-0.07%)	⬇️
lib	7.77% <11.49%> (-0.02%)	⬇️
lifecycle	18.92% <25.28%> (-0.08%)	⬇️
notification	1.02% <0.00%> (-0.01%)	⬇️
oplogPopulator	0.14% <0.00%> (-0.01%)	⬇️
replication	18.62% <12.64%> (-0.11%)	⬇️
unit	51.42% <52.87%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DarkIsDude]

No more for me. Code is clean and factorised. No issue found. Congrats

[claude]

LGTM — well-structured PR with good test coverage.

- Tracing shim in lib/tracing/index.js cleanly delegates to arsenal; lazy instrumentation loading ensures zero overhead when ENABLE_OTEL is unset.
- All 8 entry points call init() before any instrumented library is imported, which is the required ordering for OTEL monkey-patching.
- isEnabled() guard in BackbeatConsumer._processTask and LifecycleConductor.processBuckets keeps the hot path free of any OTEL overhead when tracing is off.
- Linked (not child) spans for Kafka consumers are the correct pattern for cross-process async hops.
- Trust-boundary header stripping in KafkaNotificationDestination.send prevents trace context leaking to customer Kafka clusters; the shallow-copy approach avoids mutating caller state.
- The incidental process.exit(0) fix in replicationStatusProcessor/task.js SIGTERM handler resolves a real inconsistency with the other 7 entry points.
- yarn.lock has two @opentelemetry/api resolutions (1.9.0 from a pre-existing transitive dep, 1.9.1 from the new direct dep) — not a correctness issue since the OTEL API shares its global singleton by major version, but could be cleaned up with a dedupe pass.

Review by Claude Code