Make the OpenTelemetry v2 meter provider replay safe

[patbeqo]

What changed?

• ReplaySafeOpenTelemetry now wraps the configured SdkMeterProvider in a replay-safe meter provider, so meters obtained from the global (including through meterBuilder) are replay safe inside workflows, matching the existing tracer provider.
• Added ReplaySafeMeter, which wraps every synchronous instrument (long and double counters, up-down counters, histograms, and gauges). Recordings made by replaying workflow code are dropped so they are not recorded again on every replay. Observable instruments and batch callbacks pass through unchanged because their callbacks run on the metric reader thread, not in workflow code.
• Extracted the replay predicate shared by spans and instruments into OpenTelemetrySuppression.shouldSuppress(): a workflow thread that is replaying and is not read-only. Query handlers and update validators run live at most once per request, so their telemetry is kept.

Why?

Part of #3046. This is the fourth slice, stacked on #3089.

Breaking changes?

None. The module is new and marked experimental.

Server PR

None.

Test plan

MetricsTest (added in #3089) covers live recording and replay suppression and passes with this change.

• mise exec -- ./gradlew :temporal-opentelemetry-v2:test -PtestServer=dev-server -x spotlessCheck -x spotlessJava (37 tests; the 2 InterceptorTest failures, promiseCallbackPreservesApplicationContext and workflowClientUpdateIncludesTargetRunIdAndUpdateId, fail identically on the patbeqo/otel-v2 base and are not related to this change)
• mise exec -- ./gradlew :temporal-opentelemetry-v2:spotlessCheck
• mise exec -- ./gradlew :temporal-opentelemetry-v2:javadoc

[patbeqo]

@codex

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-18T23:06:09.560859Z	88575c3	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77f4c1d568

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Prevent observable callback registration during replay

When workflow code builds an observable instrument, every cache eviction and replay executes this registration again against the same long-lived meter provider. Passing buildWithCallback through unchanged accumulates callbacks whose handles are never closed, so subsequent collections can invoke callbacks from both the old and replayed workflow instances, duplicating observable-counter data and retaining evicted instances. Observable registration should either be made lifecycle/replay-safe or explicitly rejected from workflow threads; the same issue applies to the other observable builders and batchCallback.

Useful? React with 👍 / 👎.

[patbeqo]

Added a warning about this.

[DABH]

I reproduced this with the unchanged meter code on the final stack: a workflow registered one observable-counter callback, collection reported 1, and WorkflowReplayer replayed the completed history against the same provider. The next collection reported 2 because replay registered a second callback. Those registrations also retain the reconstructed workflow instances and run from the reader thread outside workflow synchronization. The new warning describes the hazard but does not prevent it through this otherwise standard Meter API; reject observable registration from workflow threads (or provide lifecycle-safe ownership) rather than passing it through. Reopening as an actionable correctness/resource issue.

[DABH]

Adding a concrete fix recommendation after a second pass. Reject buildWithCallback, buildObserver, and batchCallback when WorkflowUnsafe.isWorkflowThread() is true, throwing an IllegalStateException in the style of WorkflowThreadMarker.enforceNonWorkflowThread(); that is deterministic because it does not depend on replay state, and it only breaks code that is already broken. A no-op only during replay would be wrong: the live registration would still capture the soon-evicted workflow instance, and the replayed instance would never be observed. Note also that the Javadoc's advice to create observable instruments from activity code has the same accumulation problem, since each activity execution registers another callback and the SDK sums same-attribute callbacks until close(); the guidance should be to register once at worker or process startup.

[DABH]

Ultrareviewed the exact stacked delta and its use from final head #3091. Synchronous recording suppression is complete across the pinned OpenTelemetry 1.66 instrument API, and the earlier Workflow.await concern is correctly fixed by isSubjectToReplay().

I found one wrapper-contract/performance issue inline. I also reproduced the previously resolved observable-instrument issue on this exact stack: one callback returned a value of 1 before replay and 2 after WorkflowReplayer reconstructed the workflow, proving that registrations accumulate. I am reopening that thread with the reproduction; a warning does not make the standard meter API replay/lifecycle safe.

Focused MetricsTest and LoggerTest runs passed locally; the worktree is clean. Current PR checks are green.

[DABH]

[P2] Report instruments disabled while replay drops their recordings

All eight synchronous wrappers delegate isEnabled() even when OpenTelemetrySuppression.shouldSuppress() guarantees that the subsequent add/record/set is discarded. Code using the standard guard therefore repeats potentially expensive attribute/value construction throughout every replay for a measurement that cannot be exported. It also violates the wrapper's effective enabled state.

Return !OpenTelemetrySuppression.shouldSuppress() && delegate.isEnabled() for each instrument, matching the replay-safe logger implementation in #3091.

[DABH]

Ultrareviewed the exact stacked delta a second time with an independent trace against the pinned OpenTelemetry 1.66.0 API. Synchronous suppression remains correct and complete, and the predicate is right: isWorkflowThread() is set only by DeterministicRunnerImpl.setCurrentThreadInternal, so activity, Nexus, local-activity, and metric-reader threads are never suppressed, while Async and detached-scope threads are. No new P1 or P2.

Both open threads stand and are small fixes. On the observable-instrument thread I have added a concrete recommendation: reject buildWithCallback, buildObserver, and batchCallback from workflow threads outright, mirroring WorkflowThreadMarker.enforceNonWorkflowThread(), because a replay-only no-op would still leave the live registration holding the soon-evicted instance. The isEnabled() finding is now also an internal inconsistency, since #3091's ReplaySafeLogger.isEnabled returns false under suppression while the eight instrument wrappers do not.

Smaller items, not blocking: the class Javadoc warning at ReplaySafeMeter.java:39-41 advises creating observable instruments from activity code, but per-activity-execution registration accumulates the same way (two same-attribute callbacks each recording 1 collect as 2), so the advice should be "register once at process startup", and it belongs on ReplaySafeOpenTelemetry and in the README rather than on an internal class; ReplaySafeOpenTelemetry exposes only close(), so users with a PeriodicMetricReader cannot forceFlush() mid-run; OtelTestBase shares one static InMemoryMetricReader that @AfterClass shuts down permanently, which only works because forkEvery = 1 isolates test classes; MetricsTest covers only LongCounter via MeterProvider.get, leaving meterBuilder(), the other instrument types, isEnabled(), and the query and validator exemptions untested; and the PR description says MetricsTest was added in #3089 (it is new here) and describes the predicate as "replaying and is not read-only" (the head uses isSubjectToReplay()). Current PR checks are green.