Skip to content

HDDS-16497. Support OM ThreadLocal context propagation for Ratis read - #11257

Open
ivandika3 wants to merge 12 commits into
apache:masterfrom
ivandika3:HDDS-16497
Open

ivandika3 wants to merge 12 commits into
apache:masterfrom
ivandika3:HDDS-16497

Conversation

@ivandika3

@ivandika3 ivandika3 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

While working on HDDS-15089, encountered an NPE issue that was caused by null
Server.getRemoteUser() in OzoneManager#getS3VolumeContext. This reason is that Ratis would call StateMachine#query in a separate thread (under RaftServerImpl#clientExecutor). Therefore, the ThreadLocal in OzoneManager and the CurrCall which is used by Server.getRemoteUser() is not going to be propagated.

This affects most read requests submitted to Ratis (executed in OzoneManagerStateMachine#query) that require using ThreadLocal one way or another.

We need a way to propagate the Server#getCurrcall ThreadLocal context to the StateMachine#query.

Note this suggests that follower read feature is not production ready until this is resolved.

The main mechanism is the introduction of OMRatisRequestContext which is a context mechanism to pass the OM ThreadLocal info OMRequest UserInfo and allows the OzoneManagerStateMachine#query (that is executed in a separate thread) to create am artificial Server.Call with getRemoteUser and getHostInetAddress so that Server.getRemoteUser() will not return null. Additionally, this OmRatisRequestContext also helps to handle propagation to OzoneManager ThreadLocal like stsTokenIdentifier.

Note that OM might need to use the RaftServer#readOnlyAsync (apache/ratis#1448) that might either be executed directly in caller thread (if we use DEFAULT ReadOption) or executed in another thread (if we use LINEARIZABLE ReadOption). Therefore, OMRatisRequestContext should handle context propagation within a thread and across differentthreads (see previousCall and previousS3Context).

I chose this approach since it does not need to change every OM read implementation. However, any suggestions to improve this design is welcome since this is a core OM logic.

This patch also includes some refactoring on OMLockDetailsUtil and S3AuthenticationContext to reduce duplications.

Saw projects like https://github.com/alibaba/transmittable-thread-local to pass the ThreadLocal context across threads, but the current solution should be enough for now. In the future, we should also think whether this ThreadLocal passing mechanisms should be a first-class feature for Ratis (or we can support something like ScopedValues in Ratis).

Generated by: GPT 5.6

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-16497

How was this patch tested?

UT and IT.

Clean CI: https://github.com/ivandika3/ozone/actions/runs/35331947876

@ivandika3 ivandika3 self-assigned this Sep 18, 2026
@ivandika3
ivandika3 marked this pull request as ready for review September 18, 2026 14:37
@ivandika3 ivandika3 changed the title HDDS-16497. Support passing OM ThreadLocal context for Ratis read HDDS-16497. Support OM ThreadLocal context propagation for Ratis read Sep 19, 2026
@adoroszlai

Copy link
Copy Markdown
Contributor

Please take a look at intermittent test failure, only seen in this PR so far.

[ERROR] Tests run: 64, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 51.18 s <<< FAILURE! -- in org.apache.hadoop.fs.ozone.TestOFS
[ERROR] org.apache.hadoop.fs.ozone.TestOFS.testGetFileStatusUsesSingleOmRpc -- Time elapsed: 0.028 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <500> but was: <501>
	at org.apache.hadoop.fs.ozone.AbstractRootedOzoneFileSystemTest.testGetFileStatusUsesSingleOmRpc(AbstractRootedOzoneFileSystemTest.java:1858)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants