HDDS-13323. [Don't merge] Merge STS feature branch into master - #11220
Conversation
…oupings of OzoneObj and Acls (apache#9239)
…oupings of OzoneObj and Acls (apache#9292)
Co-authored-by: Fabian Morgan <fmorg-git@gmail.com>
…oupings of OzoneObj and Acls (apache#9306)
…oupings of OzoneObj and Acls (apache#9315)
…e past expiration from DB (apache#9468)
…ated STS tokens useless (apache#9602)
…n policy and handle certain errors more gracefully (apache#9795) Co-authored-by: Fabian Morgan <fmorg-git@gmail.com>
…ion Policy validation errors (apache#9823)
Conflicts: hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/AssumeRoleRequest.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/RequestContext.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMMetadataReader.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/acl/TestRequestContext.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/Gateway.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/S3ErrorTable.java
Conflicts: hadoop-hdds/common/src/main/resources/ozone-default.xml hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/RequestContext.java hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestOzoneConfigurationFields.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/acl/TestRequestContext.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketAclHandler.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketCrudHandler.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/S3ErrorTable.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java
apache#10204) Co-authored-by: Fabian Morgan <fmorg-git@gmail.com>
…ecks (apache#10688) Co-authored-by: Fabian Morgan <fmorg-git@gmail.com>
…rizer if feature flag is off (apache#10771)
…upported AssumeRole parameters (apache#10875)
Conflicts: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/OzoneDataStreamOutput.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/OzoneOutputStream.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/S3SDKTestUtils.java hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/OMDBDefinition.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/helpers/OMAuditLogger.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/OMVolumeSetOwnerRequest.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/OMVolumeSetQuotaRequest.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/acl/OMVolumeAddAclRequest.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/acl/OMVolumeRemoveAclRequest.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/acl/OMVolumeSetAclRequest.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOmMetadataManager.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/TestCleanupTableInfo.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/audit/S3GAction.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/EndpointBase.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/RootEndpoint.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/S3ErrorTable.java hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/OzoneBucketStub.java hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestEndpointBase.java
Conflicts: hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v2/AbstractS3SDKV2Tests.java hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/TestOMMetadataReader.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
Conflicts: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestOzoneConfigurationFields.java hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/lifecycle/OMLifecycleSaveScanStateRequest.java hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/audit/S3GAction.java
Conflicts: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataReader.java
…pache#11073) Co-authored-by: Fabian Morgan <fmorg-git@gmail.com>
…ent non-canonical token use (apache#11186)
a1c4075 to
556c15e
Compare
There was a problem hiding this comment.
🟡 Changes recommended
There are correctness issues in the new STS ThreadLocal handling (Instant.MAX overflow risk) and STS error responses (missing explicit XML Content-Type) that should be fixed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR brings an AWS STS-compatible endpoint into the Ozone S3 Gateway and wires the end-to-end plumbing across S3G, OM, client, CLI, and test/compose scaffolding to support temporary credentials, token revocation, and improved timestamp fidelity in copy/multipart responses.
Changes:
- Add STS JAX-RS application/server (separate S3G STS HTTP server) plus STS-oriented error handling and response headers.
- Add OM support for STS operations (AssumeRole/GetCallerIdentity/RevokeSTSToken), including a revoked-token table and Ratis/apply ThreadLocal propagation.
- Propagate modification times through commit/copy/multipart flows and extend tests + compose smoketests to validate behavior.
File summaries
| File | Description |
|---|---|
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3sts/package-info.java | Adds test package documentation for STS tests. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/package-info.java | Adds test package documentation for S3 util tests. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/TestAuthorizationFilter.java | Extends auth failure coverage to include oversized STS payload hashing. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/signature/TestStringToSignProducer.java | Updates canonical request tests for payload hash handling. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/exception/TestOS3Exceptions.java | Adds tests for ExpiredToken HTTP status and Token-0 XML output. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestEndpointBase.java | Adds tests for access-denied/expired-token mapping and wrapped OMException handling. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/OzoneOutputStreamStub.java | Updates multipart commit stubs to carry modification time. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/OzoneDataStreamOutputStub.java | Updates multipart commit stubs to carry modification time. |
| hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/client/ClientProtocolStub.java | Updates stub to new ClientProtocol methods for STS and listStatusLight options. |
| hadoop-ozone/s3gateway/src/main/resources/webapps/s3g-sts/WEB-INF/web.xml | Adds webapp descriptor for the S3G STS webapp. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSResponseMetadata.java | Adds JAXB model for STS ResponseMetadata. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEnabledEndpointRequestFilter.java | Adds filter to gate STS endpoints behind a config flag. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSEnabled.java | Adds name-binding annotation for STS enablement. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3STSConfigKeys.java | Introduces STS-specific S3G config keys and defaults. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/S3GetCallerIdentityResponseXml.java | Adds JAXB model for STS GetCallerIdentity response. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/package-info.java | Adds JAXB XmlSchema defaults for STS responses. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3sts/Application.java | Registers STS JAX-RS resources and filters. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3GActionIamMapper.java | Adds mapping from S3G actions to IAM action strings. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/StringToSignProducer.java | Refactors canonical request building to use explicit payload hash. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/SignatureProcessor.java | Expands signature parsing contract to include IO/crypto exceptions. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/SignatureInfo.java | Adds request-scoped payload hash/service/session-token fields. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AuthorizationV4QueryParser.java | Captures service and sets UNSIGNED-PAYLOAD for query-based auth. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/signature/AuthorizationV4HeaderParser.java | Captures AWS service from credential scope. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/S3STSHeadersResponseFilter.java | Adds STS-specific response headers. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/Gateway.java | Starts a dedicated STS BaseHttpServer and closes it on shutdown. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/S3ErrorTable.java | Adds STS-compatible errors and token-expired/revoked mappings. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/OSTSNotFoundExceptionMapper.java | Maps STS 404s to AWS STS XML errors. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/OSTSExceptionMapper.java | Adds exception mapper for STS exceptions. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/OS3ExceptionMapper.java | Extends S3 error XML with HostId/Token-0 for expired token cases. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/exception/OS3Exception.java | Adds HostId/Token-0 support and updates XML serialization. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/S3RequestContext.java | Ensures endpoint action is applied consistently to request context. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/RootEndpoint.java | Uses S3RequestContext for auditing and timing. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpointStreaming.java | Adds expired-token mapping and returns richer copy results with mtime. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/MultipartKeyHandler.java | Adds expired-token mapping for list parts. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/CopyResult.java | Introduces copy result object including modification time. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/CopyPartResult.java | Makes part copy lastModified deterministic via OM-sourced timestamp. |
| hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/audit/S3GAction.java | Adds STS actions to S3 gateway audit actions. |
| hadoop-ozone/s3gateway/pom.xml | Adds missing Jackson annotations dependency for S3G module. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/security/acl/package-info.java | Adds test package documentation. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/package-info.java | Adds test package documentation. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/TestCleanupTableInfo.java | Excludes stateless AssumeRole response from cleanup-table validation. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/security/TestS3GetSecretRequest.java | Fixes teardown cleanup and CurCall removal to avoid test leakage. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/security/S3SecurityTestUtils.java | Adds small test helper utilities. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/security/package-info.java | Adds test package documentation. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerStateMachine.java | Adds tests ensuring STS ThreadLocal is set/cleared during apply. |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/TestOzoneManagerRatisRequest.java | Adds tests for AssumeRole gating (enabled + authorizer constraints). |
| hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ratis/package-info.java | Adds test package documentation. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerProtocolServerSideTranslatorPB.java | Clears STS ThreadLocal after request processing. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/security/S3RevokeSTSTokenResponse.java | Adds response persisting token revocation cutoff. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/security/S3DeleteRevokedSTSTokensResponse.java | Adds response deleting revoked token entries in batch. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/security/S3AssumeRoleResponse.java | Adds stateless response wrapper for AssumeRole. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ResolvedBucket.java | Adds link resolution chain tracking for buckets. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/OMVolumeSetQuotaRequest.java | Refactors preExecute to use super.preExecute result. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/OMVolumeSetOwnerRequest.java | Refactors preExecute to use super.preExecute result. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/OMVolumeDeleteRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/acl/OMVolumeSetAclRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/acl/OMVolumeRemoveAclRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/volume/acl/OMVolumeAddAclRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/snapshot/OMSnapshotMoveTableKeysRequest.java | Ensures preExecute calls super.preExecute and uses its result. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/security/OMRenewDelegationTokenRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/security/OMGetDelegationTokenRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/security/OMCancelDelegationTokenRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/tagging/S3BucketTaggingRequestBase.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/security/S3RevokeSecretRequest.java | Refactors preExecute to build from super.preExecute result. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/security/S3GetSecretRequest.java | Refactors request building and explicitly clears update message when unused. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/security/S3DeleteRevokedSTSTokensRequest.java | Adds OM request for revoked-token cleanup deletions with admin checks. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/security/OMSetSecretRequest.java | Ensures preExecute uses super.preExecute result. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadCommitPartRequest.java | Adds modification time to commit part response. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/s3/multipart/S3MultipartUploadAbortRequest.java | Minor refactor to keep keyArgs final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/lifecycle/OMLifecycleSetServiceStatusRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/lifecycle/OMLifecycleSaveScanStateRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/lifecycle/OMLifecycleConfigurationSetRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/lifecycle/OMLifecycleConfigurationDeleteRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeySetTimesRequest.java | Minor refactor to keep preExecute result final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeysDeleteRequest.java | Ensures preExecute returns the preExecute’d request. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyRenameRequest.java | Minor refactor to keep request final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyDeleteRequest.java | Minor refactor to keep request final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCreateRequest.java | Minor refactor to keep request final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequestWithFSO.java | Adds modification time into CommitKeyResponse. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyCommitRequest.java | Adds modification time into CommitKeyResponse. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeySetAclRequest.java | Ensures super.preExecute is called and merged correctly. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeyRemoveAclRequest.java | Ensures super.preExecute is called and merged correctly. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/acl/OMKeyAddAclRequest.java | Ensures super.preExecute is called and merged correctly. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMRecoverLeaseRequest.java | Minor refactor to keep request final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/file/OMFileCreateRequest.java | Minor refactor to keep request final. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetPropertyRequest.java | Ensures super.preExecute result is used when updating request. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/OMBucketSetOwnerRequest.java | Ensures super.preExecute result is used when updating request. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/acl/OMBucketSetAclRequest.java | Merges SetAcl mutation on top of super.preExecute to preserve STS auth resolution. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/acl/OMBucketRemoveAclRequest.java | Ensures super.preExecute result is used when updating request. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/bucket/acl/OMBucketAddAclRequest.java | Ensures super.preExecute result is used when updating request. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java | Adds OM request wiring for STS operations and uses a UTC Clock. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/OzoneManagerStateMachine.java | Propagates S3Auth + STS token ThreadLocals on apply path and clears them. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java | Adds s3RevokedStsTokenTable initialization and accessor. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/helpers/OMAuditLogger.java | Adds audit mapping for STS commands. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/codec/OMDBDefinition.java | Adds OM DB column family for revoked STS tokens. |
| hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/audit/OMAction.java | Adds STS actions to OM audit vocabulary. |
| hadoop-ozone/ozone-manager/pom.xml | Adds BouncyCastle provider dependency. |
| hadoop-ozone/interface-storage/src/main/java/org/apache/hadoop/ozone/om/OMMetadataManager.java | Adds revoked STS token table getter to the interface. |
| hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestOzoneConfigurationFields.java | Registers STS config keys for config/xml parity test. |
| hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v2/package-info.java | Adds package documentation. |
| hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/v1/AbstractS3SDKV1Tests.java | Adds integration tests for copy/part-copy last-modified correctness. |
| hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/S3SDKTestUtils.java | Adds OM-based helpers to fetch key/part modification times. |
| hadoop-ozone/integration-test-s3/src/test/java/org/apache/hadoop/ozone/s3/awssdk/package-info.java | Adds package documentation. |
| hadoop-ozone/dist/src/main/smoketest/security/ozone-secure-sts-polaris.sql | Adds SQL smoketest snippet for Polaris validation. |
| hadoop-ozone/dist/src/main/smoketest/security/generate_oversized_session_policy.py | Adds helper to generate oversized policies for limit testing. |
| hadoop-ozone/dist/src/main/compose/ozonesecure/docker-config | Enables STS endpoint in secure compose config. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-sts.sh | Adds STS compose test runner including Polaris smoketest. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/test-ranger.sh | Refactors to use shared ranger-testlib setup. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/ranger.yaml | Updates Ranger plugin classpath and JDK17+ javax.annotation workaround. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/polaris.yaml | Adds Polaris + Spark SQL overlay compose file for STS smoketests. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/docker-config | Enables STS endpoint in secure-HA compose config. |
| hadoop-ozone/dist/src/main/compose/ozonesecure-ha/.env | Adds FF_ENABLE_OZONE_ACTION_MATCHES_CONDITION toggle. |
| hadoop-ozone/dist/src/main/compose/common/ranger.yaml | Parameterizes init scripts and makes feature flags/Java opts configurable. |
| hadoop-ozone/dist/src/main/compose/common/init-kdc.sh | Adds service principals for Polaris-related STS testing. |
| hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/security/acl/iam/package-info.java | Adds test package documentation. |
| hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/om/helpers/TestS3STSUtilsCallerIdentity.java | Adds unit tests for caller identity helpers. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/security/acl/iam/package-info.java | Adds IAM package documentation. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java | Classifies STS commands as read-only / follower-safe appropriately. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/S3Auth.java | Extends auth context with session token and S3 action string. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java | Adds STS APIs and changes commitKey to return modification time. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/OMConfigKeys.java | Adds configs for revoked STS token cleanup service scheduling. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmMultipartCommitUploadPartInfo.java | Adds modification time to multipart commit part info. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyArgs.java | Adds listPrefix for STS auth decisions on list operations. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/CallerIdentityInfo.java | Adds protobuf wrapper for GetCallerIdentity. |
| hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/exceptions/OMException.java | Adds new OM result codes for revoked token and malformed policy doc. |
| hadoop-ozone/common/pom.xml | Adds jackson-core dependency where needed. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/protocol/ClientProtocol.java | Adds STS APIs and listStatusLight options overload. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java | Passes listPrefix through to listStatusLight for STS-aware LIST authorization. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/ObjectStore.java | Exposes STS operations via ObjectStore. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/OzoneOutputStream.java | Exposes committed key modification time. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/OzoneDataStreamOutput.java | Exposes committed key modification time for streaming output. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java | Exposes modification time from commit. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyDataStreamOutput.java | Exposes modification time from commit. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockOutputStreamEntryPool.java | Stores modification time returned from commitKey/commitMultipartUploadPart. |
| hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockDataStreamOutputEntryPool.java | Stores modification time returned from commitKey/commitMultipartUploadPart. |
| hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/s3/S3Shell.java | Adds CLI subcommand for STS token revocation. |
| hadoop-ozone/cli-shell/src/main/java/org/apache/hadoop/ozone/shell/s3/RevokeSTSTokenHandler.java | Implements ozone s3 revokeststoken CLI for revocation. |
| hadoop-hdds/dev-support/checkstyle/checkstyle.xml | Adds checkstyle rule enforcing super.preExecute usage pattern. |
| hadoop-hdds/common/src/main/resources/ozone-default.xml | Documents STS endpoint configs and STS token cleanup service configs. |
| hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java | Adds constants related to STS revoke and temp access key ID. |
| hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java | Adds STS enabled config key definitions. |
Review details
- Files reviewed: 178/205 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // where RoleArn max length is 2048 and max bytes per character in UTF-8 encoding is 12 | ||
| // (2048 * 12 = 24576) + other parameters and overheads, so setting to 32 KB | ||
| // this limit can be adjusted via configuration if needed. |
There was a problem hiding this comment.
@fmorg-git , is this a valid point? If yes, we can correct it later in any next STS patch.
There was a problem hiding this comment.
@ChenSammi - the value of 32KB seems like a good limit, although the comment seems to be inaccurate in that the limit is not configurable. It looks like the 12 is coming from this:
12 is the worst-case percent-encoded wire size for one code point in application/x-www-form-urlencoded data: a 4-byte UTF-8 sequence can become four %XX triplets
There was a problem hiding this comment.
@ChenSammi this comment is corrected under #11261
|
Thanks for the review and merge |
What changes were proposed in this pull request?
Provide a one-liner summary of the changes in the PR Title field above.
It should be in the form of
HDDS-1234. Short summary of the change.Please describe your PR in detail:
perspective not just for the reviewer.
the Jira's description if the jira is well defined.
issue investigation, github discussion, etc.
Examples of well-written pull requests:
What is the link to the Apache JIRA
Please create an issue in ASF JIRA before opening a pull request, and you need to set the title of the pull
request which starts with the corresponding JIRA issue number. (e.g. HDDS-XXXX. Fix a typo in YYY.)
If you do not have an ASF Jira account yet, please follow the first-time contributor
instructions in the Jira guideline.
(Please replace this section with the link to the Apache JIRA)
How was this patch tested?
(Please explain how this patch was tested. Ex: unit tests, manual tests, workflow run on the fork git repo.)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this.)