You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "devmadhuu (via GitHub)" <gi...@apache.org> on 2026/08/10 15:26:14 UTC

[PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

devmadhuu opened a new pull request, #10983:
URL: https://github.com/apache/ozone/pull/10983

   ## What changes were proposed in this pull request?
   
   Merge recent commits from `master` into the `HDDS-11233` feature branch to keep it in sync with mainline. This brings the branch up to date with `master` and reconciles the StorageTier feature commits with recent master changes.
   
   The feature branch's original commit history is preserved (first-parent), with a single `Merge remote-tracking branch 'upstream/master' into HDDS-11233` commit on top.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-16116
   
   ## How was this patch tested?
   
   Verified core modules compile against the merged master locally; relying on this PR's CI to validate before merge. To be merged via GitHub after CI passes and review.
   
   Made with [Cursor](https://cursor.com)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5275663779

   @devmadhuu Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu merged PR #10983:
URL: https://github.com/apache/ozone/pull/10983


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5275644039

   @ivandika3 @jojochuang @ChenSammi @xichen01 @ashishkumar50 Kindly review this merge PR for merging master to feature branch for storage policy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "F64116045 (via GitHub)" <gi...@apache.org>.
F64116045 commented on code in PR #10983:
URL: https://github.com/apache/ozone/pull/10983#discussion_r3813861648


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/SCMCommonPlacementPolicy.java:
##########


Review Comment:
   Addressed this in HDDS-16045. Thanks for pointing it out.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on code in PR #10983:
URL: https://github.com/apache/ozone/pull/10983#discussion_r3793455906


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/SCMCommonPlacementPolicy.java:
##########


Review Comment:
   All these storage space logic should be put to `SCMNodeManager#hasAvailableSpace`, otherwise we have two conflicting space accounting logic. We can address it in HDDS-16045.
   
   cc: @F64116045 



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/ECPipelineProvider.java:
##########
@@ -141,19 +146,13 @@ public Pipeline createForRead(
 
     dns.sort(Comparator.comparing(nodeStatusMap::get, CREATE_FOR_READ_COMPARATOR));
 
-    // Read Pipelines do not require storage tiers, so the calculation of storage tiers can be omitted.
-    return createPipelineInternal(replicationConfig, dns, map, null);
-  }
-
-  private Pipeline createPipelineInternal(ECReplicationConfig repConfig,
-      List<DatanodeDetails> dns, Map<DatanodeDetails, Integer> indexes, StorageTier storageTier) {
-    return Pipeline.newBuilder()
-        .setId(PipelineID.randomId())
-        .setState(Pipeline.PipelineState.ALLOCATED)
-        .setReplicationConfig(repConfig)
-        .setNodes(dns)
-        .setReplicaIndexes(indexes)
-        .setSupportedStorageTier(storageTier)
+    // Use insecureRandomId for throwaway read pipeline IDs to avoid
+    // contention on the shared SecureRandom instance.
+    // Read Pipelines do not require storage tiers, so no supported tier is set.
+    return newPipelineBuilder(replicationConfig, dns)
+        .setId(PipelineID.insecureRandomId())
+        .setReplicaIndexes(map)
+        .setSupportedStorageTier(null)

Review Comment:
   Keep the comment
   
   `// Read Pipelines do not require storage tiers, so the calculation of storage tiers can be omitted.`



##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/SimplePipelineProvider.java:
##########
@@ -117,16 +107,23 @@ public Pipeline create(StandaloneReplicationConfig replicationConfig,
   @Override
   public Pipeline createForRead(StandaloneReplicationConfig replicationConfig,
       Set<ContainerReplica> replicas) {
-    // Read Pipelines do not require storage tiers, so the calculation of storage tiers can be omitted.
-    return createPipelineInternal(replicationConfig, replicas
-        .stream()
-        .map(ContainerReplica::getDatanodeDetails)
-        .collect(Collectors.toList()), null);
+    // Use insecureRandomId for throwaway read pipeline IDs to avoid
+    // contention on the shared SecureRandom instance.
+    // Read Pipelines do not require storage tiers, so no supported tier is set.
+    return newPipelineBuilder(replicationConfig, ContainerReplica.toDatanodeDetailsList(replicas))
+        .setId(PipelineID.insecureRandomId())
+        .setSupportedStorageTier(null)

Review Comment:
   Nit: Also keep the comment
   
   `// Read Pipelines do not require storage tiers, so the calculation of storage tiers can be omitted.`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5275725273

   > Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to [HDDS-11233](https://issues.apache.org/jira/browse/HDDS-11233).
   
   Thanks @ivandika3 for helping in review. I think mostly below areas is something you can focus:
   
   1. `SCMCommonPlacementPolicy#hasEnoughSpace` — master moved to a slot-based availability check; the feature uses a StorageType + byte-based check. Signatures now using storageType + nodeManager.
   2. Pipeline providers (`RatisPipelineProvider, SimplePipelineProvider, ECPipelineProvider, PipelineProvider, PipelineStateMap, PipelineStateManager(Impl), PipelineManager(Impl), WritableECContainerProvider`) — kept master's newPipelineBuilder pattern and added setSupportedStorageTier(...) + a storage-tier filter in getPipelines.
   3. `BlockManagerImpl#allocateBlock` — master used getDefaultTier; feature adds storagePolicy + creation/fallback-tier logic. Kept the fallback logic on top of master's flow.
   4. `ContainerProtocolCalls#writeChunkAsync / BlockOutputStream` — both sides added a new trailing arg (master: containerAutoCreate, feature: storageType). Merged into one overload chain carrying both.
   5. Container state/manager layer (ContainerManager(Impl), ContainerStateManager(Impl), ContainerStateManagerInvoker, ContainerInfo) — feature's getMatchingContainerAndStorageTier rename + setStorageTier(...)
   6. `NodeUtils `— updated to master's getNode(DatanodeID) after the DatanodeID/DatanodeDetails split.
   7. Proto (`OmClientProtocol.proto, hdds.proto, DatanodeClientProtocol.proto, ScmAdminProtocol.proto, ScmServerDatanodeHeartbeatProtocol.proto`) — feature's storage-tier/policy fields (e.g. `OmBucketInfo/OmBucketArgs storagePolicy/allowFallbackStoragePolicy`) . All additive, no renumbering of existing fields.
   8. EC/Ratis under-replication handlers (`ECUnderReplicationHandler, RatisUnderReplicationHandler`) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5323560708

   @ivandika3 is this PR good to merge now ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5290437806

   > Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to [HDDS-11233](https://issues.apache.org/jira/browse/HDDS-11233).
   
   Thanks @ivandika3 for helping in review. I think mostly below areas is something you can focus:
   
   1. `SCMCommonPlacementPolicy#hasEnoughSpace` — master moved to a slot-based availability check; the feature uses a StorageType + byte-based check. Signatures now using storageType + nodeManager.
   2. Pipeline providers (`RatisPipelineProvider, SimplePipelineProvider, ECPipelineProvider, PipelineProvider, PipelineStateMap, PipelineStateManager(Impl), PipelineManager(Impl), WritableECContainerProvider`) — kept master's newPipelineBuilder pattern and added setSupportedStorageTier(...) + a storage-tier filter in getPipelines.
   3. `BlockManagerImpl#allocateBlock` — master used getDefaultTier; feature adds storagePolicy + creation/fallback-tier logic. Kept the fallback logic on top of master's flow.
   4. `ContainerProtocolCalls#writeChunkAsync / BlockOutputStream` — both sides added a new trailing arg (master: containerAutoCreate, feature: storageType). Merged into one overload chain carrying both.
   5. Container state/manager layer (ContainerManager(Impl), ContainerStateManager(Impl), ContainerStateManagerInvoker, ContainerInfo) — feature's getMatchingContainerAndStorageTier rename + setStorageTier(...)
   6. `NodeUtils `— updated to master's getNode(DatanodeID) after the DatanodeID/DatanodeDetails split.
   7. Proto (`OmClientProtocol.proto, hdds.proto, DatanodeClientProtocol.proto, ScmAdminProtocol.proto, ScmServerDatanodeHeartbeatProtocol.proto`) — feature's storage-tier/policy fields (e.g. `OmBucketInfo/OmBucketArgs storagePolicy/allowFallbackStoragePolicy`) . All additive, no renumbering of existing fields.
   8. EC/Ratis under-replication handlers (`ECUnderReplicationHandler, RatisUnderReplicationHandler`) 
   
   > > Could you help to highlight any conflicts that need reviewing? It is almost impossible for me to take a look at 100k+ LOC changes. If there are no notable conflicts, then we can proceed in merging master to [HDDS-11233](https://issues.apache.org/jira/browse/HDDS-11233).
   > 
   > Thanks @ivandika3 for helping in review. I think mostly below areas is something you can focus:
   > 
   >     1. `SCMCommonPlacementPolicy#hasEnoughSpace` — master moved to a slot-based availability check; the feature uses a StorageType + byte-based check. Signatures now using storageType + nodeManager.
   > 
   >     2. Pipeline providers (`RatisPipelineProvider, SimplePipelineProvider, ECPipelineProvider, PipelineProvider, PipelineStateMap, PipelineStateManager(Impl), PipelineManager(Impl), WritableECContainerProvider`) — kept master's newPipelineBuilder pattern and added setSupportedStorageTier(...) + a storage-tier filter in getPipelines.
   > 
   >     3. `BlockManagerImpl#allocateBlock` — master used getDefaultTier; feature adds storagePolicy + creation/fallback-tier logic. Kept the fallback logic on top of master's flow.
   > 
   >     4. `ContainerProtocolCalls#writeChunkAsync / BlockOutputStream` — both sides added a new trailing arg (master: containerAutoCreate, feature: storageType). Merged into one overload chain carrying both.
   > 
   >     5. Container state/manager layer (ContainerManager(Impl), ContainerStateManager(Impl), ContainerStateManagerInvoker, ContainerInfo) — feature's getMatchingContainerAndStorageTier rename + setStorageTier(...)
   > 
   >     6. `NodeUtils `— updated to master's getNode(DatanodeID) after the DatanodeID/DatanodeDetails split.
   > 
   >     7. Proto (`OmClientProtocol.proto, hdds.proto, DatanodeClientProtocol.proto, ScmAdminProtocol.proto, ScmServerDatanodeHeartbeatProtocol.proto`) — feature's storage-tier/policy fields (e.g. `OmBucketInfo/OmBucketArgs storagePolicy/allowFallbackStoragePolicy`) . All additive, no renumbering of existing fields.
   > 
   >     8. EC/Ratis under-replication handlers (`ECUnderReplicationHandler, RatisUnderReplicationHandler`)
   
   @ivandika3 did you get chance ? I am working on green CI.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #10983:
URL: https://github.com/apache/ozone/pull/10983#discussion_r3793895953


##########
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/SCMCommonPlacementPolicy.java:
##########


Review Comment:
   Agreed, I'll leave as it is.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "ivandika3 (via GitHub)" <gi...@apache.org>.
ivandika3 commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5312415608

   Thanks @devmadhuu , +1 after the CI is green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16116. Merge recent commits from master to HDDS-11233 [ozone]

Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on PR #10983:
URL: https://github.com/apache/ozone/pull/10983#issuecomment-5324086792

   Thanks @ivandika3 for reviewing the PR. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org