[GitHub] [hadoop] zhangshuyan0 commented on pull request #5759: HDFS-17052. Erasure coding reconstruction failed when num of storageT…

["zhangshuyan0 (via GitHub)" <gi...@apache.org>]

zhangshuyan0 commented on PR #5759:
URL: https://github.com/apache/hadoop/pull/5759#issuecomment-1606028379

   Great catch here! It's make sense to me. I have some thoughts to discuss with you. The solution here may involve multiple calls to `chooseOnce`, some of which may be unnecessary and waste some time.
   The root cause of this problem is that 'BlockPlacementPolicyDefault#getMaxNodesPerRack' cannot return an accurate value for `maxNodesPerRack`.  How about compute a right value before the loop in `chooseEvenlyFromRemainingRacks`? Will this be more efficient?
   ```
   private void chooseEvenlyFromRemainingRacks(Node writer,
         Set<Node> excludedNodes, long blocksize, int maxNodesPerRack,
         List<DatanodeStorageInfo> results, boolean avoidStaleNodes,
         EnumMap<StorageType, Integer> storageTypes, int totalReplicaExpected,
         NotEnoughReplicasException e) throws NotEnoughReplicasException {
       int numResultsOflastChoose = 0;
       NotEnoughReplicasException lastException = e;
       int bestEffortMaxNodesPerRack = maxNodesPerRack;
       Map<String, Integer> nodesPerRack = new HashMap<>();
       for (DatanodeStorageInfo dsInfo : results) {
         String rackName = dsInfo.getDatanodeDescriptor().getNetworkLocation();
         nodesPerRack.merge(rackName, 1, Integer::sum);
       }
       for (int numNodes : nodesPerRack.values()) {
         if (numNodes > bestEffortMaxNodesPerRack) {
           bestEffortMaxNodesPerRack = numNodes;
         }
       }
       while (results.size() != totalReplicaExpected &&
           numResultsOflastChoose != results.size()) {
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org