[jira] [Commented] (HDFS-17800) Observer node can return insufficient blocks for erasure coded files

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075941#comment-18075941 ] 

ASF GitHub Bot commented on HDFS-17800:
---------------------------------------

ZanderXu commented on code in PR #7754:
URL: https://github.com/apache/hadoop/pull/7754#discussion_r3136421721


##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##########
@@ -9206,16 +9197,75 @@ private boolean isObserver() {
     return haEnabled && haContext != null && haContext.getState().getServiceState() == OBSERVER;
   }
 
+  private boolean hasSufficientReplicas(LocatedBlock block,
+      ErasureCodingPolicy ecPolicy) {
+    DatanodeInfo[] locations = block.getLocations();
+    if (locations == null) {
+      return false;
+    }
+
+    if (block.isStriped()) {
+      LocatedStripedBlock stripedBlock = (LocatedStripedBlock) block;

Review Comment:
   ```
    LocatedStripedBlock stripedBlock = (LocatedStripedBlock) block;
         // For erasure coded files, require enough data units to reconstruct
         // the block, bounded by the number of cells in the block.
         long numCells = (block.getBlockSize() - 1) / ecPolicy.getCellSize() + 1;
         int minRequiredIndices = (int) Math.min(ecPolicy.getNumDataUnits(), numCells);
         
         // Units can be over-replicated, so need to account for unique indices
         byte[] blockIndices = stripedBlock.getBlockIndices();
         boolean[] seenIndices = new boolean[ecPolicy.getNumDataUnits() + ecPolicy.getNumParityUnits()];
   
         int uniqueIndexCount = 0;
         for (byte idx : blockIndices) {
           int index = idx & 0xFF;
           if (!seenIndices[index]) {
             seenIndices[index] = true;
             if (++uniqueIndexCount >= minRequiredIndices) {
               return true;
             }
           }
         }
         return false;
   ```



##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##########
@@ -9206,16 +9197,75 @@ private boolean isObserver() {
     return haEnabled && haContext != null && haContext.getState().getServiceState() == OBSERVER;
   }
 
+  private boolean hasSufficientReplicas(LocatedBlock block,

Review Comment:
   hasSufficientBlockLocations





> Observer node can return insufficient blocks for erasure coded files
> --------------------------------------------------------------------
>
>                 Key: HDFS-17800
>                 URL: https://issues.apache.org/jira/browse/HDFS-17800
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 3.4.1
>            Reporter: Adam Binford
>            Priority: Major
>              Labels: pull-request-available
>
> Similar to https://issues.apache.org/jira/browse/HDFS-13924, https://issues.apache.org/jira/browse/HDFS-16732, and https://issues.apache.org/jira/browse/HDFS-17768, except for erasure coded files. A single block may not be enough to fulfill an erasure coded read, as you need at least the number of data shards of blocks to read an erasure coded file. The checks need to be updated to factor in erasure coded files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org