You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/03/04 07:41:00 UTC
[jira] [Commented] (HDFS-17732) addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks
[ https://issues.apache.org/jira/browse/HDFS-17732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932176#comment-17932176 ]
ASF GitHub Bot commented on HDFS-17732:
---------------------------------------
LiuGuH commented on PR #7414:
URL: https://github.com/apache/hadoop/pull/7414#issuecomment-2696498082
@Hexiaoqiao @ZanderXu @tomscut , Hello Sir , please reivew this PR if you have time . Thanks very much
> addExpectedReplicasToPending should add EC block expectedStorages in PendingReconstructionBlocks
> ------------------------------------------------------------------------------------------------
>
> Key: HDFS-17732
> URL: https://issues.apache.org/jira/browse/HDFS-17732
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: liuguanghua
> Assignee: liuguanghua
> Priority: Major
> Labels: pull-request-available
>
> Now when a block commit , it should add expectedStorages into PendingReconstructionBlocks if the ibr is not sent from expected locations. And for a ec block group , only blkStriped.getRealTotalBlockNum() == expectedStorages.length will be added into PendingReconstructionBlocks.
> The problem is that a samll file with ec, it may always blkStriped.getRealTotalBlockNum() < expectedStorages.length if the file size < ecPolicy.getNumDataUnits()*ecPolicy.getCellSize().
> The following situations need to be considered:
> (1) A EC file with XOR-2-1-1024k,when we write lower than 1024k into the file,close it.
> (2) Now The file has only one ec block group, indics=(0,1,2) and suppose only 0 replica ibr is received and 2 replica ibr is delayed. The ec block group reaches minstorage and commit .
> (3) When block group commit , blkStriped.getRealTotalBlockNum()=1 expectedStorages.length=3 , so it can not add this ec block group expectedStorages into PendingReconstructionBlocks.
> (4) The ec block group complete and only has minstorage and the the namenode generates ec reconstruncation task to datanode.
> (5) The 2 replicas ibr is recived.
> (6) Datanode reconstruction task finished and ibr sends.
> (7) (5) and (6) may in a same datanode
> For other ec policy, in (7) , on a same datanode will has two different ec block indic. For ec this is not expected and if it happens , in namenode will be AddBlockResult.REPLACED replaced the former ibr. And it maybe increases the ec reconstruncation task again.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org