You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "David (Jira)" <ji...@apache.org> on 2021/01/20 08:10:00 UTC
[jira] [Resolved] (HDFS-15773) NameNode allocate five DataNodes for
a small EC file (606 bytes) in log file
[ https://issues.apache.org/jira/browse/HDFS-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David resolved HDFS-15773.
--------------------------
Resolution: Information Provided
> NameNode allocate five DataNodes for a small EC file (606 bytes) in log file
> ----------------------------------------------------------------------------
>
> Key: HDFS-15773
> URL: https://issues.apache.org/jira/browse/HDFS-15773
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: David
> Priority: Major
>
> For testing Erasure Coding, I put a small file "hosts4" (606 bytes) to a EC enabled HDFS directory.
> . In the output of the fsck on this file, there are three replicas which is not surprise cause it's small than 1M.
>
> _/ec1/hosts4 606 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s): OK_
> _0. BP-1329090504-172.25.37.132-1610344959454:blk_-9223372036854775184_1133 len=606 Live_repl=3 [blk_-9223372036854775184:DatanodeInfoWithStorage[172.25.34.135:9866,DS-b82b3dd4-cf57-4be6-9a0e-21089cf01d17,DISK], blk_-9223372036854775181:DatanodeInfoWithStorage[172.25.35.145:9866,DS-ef58c1c4-b54b-4321-96e5-eb39dd8e21cd,DISK], blk_-9223372036854775180:DatanodeInfoWithStorage[172.25.40.139:9866,DS-a160a4ae-ac88-4103-b19c-dc0f3fc0f861,DISK]]_
> . But in the NameNode log file, there were five DataNodes have been selected. However, only three DNs reported FINALIZED status, and there is a gap on the block IDs.
>
> _2021-01-12 11:12:54,204 DEBUG org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.addBlock: /ec1/hosts4._COPYING_ with blk_-9223372036854775184_1133 block is added to the in-memory file system_
> _2021-01-12 11:12:54,204 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_-9223372036854775184_1133, replicas=172.25.34.135:9866, 172.25.38.155:9866, 172.25.37.147:9866, 172.25.35.145:9866, 172.25.40.139:9866 for /ec1/hosts4._COPYING__
>
> _2021-01-12 11:12:54,288 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block blk_-9223372036854775184_1133 on 172.25.34.135:9866 size 606 replicaState = FINALIZED_
> _2021-01-12 11:12:54,291 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block blk_-9223372036854775181_1133 on 172.25.35.145:9866 size 606 replicaState = FINALIZED_
> _2021-01-12 11:12:54,293 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block blk_-9223372036854775180_1133 on 172.25.40.139:9866 size 606 replicaState = FINALIZED_
>
> This issue can be easily reproduced in HADOOP 3.0.x and 3.1.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org