You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "David (Jira)" <ji...@apache.org> on 2021/01/12 11:43:00 UTC
[jira] [Created] (HDFS-15773) NameNode allocate five DataNodes for
a small file (606 bytes) in log file
David created HDFS-15773:
----------------------------
Summary: NameNode allocate five DataNodes for a small file (606 bytes) in log file
Key: HDFS-15773
URL: https://issues.apache.org/jira/browse/HDFS-15773
Project: Hadoop HDFS
Issue Type: Bug
Reporter: David
For testing Erasure Coding, I put a small file "hosts4" (606 bytes) to a EC enabled HDFS directory.
. In the output of the fsck on this file, there are three replicas which is not surprise cause it's small than 1M.
_/ec1/hosts4 606 bytes, erasure-coded: policy=RS-3-2-1024k, 1 block(s): OK_
_0. BP-1329090504-172.25.37.132-1610344959454:blk_-9223372036854775184_1133 len=606 Live_repl=3 [blk_-9223372036854775184:DatanodeInfoWithStorage[172.25.34.135:9866,DS-b82b3dd4-cf57-4be6-9a0e-21089cf01d17,DISK], blk_-9223372036854775181:DatanodeInfoWithStorage[172.25.35.145:9866,DS-ef58c1c4-b54b-4321-96e5-eb39dd8e21cd,DISK], blk_-9223372036854775180:DatanodeInfoWithStorage[172.25.40.139:9866,DS-a160a4ae-ac88-4103-b19c-dc0f3fc0f861,DISK]]_
. But in the NameNode log file, there were five DataNodes have been selected. However, only three DNs reported FINALIZED status, and there is a gap on the block IDs.
_2021-01-12 11:12:54,204 DEBUG org.apache.hadoop.hdfs.StateChange: DIR* FSDirectory.addBlock: /ec1/hosts4._COPYING_ with blk_-9223372036854775184_1133 block is added to the in-memory file system_
_2021-01-12 11:12:54,204 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_-9223372036854775184_1133, replicas=172.25.34.135:9866, 172.25.38.155:9866, 172.25.37.147:9866, 172.25.35.145:9866, 172.25.40.139:9866 for /ec1/hosts4._COPYING__
_2021-01-12 11:12:54,288 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block blk_-9223372036854775184_1133 on 172.25.34.135:9866 size 606 replicaState = FINALIZED_
_2021-01-12 11:12:54,291 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block blk_-9223372036854775181_1133 on 172.25.35.145:9866 size 606 replicaState = FINALIZED_
_2021-01-12 11:12:54,293 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Reported block blk_-9223372036854775180_1133 on 172.25.40.139:9866 size 606 replicaState = FINALIZED_
This issue can be easily reproduced in HADOOP 3.0.x and 3.1.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org