Block is not getting deleting which is not present in Memory in following scenario

[Brahma Reddy Battula <br...@huawei.com>]

Hi

Started Name Node and Data Node by configuring multiple DataDirs
Written one file
copied same block to another dataDir

What I expect is,Block will be removed from copied Dir since it's not present memory.

But here it's keep on logging following for every directory scan ...Please let me know the behavior...

2012-06-18 16:30:55,152 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Block file /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/blk_92866283154406728 does not match file found by scan /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/subdir9/blk_92866283154406728
2012-06-18 16:35:55,152 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1134724852-*********-1339770849071 Total blocks: 389, missing metadata files:0, missing block files:0, missing blocks in memory:1, mismatched blocks:0
2012-06-18 16:35:55,152 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Block file /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/blk_92866283154406728 does not match file found by scan /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/subdir9/blk_92866283154406728

RE: Block is not getting deleting which is not present in Memory in following scenario

[Amith D K <am...@huawei.com>]

Hi Brahma

To my understanding Directory scanner will the Block to the DN memory if it is not present.

If u try to add a new Block manually to a new DN Directory scanner will add the block to DN memory,
when DN sends a block report this will be sent to NN and since it is invalid NN will invalidate the block.

In your case u are just copying the Block from 1 DataDir1 to DataDir2, as expected it has updated its memory,
but I am not clear will NN invalidate it.

Amith



________________________________

From: Brahma Reddy Battula [brahmareddy.battula@huawei.com]
Sent: Monday, June 18, 2012 7:39 PM
To: hdfs-user@hadoop.apache.org
Subject: Block is not getting deleting which is not present in Memory in following scenario

Hi

Started Name Node and Data Node by configuring multiple DataDirs
Written one file
copied same block to another dataDir

What I expect is,Block will be removed from copied Dir since it's not present memory.

But here it's keep on logging following for every directory scan ...Please let me know the behavior...

2012-06-18 16:30:55,152 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Block file /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/blk_92866283154406728 does not match file found by scan /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/subdir9/blk_92866283154406728
2012-06-18 16:35:55,152 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1134724852-*********-1339770849071 Total blocks: 389, missing metadata files:0, missing block files:0, missing blocks in memory:1, mismatched blocks:0
2012-06-18 16:35:55,152 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Block file /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/blk_92866283154406728 does not match file found by scan /opt/dfs12/current/BP-1134724852-*********-1339770849071/current/finalized/subdir9/blk_92866283154406728