You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Brian Bockelman (JIRA)" <ji...@apache.org> on 2008/10/30 00:58:44 UTC

[jira] Created: (HADOOP-4543) Namenode does not recognize incorrectly sized blocks

Namenode does not recognize incorrectly sized blocks
----------------------------------------------------

                 Key: HADOOP-4543
                 URL: https://issues.apache.org/jira/browse/HADOOP-4543
             Project: Hadoop Core
          Issue Type: Bug
    Affects Versions: 0.18.1
            Reporter: Brian Bockelman


We had a lot of file system corruption resulting in incorrectly sized blocks (on disk, they're truncated to 192KB when they should be 64MB).

However, I cannot make Hadoop realize that these blocks are incorrectly sized.  When I try to drain off the node, I get the following messages:

2008-10-29 18:46:51,293 WARN org.apache.hadoop.fs.FSNamesystem: Inconsistent size for block blk_-4403534125663454855_9937 reported from 172.16.1.150:50010 current size is 67108864 reported size is 196608

Here 172.16.1.150 is not the node which has the problematic block, but the destination of the file transfer.  I propose that Hadoop should either:

a) Upon startup, make sure that all blocks are properly sized (pro: rather cheap check; con: doesn't catch any truncations which happen while on disk)
b) Upon detecting the incorrectly sized copy, Hadoop should ask the source of the block to perform a block verification.

Thanks,

Brian

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.