You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2011/12/13 04:37:30 UTC
[jira] [Created] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
-------------------------------------------------------------------------------------------------
Key: HDFS-2668
URL: https://issues.apache.org/jira/browse/HDFS-2668
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.23.0
Reporter: Todd Lipcon
I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
{code}
// Ignore replicas already scheduled to be removed from the DN
if(invalidateBlocks.contains(dn.getStorageID(), block)) {
assert storedBlock.findDatanode(dn) < 0 : "Block " + block
+ " in recentInvalidatesSet should not appear in DN " + dn;
{code}
The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2668:
--------------------------------------
Attachment: TestToReproduceHDFS-2668.patch
> Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-2668
> URL: https://issues.apache.org/jira/browse/HDFS-2668
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: TestToReproduceHDFS-2668.patch
>
>
> I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
> {code}
> // Ignore replicas already scheduled to be removed from the DN
> if(invalidateBlocks.contains(dn.getStorageID(), block)) {
> assert storedBlock.findDatanode(dn) < 0 : "Block " + block
> + " in recentInvalidatesSet should not appear in DN " + dn;
> {code}
> The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171910#comment-13171910 ]
Uma Maheswara Rao G commented on HDFS-2668:
-------------------------------------------
I make sure the issue: i just added throws RuntimeException in processReportedBlock
in BlockManager#processReportedBlock
{code}
// Ignore replicas already scheduled to be removed from the DN
if(invalidateBlocks.contains(dn.getStorageID(), block)) {
assert storedBlock.findDatanode(dn) < 0 : "Block " + block
+ " in recentInvalidatesSet should not appear in DN " + dn;
if(storedBlock.findDatanode(dn) >= 0)
throw new RuntimeException("Block already added into invalidateBlocks. But still this block associated with DN.... storedBlock.findDatanode(dn) =" + storedBlock.findDatanode(dn));
return storedBlock;
}
{code}
After this i ran the above attached Test.
Below are the logs that proves the issue.
2011-12-18 23:02:42,066 INFO FSNamesystem.audit (FSNamesystem.java:logAuditEvent(220)) - ugi=uma (auth:SIMPLE) ip=/127.0.0.1 cmd=open src=/tmp/testBadBlockReportOnTransfer/file1 dst=null perm=null
All blocks of file /tmp/testBadBlockReportOnTransfer/file1 verified to have replication factor 3
2011-12-18 23:02:42,073 INFO blockmanagement.BlockManager (BlockManager.java:setReplication(1814)) - Decreasing replication from 3 to 1 for /tmp/testBadBlockReportOnTransfer/file1
2011-12-18 23:02:42,073 INFO hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_5137102758256792519_1001 to 127.0.0.1:54432
2011-12-18 23:02:42,073 INFO hdfs.StateChange (BlockManager.java:chooseExcessReplicates(1954)) - BLOCK* chooseExcessReplicates: (127.0.0.1:54432, blk_5137102758256792519_1001) is added to recentInvalidateSets
2011-12-18 23:02:42,073 INFO hdfs.StateChange (InvalidateBlocks.java:add(77)) - BLOCK* InvalidateBlocks: add blk_5137102758256792519_1001 to 127.0.0.1:54418
2011-12-18 23:02:42,073 INFO hdfs.StateChange (BlockManager.java:chooseExcessReplicates(1954)) - BLOCK* chooseExcessReplicates: (127.0.0.1:54418, blk_5137102758256792519_1001) is added to recentInvalidateSets
2011-12-18 23:02:42,076 INFO FSNamesystem.audit (FSNamesystem.java:logAuditEvent(220)) - ugi=uma (auth:SIMPLE) ip=/127.0.0.1 cmd=setReplication src=/tmp/testBadBlockReportOnTransfer/file1 dst=null perm=null
..................
...................
2011-12-18 23:02:43,343 WARN datanode.DataNode (BPOfferService.java:offerService(537)) - RemoteException in offerService
java.lang.RuntimeException: java.lang.RuntimeException: Block already added into invalidateBlocks. But still this block associated with DN.... storedBlock.findDatanode(dn) =1
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:1498)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1418)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1328)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1303)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:847)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:130)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:16189)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:417)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:834)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1605)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1)
at java.security.AccessController.doPrivileged(Native Method)
> Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-2668
> URL: https://issues.apache.org/jira/browse/HDFS-2668
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: TestToReproduceHDFS-2668.patch
>
>
> I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
> {code}
> // Ignore replicas already scheduled to be removed from the DN
> if(invalidateBlocks.contains(dn.getStorageID(), block)) {
> assert storedBlock.findDatanode(dn) < 0 : "Block " + block
> + " in recentInvalidatesSet should not appear in DN " + dn;
> {code}
> The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171904#comment-13171904 ]
Hadoop QA commented on HDFS-2668:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507837/TestToReproduceHDFS-2668.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 90 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.security.token.block.TestBlockToken
org.apache.hadoop.hdfs.TestFileAppend2
org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
org.apache.hadoop.hdfs.security.TestDelegationToken
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1725//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1725//console
This message is automatically generated.
> Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-2668
> URL: https://issues.apache.org/jira/browse/HDFS-2668
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: TestToReproduceHDFS-2668.patch
>
>
> I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
> {code}
> // Ignore replicas already scheduled to be removed from the DN
> if(invalidateBlocks.contains(dn.getStorageID(), block)) {
> assert storedBlock.findDatanode(dn) < 0 : "Block " + block
> + " in recentInvalidatesSet should not appear in DN " + dn;
> {code}
> The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2668:
--------------------------------------
Status: Patch Available (was: Open)
Attached the Test patch, which should reproduce the issue.
I will remove the wrong assertion in BlockManager with next patch.
> Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-2668
> URL: https://issues.apache.org/jira/browse/HDFS-2668
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: TestToReproduceHDFS-2668.patch
>
>
> I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
> {code}
> // Ignore replicas already scheduled to be removed from the DN
> if(invalidateBlocks.contains(dn.getStorageID(), block)) {
> assert storedBlock.findDatanode(dn) < 0 : "Block " + block
> + " in recentInvalidatesSet should not appear in DN " + dn;
> {code}
> The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Posted by "Uma Maheswara Rao G (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225086#comment-13225086 ]
Uma Maheswara Rao G commented on HDFS-2668:
-------------------------------------------
I see that this particular assertion already commented in code by refering this issue.
{code}
/* TODO: following assertion is incorrect, see HDFS-2668
assert storedBlock.findDatanode(dn) < 0 : "Block " + block
+ " in recentInvalidatesSet should not appear in DN " + dn; */
{code}
I think we can remove that commented code(assertion) completely right?
> Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-2668
> URL: https://issues.apache.org/jira/browse/HDFS-2668
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: TestToReproduceHDFS-2668.patch
>
>
> I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
> {code}
> // Ignore replicas already scheduled to be removed from the DN
> if(invalidateBlocks.contains(dn.getStorageID(), block)) {
> assert storedBlock.findDatanode(dn) < 0 : "Block " + block
> + " in recentInvalidatesSet should not appear in DN " + dn;
> {code}
> The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2668) Incorrect assertion in BlockManager
when block report arrives shortly after invalidation decision
Posted by "Uma Maheswara Rao G (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uma Maheswara Rao G updated HDFS-2668:
--------------------------------------
Status: Open (was: Patch Available)
> Incorrect assertion in BlockManager when block report arrives shortly after invalidation decision
> -------------------------------------------------------------------------------------------------
>
> Key: HDFS-2668
> URL: https://issues.apache.org/jira/browse/HDFS-2668
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
> Attachments: TestToReproduceHDFS-2668.patch
>
>
> I haven't written a test case to verify this yet, but I believe the following assertion is incorrect:
> {code}
> // Ignore replicas already scheduled to be removed from the DN
> if(invalidateBlocks.contains(dn.getStorageID(), block)) {
> assert storedBlock.findDatanode(dn) < 0 : "Block " + block
> + " in recentInvalidatesSet should not appear in DN " + dn;
> {code}
> The problem is that, when a block is invalidated due to over-replication, it is not immediately removed from the block map. So, if a block report arrives just after a block has been marked as invalidated, but before the block is actually deleted, I think this assertion will trigger incorrectly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira