You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (JIRA)" <ji...@apache.org> on 2012/10/17 06:54:03 UTC
[jira] [Created] (HDFS-4067) TestUnderReplicatedBlocks may fail due
to ReplicaAlreadyExistsException
Eli Collins created HDFS-4067:
---------------------------------
Summary: TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
Key: HDFS-4067
URL: https://issues.apache.org/jira/browse/HDFS-4067
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
{noformat}
org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483395#comment-13483395 ]
Jing Zhao commented on HDFS-4067:
---------------------------------
Run the testcase ~800 times and all of them passed.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484363#comment-13484363 ]
Hudson commented on HDFS-4067:
------------------------------
Integrated in Hadoop-trunk-Commit #2927 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/2927/])
HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to ReplicaAlreadyExistsException. Contributed by Jing Zhao. (Revision 1402261)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1402261
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Fix For: 3.0.0
>
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4067) TestUnderReplicatedBlocks may fail due
to ReplicaAlreadyExistsException
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HDFS-4067:
----------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I committed the patch. Thank you Jing.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Fix For: 3.0.0
>
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484937#comment-13484937 ]
Hudson commented on HDFS-4067:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1237 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1237/])
HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to ReplicaAlreadyExistsException. Contributed by Jing Zhao. (Revision 1402261)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1402261
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Fix For: 3.0.0
>
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482131#comment-13482131 ]
Jing Zhao commented on HDFS-4067:
---------------------------------
testcase failure reported in HDFS-3948 before. Will run TestUnderReplicatedBlocks in loop later.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480535#comment-13480535 ]
Jing Zhao commented on HDFS-4067:
---------------------------------
And I guess that's also the reason for HDFS-342? Since the initial replication request is ignored, the replication on D1 can only be done after the pending replication timeout.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484353#comment-13484353 ]
Suresh Srinivas commented on HDFS-4067:
---------------------------------------
+1 for the patch.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484895#comment-13484895 ]
Hudson commented on HDFS-4067:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1207/])
HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to ReplicaAlreadyExistsException. Contributed by Jing Zhao. (Revision 1402261)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1402261
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Fix For: 3.0.0
>
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480529#comment-13480529 ]
Jing Zhao commented on HDFS-4067:
---------------------------------
Move the discussion from HDFS-4061 here:
When the NameNode invalides a block for a datanode D1 and remove the datanode-block pair from the blockMap, and before the invalidation request is sent to the datanode D1, the BlockManager#computeDataNodeWork also starts to work and schedule the replication to D1. So the invalidation and replication request will be sent to D1 at the same time. D1 will then ignore the replication request (also throws a ReplicaAlreadyExistsException), and delete the replica. Thus NN cannot receive the blockreceived msg from D1. And the testcast will timeout in 5min which is smaller than the timeout of PendingReplication request (usually 5~10 min).
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13484845#comment-13484845 ]
Hudson commented on HDFS-4067:
------------------------------
Integrated in Hadoop-Yarn-trunk #17 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/17/])
HDFS-4067. TestUnderReplicatedBlocks intermittently fails due to ReplicaAlreadyExistsException. Contributed by Jing Zhao. (Revision 1402261)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1402261
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Fix For: 3.0.0
>
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482127#comment-13482127 ]
Hadoop QA commented on HDFS-4067:
---------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12550126/HDFS-4067.trunk.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3383//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3383//console
This message is automatically generated.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4067) TestUnderReplicatedBlocks may fail due
to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-4067:
----------------------------
Attachment: HDFS-4067.trunk.001.patch
Initial patch to fix.
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-4067) TestUnderReplicatedBlocks may fail
due to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao reassigned HDFS-4067:
-------------------------------
Assignee: Jing Zhao
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4067) TestUnderReplicatedBlocks may fail due
to ReplicaAlreadyExistsException
Posted by "Jing Zhao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-4067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jing Zhao updated HDFS-4067:
----------------------------
Status: Patch Available (was: Open)
> TestUnderReplicatedBlocks may fail due to ReplicaAlreadyExistsException
> -----------------------------------------------------------------------
>
> Key: HDFS-4067
> URL: https://issues.apache.org/jira/browse/HDFS-4067
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.0.0-alpha
> Reporter: Eli Collins
> Assignee: Jing Zhao
> Labels: test-fail
> Attachments: HDFS-4067.trunk.001.patch
>
>
> After adding the timeout to TestUnderReplicatedBlocks in HDFS-4061 we can see the root cause of the failure is ReplicaAlreadyExistsException:
> {noformat}
> org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Block BP-1541130889-172.29.121.238-1350435573411:blk_-3437032108997618258_1002 already exists in state FINALIZED and thus cannot be created.
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:799)
> at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createTemporary(FsDatasetImpl.java:90)
> at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:155)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
> at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
> at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira