You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Gabor Bota (JIRA)" <ji...@apache.org> on 2018/01/25 10:31:00 UTC
[jira] [Commented] (HDFS-13065)
TestErasureCodingMultipleRacks#testSkewedRack3 is failing
[ https://issues.apache.org/jira/browse/HDFS-13065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16339040#comment-16339040 ]
Gabor Bota commented on HDFS-13065:
-----------------------------------
The testSkewedRack3 test checks if a block has the redundancy needed for erasure coding after creating a file. The test does this check 10 times, each time with a different test file. During the iteration, the test checks whether the blocks are replicated 9 times - on 9 different hosts.
The issue seems to be that the check is done without a delay or a retry:
{code:java}
DFSTestUtil.writeFile(dfs, path, contents);
BlockLocation[] blocks =
dfs.getFileBlockLocations(path, 0, Long.MAX_VALUE);
assertEquals(ecPolicy.getNumDataUnits() + ecPolicy.getNumParityUnits(),
blocks[0].getHosts().length);
{code}
Adding a retry with 10ms delay between the {{writeFile}} and {{getFileBlockLocations}} eliminates the issue and during my tests, the maximum number of retries was 3, so that means after ~30ms delay the issue is gone.
A solution to eliminate this issue could be to add a retry with a 50ms delay, with a low maximal number of retries.
If such a delay cannot be accepted during this test, then further debugging, maybe optimization needed.
> TestErasureCodingMultipleRacks#testSkewedRack3 is failing
> ---------------------------------------------------------
>
> Key: HDFS-13065
> URL: https://issues.apache.org/jira/browse/HDFS-13065
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.0.0
> Reporter: Gabor Bota
> Assignee: Gabor Bota
> Priority: Major
> Fix For: 3.0.1
>
>
> Test fails intermittently.
> java.lang.AssertionError: expected:<9> but was:<7>
> at org.apache.hadoop.hdfs.TestErasureCodingMultipleRacks.testSkewedRack3(TestErasureCodingMultipleRacks.java:178)
> 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG net.NetworkTopology (DFSNetworkTopology.java:chooseRandomWithStorageTypeTwoTrial(119)) - No node to choose.
> 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] WARN blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(142)) - Only able to place 7 of total expected 9 (maxNodesPerRack=2, numOfReplicas=4) nodes evenly across racks, falling back to evenly place on the remaining racks. This may not guarantee rack-level fault tolerance. Please check if the racks are configured properly.
> 2017-12-12 20:25:46,751 [IPC Server handler 6 on 47378] DEBUG blockmanagement.BlockPlacementPolicy (BlockPlacementPolicyRackFaultTolerant.java:chooseTargetInOrder(148)) - Caught exception was:
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException:
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:792)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseOnce(BlockPlacementPolicyRackFaultTolerant.java:224)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant.chooseTargetInOrder(BlockPlacementPolicyRackFaultTolerant.java:139)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:392)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:268)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:121)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:137)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2091)
> at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:287)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2602)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:864)
> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:549)
> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org