You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by "Marton Elek (Jira)" <ji...@apache.org> on 2019/11/21 10:45:00 UTC
[jira] [Created] (HDDS-2606) Acceptance tests are flaky due to
async pipeline creation
Marton Elek created HDDS-2606:
---------------------------------
Summary: Acceptance tests are flaky due to async pipeline creation
Key: HDDS-2606
URL: https://issues.apache.org/jira/browse/HDDS-2606
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: SCM
Reporter: Marton Elek
Some of the acceptance tests are failing because the first Ratis.THREE pipeline is not created on time:
For example in a HA proxy acceptance test the first block allocation is tarted before moving the Ratis.THREE pipeline to OPEN state.
{code:java}
scm_1 | 2019-11-20 14:45:38,359 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 4d27f405-d257-4b1b-a7b3-51bbd57356db, Nodes: 2c010ef7-8c0e-45e3-b230-326bf759773b{ip: 172.25.0.7, host: ozones3-haproxy_datanode_2.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] moved to OPEN state
scm_1 | 2019-11-20 14:45:46,944 WARN block.BlockManagerImpl: Pipeline creation failed for type:RATIS factor:THREE. Retrying get pipelines call once.
scm_1 | org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot create pipeline of factor 3 using 0 nodes.
scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:153)
scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:58)
scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:155)
scm_1 | at org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:198)
scm_1 | at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:187)
scm_1 | at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:159)
scm_1 | at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
scm_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
scm_1 | at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:98)
scm_1 | at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:13157)
scm_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
scm_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
scm_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
scm_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
scm_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
scm_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
scm_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
scm_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
scm_1 | 2019-11-20 14:45:46,944 INFO block.BlockManagerImpl: Could not find available pipeline of type:RATIS and factor:THREE even after retrying
scm_1 | 2019-11-20 14:45:46,945 ERROR block.BlockManagerImpl: Unable to allocate a block for the size: 268435456, type: RATIS, factor: THREE
scm_1 | 2019-11-20 14:45:49,274 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: e208588c-9a16-4519-89dc-7bad5bae4331, Nodes: 1da10d32-12be-4328-bc0a-f3d8de21b056{ip: 172.25.0.8, host: ozones3-haproxy_datanode_3.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}d9edb776-ee02-48a1-8c73-40f33bc0d128{ip: 172.25.0.5, host: ozones3-haproxy_datanode_1.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}2c010ef7-8c0e-45e3-b230-326bf759773b{ip: 172.25.0.7, host: ozones3-haproxy_datanode_2.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] moved to OPEN state {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org