You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Mukul Kumar Singh (Jira)" <ji...@apache.org> on 2019/12/17 17:43:00 UTC
[jira] [Commented] (HDDS-2606) Acceptance tests are flaky due to
async pipeline creation
[ https://issues.apache.org/jira/browse/HDDS-2606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998416#comment-16998416 ]
Mukul Kumar Singh commented on HDDS-2606:
-----------------------------------------
[~nanda]& [~elek] is this same as HDDS-2679 ?
> Acceptance tests are flaky due to async pipeline creation
> ---------------------------------------------------------
>
> Key: HDDS-2606
> URL: https://issues.apache.org/jira/browse/HDDS-2606
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
> Reporter: Marton Elek
> Assignee: Marton Elek
> Priority: Major
> Attachments: docker-ozones3-haproxy-ozones3-haproxy-basic-scm.log, log.html, report.html, robot-ozones3-haproxy-ozones3-haproxy-basic-scm.xml, summary.html
>
>
> Some of the acceptance tests are failing because the first Ratis.THREE pipeline is not created on time:
> For example in a HA proxy acceptance test the first block allocation is tarted before moving the Ratis.THREE pipeline to OPEN state.
> {code:java}
> scm_1 | 2019-11-20 14:45:38,359 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: 4d27f405-d257-4b1b-a7b3-51bbd57356db, Nodes: 2c010ef7-8c0e-45e3-b230-326bf759773b{ip: 172.25.0.7, host: ozones3-haproxy_datanode_2.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:ONE, State:OPEN] moved to OPEN state
> scm_1 | 2019-11-20 14:45:46,944 WARN block.BlockManagerImpl: Pipeline creation failed for type:RATIS factor:THREE. Retrying get pipelines call once.
> scm_1 | org.apache.hadoop.hdds.scm.pipeline.InsufficientDatanodesException: Cannot create pipeline of factor 3 using 0 nodes.
> scm_1 | at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineProvider.create(RatisPipelineProvider.java:153)
> scm_1 | at org.apache.hadoop.hdds.scm.pipeline.PipelineFactory.create(PipelineFactory.java:58)
> scm_1 | at org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager.createPipeline(SCMPipelineManager.java:155)
> scm_1 | at org.apache.hadoop.hdds.scm.block.BlockManagerImpl.allocateBlock(BlockManagerImpl.java:198)
> scm_1 | at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.allocateBlock(SCMBlockProtocolServer.java:187)
> scm_1 | at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.allocateScmBlock(ScmBlockLocationProtocolServerSideTranslatorPB.java:159)
> scm_1 | at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
> scm_1 | at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
> scm_1 | at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:98)
> scm_1 | at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:13157)
> scm_1 | at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
> scm_1 | at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
> scm_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
> scm_1 | at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
> scm_1 | at java.base/java.security.AccessController.doPrivileged(Native Method)
> scm_1 | at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> scm_1 | at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> scm_1 | at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
> scm_1 | 2019-11-20 14:45:46,944 INFO block.BlockManagerImpl: Could not find available pipeline of type:RATIS and factor:THREE even after retrying
> scm_1 | 2019-11-20 14:45:46,945 ERROR block.BlockManagerImpl: Unable to allocate a block for the size: 268435456, type: RATIS, factor: THREE
> scm_1 | 2019-11-20 14:45:49,274 INFO pipeline.PipelineStateManager: Pipeline Pipeline[ Id: e208588c-9a16-4519-89dc-7bad5bae4331, Nodes: 1da10d32-12be-4328-bc0a-f3d8de21b056{ip: 172.25.0.8, host: ozones3-haproxy_datanode_3.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}d9edb776-ee02-48a1-8c73-40f33bc0d128{ip: 172.25.0.5, host: ozones3-haproxy_datanode_1.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}2c010ef7-8c0e-45e3-b230-326bf759773b{ip: 172.25.0.7, host: ozones3-haproxy_datanode_2.ozones3-haproxy_default, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN] moved to OPEN state {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org