You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Shashikant Banerjee (Jira)" <ji...@apache.org> on 2020/04/02 09:12:00 UTC
[jira] [Commented] (HDDS-3314) scmcli container info command
failing intermittently
[ https://issues.apache.org/jira/browse/HDDS-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073539#comment-17073539 ]
Shashikant Banerjee commented on HDDS-3314:
-------------------------------------------
The issue seems to be happening bcoz for reading container it uses ratis pipeline to read which only reads from the leader. In cases where the two datanodes are shutdown where one of them can potentially be the leader, the readContainer call won't retry on a different datanode. While reading container it should use the grpc client not the ratis client.
{code:java}
diff --git a/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java b/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
index 15f6ce82c..8fe5ef799 100644
--- a/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
+++ b/hadoop-hdds/tools/src/main/java/org/apache/hadoop/hdds/scm/cli/ContainerOperationClient.java
@@ -377,7 +377,7 @@ public ContainerDataProto readContainer(long containerID,
Pipeline pipeline) throws IOException {
XceiverClientSpi client = null;
try {
- client = xceiverClientManager.acquireClient(pipeline);
+ client = xceiverClientManager.acquireClientForReadData(pipeline);
ReadContainerResponseProto response =
ContainerProtocolCalls.readContainer(client, containerID, null);
if (LOG.isDebugEnabled()) {
{code}
[~nilotpalnandi], would you like to take this one up?
> scmcli container info command failing intermittently
> -----------------------------------------------------
>
> Key: HDDS-3314
> URL: https://issues.apache.org/jira/browse/HDDS-3314
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM, SCM Client
> Reporter: Nilotpal Nandi
> Priority: Major
>
> config set before running the command :
> "ozone.scm.stale.node.interval": "2m",
> "ozone.scm.dead.node.interval": "4m",
> "hdds.scm.replication.thread.interval": "12s",
> "ozone.scm.container.size": "1GB"
>
> steps taken :
> 1) write a key (less than a block size)
> 2) shutdown two container replica datanodes.
> 3) Tried to query container info
> Container info command failed .
>
>
> {noformat}
> ozone scmcli container info 33 | egrep 'Container|Datanodes'
> Failed to execute command cmdType: ReadContainer
> {noformat}
>
> scm log during that time range :
> {noformat}
> 2020-04-01 10:09:29,665 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for hrt_qa@ROOT.HWX.SITE (auth:KERBEROS)
> 2020-04-01 10:09:29,706 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for hrt_qa@ROOT.HWX.SITE (auth:KERBEROS) for protocol=interface org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol
> 2020-04-01 10:09:55,283 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/quasar-fjgcwr-2.quasar-fjgcwr.root.hwx.site@ROOT.HWX.SITE (auth:KERBEROS)
> 2020-04-01 10:09:55,287 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for dn/quasar-fjgcwr-2.quasar-fjgcwr.root.hwx.site@ROOT.HWX.SITE (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol
> 2020-04-01 10:09:55,474 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Starting Replication Monitor Thread.
> 2020-04-01 10:09:55,486 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor Thread took 10 milliseconds for processing 33 containers.
> 2020-04-01 10:10:07,488 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor Thread took 2 milliseconds for processing 33 containers.
> 2020-04-01 10:10:17,996 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for dn/quasar-fjgcwr-7.quasar-fjgcwr.root.hwx.site@ROOT.HWX.SITE (auth:KERBEROS)
> 2020-04-01 10:10:18,001 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successful for dn/quasar-fjgcwr-7.quasar-fjgcwr.root.hwx.site@ROOT.HWX.SITE (auth:KERBEROS) for protocol=interface org.apache.hadoop.ozone.protocol.StorageContainerDatanodeProtocol
> 2020-04-01 10:10:19,491 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor Thread took 3 milliseconds for processing 33 containers.
> 2020-04-01 10:10:31,494 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor Thread took 2 milliseconds for processing 33 containers.
> 2020-04-01 10:10:43,495 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Replication Monitor Thread took 1 milliseconds for processing 33 containers.
> 2020-04-01 10:10:47,987 ERROR org.apache.hadoop.hdds.scm.pipeline.PipelineActionHandler: Received pipeline action CLOSE for Pipeline[ Id: 763bd379-a703-4dc0-85c5-bf385cdc0b18, Nodes: 92f73ec3-9ed8-41c8-9103-c4c1b2b365e1{ip: 172.27.120.0, host: quasar-fjgcwr-1.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: null}b60097cf-7dff-44dc-800f-3500dda636f6{ip: 172.27.123.128, host: quasar-fjgcwr-4.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: null}ea2322d9-8ede-4f48-a72d-693e809d2b95{ip: 172.27.12.195, host: quasar-fjgcwr-7.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:b60097cf-7dff-44dc-800f-3500dda636f6, CreationTimestamp2020-04-01T10:04:47.723688Z] from datanode ea2322d9-8ede-4f48-a72d-693e809d2b95{ip: 172.27.12.195, host: quasar-fjgcwr-7.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: 12651664310640168}. Reason : ea2322d9-8ede-4f48-a72d-693e809d2b95 is in candidate state for 61616ms
> 2020-04-01 10:10:47,988 INFO org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: Destroying pipeline:Pipeline[ Id: 763bd379-a703-4dc0-85c5-bf385cdc0b18, Nodes: 92f73ec3-9ed8-41c8-9103-c4c1b2b365e1{ip: 172.27.120.0, host: quasar-fjgcwr-1.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: null}b60097cf-7dff-44dc-800f-3500dda636f6{ip: 172.27.123.128, host: quasar-fjgcwr-4.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: null}ea2322d9-8ede-4f48-a72d-693e809d2b95{ip: 172.27.12.195, host: quasar-fjgcwr-7.quasar-fjgcwr.root.hwx.site, networkLocation: /default-rack, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:b60097cf-7dff-44dc-800f-3500dda636f6, CreationTimestamp2020-04-01T10:04:47.723688Z]{noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org