You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2021/05/19 11:23:52 UTC
[GitHub] [ozone] lokeshj1703 opened a new pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
lokeshj1703 opened a new pull request #2267:
URL: https://github.com/apache/ozone/pull/2267
## What changes were proposed in this pull request?
After running decommissioning tests and aborting few decommissioning command, seeing PipelineNotFoundException INFO logs continuously in scm log.
Although these are INFO logs, raising the issue as the exceptions are thrown in loop.
```
2021-05-16 20:06:53,421 INFO org.apache.hadoop.ipc.Server: IPC Server handler 53 on 9860, call Call#102 Retry#23 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
2021-05-16 20:06:55,426 INFO org.apache.hadoop.ipc.Server: IPC Server handler 84 on 9860, call Call#102 Retry#24 org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest from 172.27.176.196:41982
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException: PipelineID=b1431673-adf0-402f-968f-e9241ea37609 not found
at org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:121)
at org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerV2Impl.getPipeline(PipelineStateManagerV2Impl.java:125)
at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:83)
at org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:68)
at com.sun.proxy.$Proxy14.getPipeline(Unknown Source)
at org.apache.hadoop.hdds.scm.pipeline.PipelineManagerV2Impl.getPipeline(PipelineManagerV2Impl.java:198)
at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:529)
at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:497)
at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:302)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:153)
at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:46400)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:986)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:914)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2887)
Options
````
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5248
## How was this patch tested?
Adds UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#discussion_r636687565
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineNotFoundException.java
##########
@@ -18,18 +18,18 @@
package org.apache.hadoop.hdds.scm.pipeline;
-import java.io.IOException;
+import org.apache.hadoop.hdds.scm.exceptions.SCMException;
/**
* Signals that a pipeline is missing from PipelineManager.
*/
-public class PipelineNotFoundException extends IOException{
+public class PipelineNotFoundException extends SCMException {
/**
* Constructs an {@code PipelineNotFoundException} with {@code null}
* as its error detail message.
*/
public PipelineNotFoundException() {
- super();
+ super(ResultCodes.PIPELINE_NOT_FOUND);
Review comment:
RemoteException unwrapException code and instantiateException, only message will be saved.
```public IOException unwrapRemoteException() {
try {
Class<?> realClass = Class.forName(getClassName());
return instantiateException(realClass.asSubclass(IOException.class));
} catch(Exception e) {
// cannot instantiate the original exception, just return this
}
return this;
}
private IOException instantiateException(Class<? extends IOException> cls)
throws Exception {
Constructor<? extends IOException> cn = cls.getConstructor(String.class);
cn.setAccessible(true);
IOException ex = cn.newInstance(this.getMessage());
ex.initCause(this);
return ex;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#issuecomment-846770133
@bharatviswa504 Thanks for the review! I have committed the PR to master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#discussion_r636677245
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineNotFoundException.java
##########
@@ -18,18 +18,18 @@
package org.apache.hadoop.hdds.scm.pipeline;
-import java.io.IOException;
+import org.apache.hadoop.hdds.scm.exceptions.SCMException;
/**
* Signals that a pipeline is missing from PipelineManager.
*/
-public class PipelineNotFoundException extends IOException{
+public class PipelineNotFoundException extends SCMException {
/**
* Constructs an {@code PipelineNotFoundException} with {@code null}
* as its error detail message.
*/
public PipelineNotFoundException() {
- super();
+ super(ResultCodes.PIPELINE_NOT_FOUND);
Review comment:
Looks like this ResultCode is lost when received at client.
But I see the message with whole stacktrace with debug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] bharatviswa504 commented on a change in pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#discussion_r636686401
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineNotFoundException.java
##########
@@ -18,18 +18,18 @@
package org.apache.hadoop.hdds.scm.pipeline;
-import java.io.IOException;
+import org.apache.hadoop.hdds.scm.exceptions.SCMException;
/**
* Signals that a pipeline is missing from PipelineManager.
*/
-public class PipelineNotFoundException extends IOException{
+public class PipelineNotFoundException extends SCMException {
/**
* Constructs an {@code PipelineNotFoundException} with {@code null}
* as its error detail message.
*/
public PipelineNotFoundException() {
- super();
+ super(ResultCodes.PIPELINE_NOT_FOUND);
Review comment:
Yes. I believe the ResultCodes in proto will help when we want to return ResultCodes in Response and construct the Exception based on that.
I believe when Rpc Layer constructs this Exception it used IOException default constructor and the ResultCode will be lost, and the message will be present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on a change in pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on a change in pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#discussion_r636684226
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineNotFoundException.java
##########
@@ -18,18 +18,18 @@
package org.apache.hadoop.hdds.scm.pipeline;
-import java.io.IOException;
+import org.apache.hadoop.hdds.scm.exceptions.SCMException;
/**
* Signals that a pipeline is missing from PipelineManager.
*/
-public class PipelineNotFoundException extends IOException{
+public class PipelineNotFoundException extends SCMException {
/**
* Constructs an {@code PipelineNotFoundException} with {@code null}
* as its error detail message.
*/
public PipelineNotFoundException() {
- super();
+ super(ResultCodes.PIPELINE_NOT_FOUND);
Review comment:
The latest commit adds the result code in the protobuf. Did you see that behavior in latest commit @bharatviswa504 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#issuecomment-844998198
@bharatviswa504 @bshashikant As per discussion, I have updated the PR.
Some APIs throw IOException directly. Those APIs will still need to be handled separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 commented on pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 commented on pull request #2267:
URL: https://github.com/apache/ozone/pull/2267#issuecomment-845750986
HDDS-5259 has been created to handle the IOException being thrown by the apis in StorageContainerLocationProtocol.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] lokeshj1703 closed pull request #2267: HDDS-5248. SCM HA: Continuous PipelineNotFoundException seen in SCM log.
Posted by GitBox <gi...@apache.org>.
lokeshj1703 closed pull request #2267:
URL: https://github.com/apache/ozone/pull/2267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org