You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/07/21 09:53:48 UTC
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
maobaolong opened a new pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228
## What changes were proposed in this pull request?
Fix s3g met NPE exception while write file by multiPartUpload
## What is the link to the Apache JIRA
HDDS-3995
## How was this patch tested?
Send a put request with an uploadId.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r458562528
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
BTW, I find OM met NPE too, HDDS-3999 it is also related the `commitUploadPartInfo`, not sure the relation between these two issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#issuecomment-708134370
@GlenGeng @bharatviswa504 Thanks for your suggestion, I push a new commit to fix this.
PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r458516884
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
> Do we know why omMultipartCommitUploadPartInfo is null in this case
As `IOUtils.closeQuietly(ozoneOutputStream)` called in the finally block of the method createMultipartKey, every IOException has been swallowed by the closeQuietly, so when closing ozoneOutputStream, maybe there are some exception occurred.
And the close method of KeyOutputStream, we can see `blockOutputStreamEntryPool.commitKey` is called here.
```java
public void close() throws IOException {
if (closed) {
return;
}
closed = true;
try {
handleFlushOrClose(StreamAction.CLOSE);
if (!isException) {
Preconditions.checkArgument(writeOffset == offset);
}
blockOutputStreamEntryPool.commitKey(offset);
} finally {
blockOutputStreamEntryPool.cleanup();
}
}
```
Lets guess, if `commitMultipartUploadPart` failed with an IOException, `commitUploadPartInfo` would be `null`, so the NPE occurred.
```java
void commitKey(long offset) throws IOException {
if (keyArgs != null) {
// in test, this could be null
long length = getKeyLength();
Preconditions.checkArgument(offset == length);
keyArgs.setDataSize(length);
keyArgs.setLocationInfoList(getLocationInfoList());
// When the key is multipart upload part file upload, we should not
// commit the key, as this is not an actual key, this is a just a
// partial key of a large file.
if (keyArgs.getIsMultipartKey()) {
commitUploadPartInfo =
omClient.commitMultipartUploadPart(keyArgs, openID);
} else {
omClient.commitKey(keyArgs, openID);
}
} else {
LOG.warn("Closing KeyOutputStream, but key args is null");
}
}
```
> we should return an proper s3 error code from there.
Sorry, i'm not very familiar with S3, do you have any idea and suggestion about the return error type
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r459188437
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
@bharatviswa504 Thanks for your reply, look forward to your feedback about this PR, I think this PR is also needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r503653865
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Hey @bharatviswa504
Seems this NPE in S3G occurred again, I got some clue, but haven't found the root cause yet.
OM side, `S3MultipartUploadCommitPartRequest` failed due to upload id not found.
S3G side,
```
} finally {
IOUtils.closeQuietly(ozoneOutputStream);
}
```
`OzoneOutputStream.close()` will call `OmMultipartCommitUploadPartInfo commitMultipartUploadPart(
OmKeyArgs omKeyArgs, long clientID) throws IOException;` to commit the key, but `IOUtils.closeQuietly()` will swallow IOException that relates to this error, thus
```
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
```
will return null, trigger the NPE issue.
OM Trace
> 2020-10-10 21:29:26,644 ERROR org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest: MultipartUpload Commit is failed for Key:python_1_1G.dat in Volume/Bucket s325d55ad283aa400af464c76d713c07ad/pythonbucket
NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload is with specified uploadId fc5051bf-97a7-454b-9761-18669f7d3b02-105010697301721222
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:174)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:224)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:145)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:113)
at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
S3G Trace
> 2020-10-10 21:29:26,544 WARN org.apache.ratis.metrics.impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
2020-10-10 21:29:26,648 WARN org.eclipse.jetty.server.HttpChannel: /pythonbucket/python_1_1G.dat
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: java.lang.NullPointerException
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
... 17 more
... 17 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:606)
at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:156)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
How about we avoid call `IOUtils.closeQuietly(ozoneOutputStream);` to throw out the `NO_SUCH_MULTIPART_UPLOAD_ERROR` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r503653865
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Hey @bharatviswa504
Seems this NPE in S3G occurred again(in the latest master), I got some clue, but haven't found the root cause yet.
OM side, `S3MultipartUploadCommitPartRequest` failed due to upload id not found.
S3G side,
```
} finally {
IOUtils.closeQuietly(ozoneOutputStream);
}
```
`OzoneOutputStream.close()` will call `OmMultipartCommitUploadPartInfo commitMultipartUploadPart(
OmKeyArgs omKeyArgs, long clientID) throws IOException;` to commit the key, but `IOUtils.closeQuietly()` will swallow IOException that relates to this error, thus
```
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
```
will return null, trigger the NPE issue.
OM Trace
> 2020-10-10 21:29:26,644 ERROR org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest: MultipartUpload Commit is failed for Key:python_1_1G.dat in Volume/Bucket s325d55ad283aa400af464c76d713c07ad/pythonbucket
NO_SUCH_MULTIPART_UPLOAD_ERROR org.apache.hadoop.ozone.om.exceptions.OMException: No such Multipart upload is with specified uploadId fc5051bf-97a7-454b-9761-18669f7d3b02-105010697301721222
at org.apache.hadoop.ozone.om.request.s3.multipart.S3MultipartUploadCommitPartRequest.validateAndUpdateCache(S3MultipartUploadCommitPartRequest.java:174)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestDirectlyToOM(OzoneManagerProtocolServerSideTranslatorPB.java:224)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:145)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:113)
at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:999)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:927)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2915)
S3G Trace
> 2020-10-10 21:29:26,544 WARN org.apache.ratis.metrics.impl.MetricRegistriesImpl: First MetricRegistry has been created without registering reporters. You may need to call MetricRegistries.global().addReportRegistration(...) before.
2020-10-10 21:29:26,648 WARN org.eclipse.jetty.server.HttpChannel: /pythonbucket/python_1_1G.dat
javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.servlet.ServletException: java.lang.NullPointerException
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1666)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
... 17 more
... 17 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:606)
at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:156)
at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
How about we avoid call `IOUtils.closeQuietly(ozoneOutputStream);` to throw out the `NO_SUCH_MULTIPART_UPLOAD_ERROR` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r458478856
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Do we know why omMultipartCommitUploadPartInfo is null in this case?
Because for any error, we throw exception, and that should catch the error, and we should return an proper s3 error code from there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504746102
##########
File path: tencent/python-client/OzoneS3.py
##########
@@ -0,0 +1,89 @@
+import boto3
+from botocore.config import Config
+from botocore.exceptions import ClientError
+import xml.etree.ElementTree as ET
+from itertools import izip_longest
+from random import randrange
+
+# Ozone doesn't care access key and secret key in non-SafeMode
+AWS_ACCESS_KEY_ID = '12345678'
+AWS_SECRET_ACCESS_KEY = '12345678'
+
+# Ozone S3 gateway endpoints
+root = ET.parse('ozone_idex_config.xml').getroot()
+endpoints = root.findall('property/entry')
+SERVER_LIST = []
+for endpoint in endpoints:
+ SERVER_LIST.append(endpoint.get('ozone.s3g.endpoint'))
+#endpoint = 'http://localhost:9878'
+
+# Ozone S3 bucket name
+bucket_name = 'idex_bucket'
+
+def round_robin(cur = [0]):
+ length = len(SERVER_LIST)
+ ret = SERVER_LIST[cur[0] % length]
+ cur[0] = (cur[0] + 1) % length
+ return ret
+
+def s3_resource():
+ session = boto3.session.Session(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ endpoint = round_robin([randrange(len(SERVER_LIST))])
+ s3 = session.resource(service_name='s3', endpoint_url=endpoint)
+ print("current endpoint: " + endpoint)
+ return s3
+
+class OzoneS3:
+ @staticmethod
+ def upload_file(file_path):
+ try:
+ s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+ except Exception as e:
+ print("retry one time. exception: " + str(e))
+ s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+
+
+ @staticmethod
+ def upload_stream(key, byte_stream):
+ try:
+ s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+ except Exception as e:
+ s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+
+ @staticmethod
+ def download_file(key, output_file_path):
+ try:
+ s3_resource().Bucket(bucket_name).download_file(key, output_file_path)
+ except Exception as e:
+ s3_resource().Bucket(bucket_name).download_file(key, output_file_path)
+
+ @staticmethod
+ def download_stream(key):
+ try:
+ obj = s3_resource().Object(bucket_name, key)
+ return obj.get()['Body'].read()
+ except Exception as e:
+ obj = s3_resource().Object(bucket_name, key)
+ return obj.get()['Body'].read()
+
+ @staticmethod
+ def iter_n(iterable, n, fillvalue=None):
+ args = [iter(iterable)] * n
Review comment:
Are these changes related to this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504378327
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
@GlenGeng @bharatviswa504 Thanks for your suggestion, I push a new commit to fix this.
PTAL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504746102
##########
File path: tencent/python-client/OzoneS3.py
##########
@@ -0,0 +1,89 @@
+import boto3
+from botocore.config import Config
+from botocore.exceptions import ClientError
+import xml.etree.ElementTree as ET
+from itertools import izip_longest
+from random import randrange
+
+# Ozone doesn't care access key and secret key in non-SafeMode
+AWS_ACCESS_KEY_ID = '12345678'
+AWS_SECRET_ACCESS_KEY = '12345678'
+
+# Ozone S3 gateway endpoints
+root = ET.parse('ozone_idex_config.xml').getroot()
+endpoints = root.findall('property/entry')
+SERVER_LIST = []
+for endpoint in endpoints:
+ SERVER_LIST.append(endpoint.get('ozone.s3g.endpoint'))
+#endpoint = 'http://localhost:9878'
+
+# Ozone S3 bucket name
+bucket_name = 'idex_bucket'
+
+def round_robin(cur = [0]):
+ length = len(SERVER_LIST)
+ ret = SERVER_LIST[cur[0] % length]
+ cur[0] = (cur[0] + 1) % length
+ return ret
+
+def s3_resource():
+ session = boto3.session.Session(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
+ endpoint = round_robin([randrange(len(SERVER_LIST))])
+ s3 = session.resource(service_name='s3', endpoint_url=endpoint)
+ print("current endpoint: " + endpoint)
+ return s3
+
+class OzoneS3:
+ @staticmethod
+ def upload_file(file_path):
+ try:
+ s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+ except Exception as e:
+ print("retry one time. exception: " + str(e))
+ s3_resource().Bucket(bucket_name).upload_file(file_path, file_path)
+
+
+ @staticmethod
+ def upload_stream(key, byte_stream):
+ try:
+ s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+ except Exception as e:
+ s3_resource().Object(bucket_name, key).put(Body=byte_stream)
+
+ @staticmethod
+ def download_file(key, output_file_path):
+ try:
+ s3_resource().Bucket(bucket_name).download_file(key, output_file_path)
+ except Exception as e:
+ s3_resource().Bucket(bucket_name).download_file(key, output_file_path)
+
+ @staticmethod
+ def download_stream(key):
+ try:
+ obj = s3_resource().Object(bucket_name, key)
+ return obj.get()['Body'].read()
+ except Exception as e:
+ obj = s3_resource().Object(bucket_name, key)
+ return obj.get()['Body'].read()
+
+ @staticmethod
+ def iter_n(iterable, n, fillvalue=None):
+ args = [iter(iterable)] * n
Review comment:
Are these changes related to this PR? It has files named under directory Tencent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504192790
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Sorry @maobaolong for the analysis, I missed it.
Yes, it makes sense. Thank You @GlenGeng and @maobaolong for the detailed analysis.
>How about we avoid call IOUtils.closeQuietly(ozoneOutputStream); to throw out the >NO_SUCH_MULTIPART_UPLOAD_ERROR
I am +1 for this. In this way, we can return the proper error code when during close if it happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r458479671
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Because when No_SUCH_UPLOAD case is already handled.
```
else if (ex.getResult() == ResultCodes.NO_SUCH_MULTIPART_UPLOAD_ERROR) {
throw S3ErrorTable.newError(NO_SUCH_UPLOAD, uploadID);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#issuecomment-661814899
@adoroszlai The HDDS-3994 is triggered by HDDS-3350, about this one, maybe no relation with HDDS-3350.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#issuecomment-708672417
Closing this, it is committed thru #1499
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 closed pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 closed pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r459087101
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Thanks @maobaolong for reporting.
I will look into this, and get back to you.
From my initial analysis HDDS-3999 is caused by HDDS-3685
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#discussion_r504374249
##########
File path: hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/ObjectEndpoint.java
##########
@@ -562,13 +562,18 @@ private Response createMultipartKey(String bucket, String key, long length,
OmMultipartCommitUploadPartInfo omMultipartCommitUploadPartInfo =
ozoneOutputStream.getCommitUploadPartInfo();
- String eTag = omMultipartCommitUploadPartInfo.getPartName();
+ if (omMultipartCommitUploadPartInfo != null) {
Review comment:
Hi @maobaolong, could you please add a new patch, just revert the change and replace
```
} finally {
IOUtils.closeQuietly(ozoneOutputStream);
}
```
with
```
} finally {
if (ozoneOutputStream != null) {
ozoneOutputStream.close();
}
}
```
?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on pull request #1228: HDDS-3995. Fix s3g met NPE exception while write file by multiPartUpload
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on pull request #1228:
URL: https://github.com/apache/hadoop-ozone/pull/1228#issuecomment-708134810
+1, LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org