You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Bharat Viswanadham (Jira)" <ji...@apache.org> on 2020/10/14 21:33:00 UTC
[jira] [Resolved] (HDDS-3995) s3g met NPE exception while write
file by multiPartUpload
[ https://issues.apache.org/jira/browse/HDDS-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bharat Viswanadham resolved HDDS-3995.
--------------------------------------
Fix Version/s: 1.1.0
Resolution: Fixed
> s3g met NPE exception while write file by multiPartUpload
> ---------------------------------------------------------
>
> Key: HDDS-3995
> URL: https://issues.apache.org/jira/browse/HDDS-3995
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: S3
> Affects Versions: 0.5.0
> Reporter: maobaolong
> Assignee: maobaolong
> Priority: Major
> Labels: pull-request-available, s3g
> Fix For: 1.1.0
>
>
> *strong text*The following is the exception
>
> 2020-07-20 17:26:17,692 [java.util.concurrent.ThreadPoolExecutor$Worker@2b70f372[State = -1, empty queue]] ERROR org.apache.hadoop.hdds.scm.storage.BlockOutputStream: writing chunk failed 104545451305649559_chunk_1 blockID conID: 284442 locID: 104545451305649559 bcsId: 0 with exception org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-96F1E0530C66->RAFT is closed.
> 2020-07-20 17:26:17,700 [java.util.concurrent.ThreadPoolExecutor$Worker@2b70f372[State = -1, empty queue]] ERROR org.apache.hadoop.hdds.scm.storage.BlockOutputStream: writing chunk failed 104545451305649559_chunk_2 blockID conID: 284442 locID: 104545451305649559 bcsId: 0 with exception org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-96F1E0530C66->RAFT is closed.
> 2020-07-20 17:26:17,700 [java.util.concurrent.ThreadPoolExecutor$Worker@2b70f372[State = -1, empty queue]] ERROR org.apache.hadoop.hdds.scm.storage.BlockOutputStream: writing chunk failed 104545451305649559_chunk_3 blockID conID: 284442 locID: 104545451305649559 bcsId: 0 with exception org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-96F1E0530C66->RAFT is closed.
> 2020-07-20 17:26:17,726 [qtp2131952342-276] WARN org.apache.hadoop.ozone.client.io.KeyOutputStream: Encountered exception java.io.IOException: Unexpected Storage Container Exception: java.util.concurrent.CompletionException: java.util.concurrent.CompletionException: org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client:client-96F1E0530C66->RAFT is closed. on the pipeline Pipeline[ Id: 58e4eef0-d2b1-4246-a152-83a28012848f, Nodes: 38103930-07b9-4434-a657-621276c65683\{ip: 9.180.20.47, host: host-9-180-20-47, networkLocation: /rack1, certSerialId: null}207b98d9-ad64-45a8-940f-504b514feff5\{ip: 9.180.21.88, host: host-9-180-21-88, networkLocation: /rack2, certSerialId: null}d3336357-8920-4a4e-a12f-e57da1640c4d\{ip: 9.180.20.94, host: host-9-180-20-94, networkLocation: /rack1, certSerialId: null}, Type:RATIS, Factor:THREE, State:OPEN, leaderId:207b98d9-ad64-45a8-940f-504b514feff5, CreationTimestamp2020-07-07T15:11:10.887Z]. The last committed block length is 0, uncommitted data length is 10485760 retry count 0
> 2020-07-20 17:26:17,726 [qtp2131952342-276] INFO org.apache.hadoop.ozone.client.io.BlockOutputStreamEntryPool: Allocating block with ExcludeList \{datanodes = [], containerIds = [], pipelineIds = [PipelineID=58e4eef0-d2b1-4246-a152-83a28012848f]}
> 2020-07-20 17:26:18,238 [qtp2131952342-276] WARN org.eclipse.jetty.server.HttpChannel: /bucketmbl/114.dat
> javax.servlet.ServletException: javax.servlet.ServletException: java.lang.NullPointerException
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:162)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.Server.handle(Server.java:500)
> at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
> at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
> at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
> at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: javax.servlet.ServletException: java.lang.NullPointerException
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:432)
> at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:370)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:389)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:342)
> at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:229)
> at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395)
> at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
> at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
> at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1645)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
> at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.createMultipartKey(ObjectEndpoint.java:565)
> at org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.put(ObjectEndpoint.java:136)
> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:76)
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:148)
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:191)
> at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:200)
> at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:103)
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:493)
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:415)
> at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:104)
> at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:277)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:416)
> ... 45 more
> After switch the log level to TRACE, i see the following error
> 2020-07-21 15:31:03,212 [qtp1150963491-534] ERROR exception.S3ErrorTable: <?xml version="1.0" encoding="UTF-8"?>
> <Error>
> <Code>NoSuchUpload</Code>
> <Message>The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.</Message>
> <Resource>828e5250-d3e5-42dd-a273-e2c687293e2d-104550574711057985</Resource>
> <RequestId/>
> </Error>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org