You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/08/19 18:14:12 UTC

[GitHub] [hadoop] steveloughran commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections

steveloughran commented on PR #4766:
URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1220968467

   
   
   The async stream draining only executes (in the other thread) if the original invoker waits for the result.
   
   this is not caused by some synchronized conflict -I made the method being invoked static to ensure this.
   
   And the logging of the start/finish of the call are present. just not the bit in the middle.
   
   ## test run with the {{join()}} to make it blocking.
   
   the `[JUnit-testUnbufferDraining]` thread is the one doing the unbuffer()/read() calls; the `[s3a-transfer-stevel-london-unbounded-pool]` threads are the unbounded pool into which work
   is queued. it is the unbounded pool; so that is not the limit.
   
   That would appear to leave some aspect of CompletableFuture, possibly related to how things are being wrapped in duration tracking, audit spans etc. But I can't see this, and they all seem to work everywhere else.
   
   
   
   ```
   
   {code}
   2022-08-19 18:52:50,102 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInputStream (S3AInputStream.java:closeStream(663)) - Closing stream unbuffer(): soft
   2022-08-19 18:52:50,102 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInputStream (S3AInputStream.java:closeStream(676)) - initiating asynchronous drain of 998 bytes
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:lambda$null$0(1609)) - Starting submitted operation in ec003fc0-e885-4793-b9b9-34296cc34f3a-00000010
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AInputStream (S3AInputStream.java:drainOrAbortHttpStream(761)) - drain or abort reason unbuffer() remaining=998 abort=false
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AInputStream (S3AInputStream.java:drainOrAbortHttpStream(771)) - draining 998 bytes
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AInputStream (S3AInputStream.java:drainOrAbortHttpStream(789)) - Drained stream of 998 bytes
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AInputStream (S3AInputStream.java:drainOrAbortHttpStream(801)) - Closing stream
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AInputStream (S3AInputStream.java:drainOrAbortHttpStream(826)) - Stream s3a://stevel-london/test/testUnbufferDraining closed: unbuffer(); remaining=0
   2022-08-19 18:52:50,102 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:lambda$null$0(1613)) - Completed submitted operation in ec003fc0-e885-4793-b9b9-34296cc34f3a-00000010
   2022-08-19 18:52:50,104 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInstrumentation (S3AInstrumentation.java:merge(1172)) - Merging statistics into FS statistics in unbuffer(): ...
   
   2022-08-19 18:52:50,104 [JUnit-testUnbufferDraining] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(219)) -
   
   testUnbufferDraining: Starting read/unbuffer #2
   
   {code}
   ```
   
   ## no waiting
   But without the join, no joy. even though the log above shows the drain is being executed in the new thread.
   
   ```
   2022-08-19 18:36:56,109 [JUnit-testUnbufferDraining] DEBUG s3a.Invoker (DurationInfo.java:close(101)) - read: duration 0:00.000s
   2022-08-19 18:36:56,109 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInputStream (S3AInputStream.java:closeStream(663)) - Closing stream unbuffer(): soft
   2022-08-19 18:36:56,109 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInputStream (S3AInputStream.java:closeStream(676)) - initiating asynchronous drain of 998 bytes
   2022-08-19 18:36:56,109 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:lambda$null$0(1609)) - Starting submitted operation in 54f63eab-e0d8-48b5-a0c1-9a877576a450-00000010
   2022-08-19 18:36:56,109 [s3a-transfer-stevel-london-unbounded-pool5-t3] DEBUG s3a.S3AFileSystem (S3AFileSystem.java:lambda$null$0(1613)) - Completed submitted operation in 54f63eab-e0d8-48b5-a0c1-9a877576a450-00000010
   
   2022-08-19 18:36:56,110 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInstrumentation (S3AInstrumentation.java:merge(1172)) - Merging statistics into FS statistics in unbuffer():  ...
   2022-08-19 18:36:56,111 [JUnit-testUnbufferDraining] INFO s3a.AbstractS3ATestBase (AbstractS3ATestBase.java:describe(219)) -
   
   testUnbufferDraining: Starting read/unbuffer #2
   
   2022-08-19 18:36:56,111 [JUnit-testUnbufferDraining] DEBUG s3a.Invoker (DurationInfo.java:<init>(80)) - Starting: lazySeek
   2022-08-19 18:36:56,111 [JUnit-testUnbufferDraining] DEBUG s3a.S3AInputStream (S3AInputStream.java:reopen(263)) - reopen(s3a://stevel-london/test/testUnbufferDraining) for read from new offset range[49001-50000], length=1, streamPosition=49002, nextReadPosition=49001, policy=random
   2022-08-19 18:36:56,111 [JUnit-testUnbufferDraining] DEBUG impl.ChangeDetectionPolicy (ChangeDetectionPolicy.java:applyRevisionConstraint(430)) - Restricting get request to version RFW1GsJZAc_mL5eKKWRnB7tX2K9B3I98
   2022-08-19 18:36:56,112 [JUnit-testUnbufferDraining] DEBUG impl.LoggingAuditor (LoggingAuditor.java:beforeExecution(327)) - [11] 54f63eab-e0d8-48b5-a0c1-9a877576a450-00000010 Executing op_open with {action_http_get_request 'test/testUnbufferDraining' size=998, mutating=false}; https://audit.example.org/hadoop/1/op_open/54f63eab-e0d8-48b5-a0c1-9a877576a450-00000010/?op=op_open&p1=test/testUnbufferDraining&pr=stevel&ps=6cd73b0d-cfa1-41ec-ba5a-305b3abff0b0&id=54f63eab-e0d8-48b5-a0c1-9a877576a450-00000010&t0=30&fs=54f63eab-e0d8-48b5-a0c1-9a877576a450&t1=11&ts=1660930615787
   2022-08-19 18:36:58,223 [JUnit-testUnbufferDraining] DEBUG s3a.Invoker (DurationInfo.java:close(101)) - lazySeek: duration 0:02.112s
   
   ```
   
   then the timeout surfaces on the next read()
   ```
   
   org.apache.hadoop.net.ConnectTimeoutException: re-open s3a://stevel-london/test/testUnbufferDraining at 49001 on s3a://stevel-london/test/testUnbufferDraining: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
   
   	at org.apache.hadoop.fs.s3a.S3AUtils.translateInterruptedException(S3AUtils.java:385)
   	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:198)
   	at org.apache.hadoop.fs.s3a.Invoker.onceTrackingDuration(Invoker.java:149)
   	at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:276)
   	at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$lazySeek$1(S3AInputStream.java:429)
   	at org.apache.hadoop.fs.s3a.Invoker.lambda$maybeRetry$3(Invoker.java:284)
   	at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:122)
   	at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:410)
   	at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:282)
   	at org.apache.hadoop.fs.s3a.Invoker.maybeRetry(Invoker.java:326)
   	at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:421)
   	at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:455)
   	at java.io.FilterInputStream.read(FilterInputStream.java:83)
   	at org.apache.hadoop.fs.s3a.performance.ITestUnbufferDraining.testUnbufferDraining(ITestUnbufferDraining.java:141)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
   	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
   	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   	at java.lang.Thread.run(Thread.java:750)
   
   Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
   at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
   at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
   at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
   at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
   at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1524)
   at org.apache.hadoop.fs.s3a.S3AFileSystem$InputStreamCallbacksImpl.getObject(S3AFileSystem.java:1600)
   at org.apache.hadoop.fs.s3a.S3AInputStream.lambda$reopen$0(S3AInputStream.java:278)
   at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
   at org.apache.hadoop.fs.s3a.Invoker.onceTrackingDuration(Invoker.java:147)
   ... 26 more
   Caused by: com.amazonaws.thirdparty.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
   at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)
   at com.amazonaws.thirdparty.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:282)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:498)
   at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
   at com.amazonaws.http.conn.$Proxy22.get(Unknown Source)
   at com.amazonaws.thirdparty.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
   at com.amazonaws.thirdparty.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
   at com.amazonaws.thirdparty.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
   at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
   at com.amazonaws.thirdparty.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1346)
   at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
   ... 40 more
   
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org