[jira] [Commented] (HUDI-1497) Timeout Exception during getFileStatus()

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259759#comment-17259759 ] 

Steve Loughran commented on HUDI-1497:
--------------------------------------

Will happen if you have too few connections in the http pool. You could set a larger one from the relevant fs.s3a property (go for it!) but this may be a sign of something holding on to connections, especially open input streams, for too long.

if you want to keep an S3A Input stream around for a while, use unbuffer() to close the connection. next read will reopen the file without needing to do any HEAD requests to probe existence/length

> Timeout Exception during getFileStatus() 
> -----------------------------------------
>
>                 Key: HUDI-1497
>                 URL: https://issues.apache.org/jira/browse/HUDI-1497
>             Project: Apache Hudi
>          Issue Type: Sub-task
>          Components: Writer Core
>            Reporter: Balaji Varadarajan
>            Priority: Major
>
> Seeing this happening when running RFC-15 branch in long running mode. There could be a resource leak as I am seeing this consistently after every 1 or 2 hour period runs.  The below log shows it is during accessing bootstrap index but I am seeing it in getFileStatus() for other files too.
>  
>  
> Caused by: java.io.InterruptedIOException: getFileStatus on s3://robinhood-encrypted-hudi-data-cove/dummy/balaji/sickle/public/client_ledger_clientledgerbalance/test_v4/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from poolCaused by: java.io.InterruptedIOException: getFileStatus on s3://robinhood-encrypted-hudi-data-cove/dummy/balaji/sickle/public/client_ledger_clientledgerbalance/test_v4/.hoodie/.aux/.bootstrap/.partitions/00000000-0000-0000-0000-000000000000-0_1-0-1_00000000000001.hfile: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:141) at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:117) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1859) at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1823) at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1763) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1627) at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:2500) at org.apache.hudi.common.fs.HoodieWrapperFileSystem.exists(HoodieWrapperFileSystem.java:549) at org.apache.hudi.common.bootstrap.index.HFileBootstrapIndex.<init>(HFileBootstrapIndex.java:102) ... 33 moreCaused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1113) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1063) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176) at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:1053) at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1841) ... 39 more



--
This message was sent by Atlassian Jira
(v8.3.4#803005)