You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xccui (via GitHub)" <gi...@apache.org> on 2023/04/03 19:29:19 UTC
[GitHub] [hudi] xccui commented on issue #8305: [SUPPORT] Potential FileSystem http connection leaking
xccui commented on issue #8305:
URL: https://github.com/apache/hudi/issues/8305#issuecomment-1494861495
Hi @danny0405, I looked into this again. You are right, `returnContent()` will release the connection. Actually, I was misled by the code. There will be two `PoolingHttpClientConnectionManager`s at runtime.
```
leaseConnection:306, PoolingHttpClientConnectionManager (com.amazonaws.thirdparty.apache.http.impl.conn)
get:282, PoolingHttpClientConnectionManager$1 (com.amazonaws.thirdparty.apache.http.impl.conn)
invoke:-1, GeneratedMethodAccessor24 (jdk.internal.reflect)
invoke:43, DelegatingMethodAccessorImpl (jdk.internal.reflect)
invoke:566, Method (java.lang.reflect)
invoke:70, ClientConnectionRequestFactory$Handler (com.amazonaws.http.conn)
get:-1, $Proxy51 (com.amazonaws.http.conn)
execute:190, MainClientExec (com.amazonaws.thirdparty.apache.http.impl.execchain)
execute:186, ProtocolExec (com.amazonaws.thirdparty.apache.http.impl.execchain)
doExecute:185, InternalHttpClient (com.amazonaws.thirdparty.apache.http.impl.client)
execute:83, CloseableHttpClient (com.amazonaws.thirdparty.apache.http.impl.client)
execute:56, CloseableHttpClient (com.amazonaws.thirdparty.apache.http.impl.client)
execute:72, SdkHttpClient (com.amazonaws.http.apache.client.impl)
executeOneRequest:1346, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
executeHelper:1157, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
doExecute:814, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
executeWithTimer:781, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
execute:755, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
access$500:715, AmazonHttpClient$RequestExecutor (com.amazonaws.http)
execute:697, AmazonHttpClient$RequestExecutionBuilderImpl (com.amazonaws.http)
execute:561, AmazonHttpClient (com.amazonaws.http)
execute:541, AmazonHttpClient (com.amazonaws.http)
invoke:5456, AmazonS3Client (com.amazonaws.services.s3)
invoke:5403, AmazonS3Client (com.amazonaws.services.s3)
getObjectMetadata:1372, AmazonS3Client (com.amazonaws.services.s3)
lambda$getObjectMetadata$10:2545, S3AFileSystem (org.apache.hadoop.fs.s3a)
apply:-1, 497983073 (org.apache.hadoop.fs.s3a.S3AFileSystem$$Lambda$1189)
retryUntranslated:414, Invoker (org.apache.hadoop.fs.s3a)
retryUntranslated:377, Invoker (org.apache.hadoop.fs.s3a)
getObjectMetadata:2533, S3AFileSystem (org.apache.hadoop.fs.s3a)
getObjectMetadata:2513, S3AFileSystem (org.apache.hadoop.fs.s3a)
s3GetFileStatus:3776, S3AFileSystem (org.apache.hadoop.fs.s3a)
innerGetFileStatus:3688, S3AFileSystem (org.apache.hadoop.fs.s3a)
lambda$getFileStatus$24:3556, S3AFileSystem (org.apache.hadoop.fs.s3a)
apply:-1, 718057245 (org.apache.hadoop.fs.s3a.S3AFileSystem$$Lambda$2610)
lambda$trackDurationOfOperation$5:499, IOStatisticsBinding (org.apache.hadoop.fs.statistics.impl)
apply:-1, 2039613101 (org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding$$Lambda$1168)
trackDuration:444, IOStatisticsBinding (org.apache.hadoop.fs.statistics.impl)
trackDurationAndSpan:2337, S3AFileSystem (org.apache.hadoop.fs.s3a)
trackDurationAndSpan:2356, S3AFileSystem (org.apache.hadoop.fs.s3a)
getFileStatus:3554, S3AFileSystem (org.apache.hadoop.fs.s3a)
lambda$getFileStatus$17:410, HoodieWrapperFileSystem (org.apache.hudi.common.fs)
get:-1, 589863653 (org.apache.hudi.common.fs.HoodieWrapperFileSystem$$Lambda$2609)
executeFuncWithTimeMetrics:114, HoodieWrapperFileSystem (org.apache.hudi.common.fs)
getFileStatus:404, HoodieWrapperFileSystem (org.apache.hudi.common.fs)
checkTableValidity:51, TableNotFoundException (org.apache.hudi.exception)
<init>:137, HoodieTableMetaClient (org.apache.hudi.common.table)
newMetaClient:689, HoodieTableMetaClient (org.apache.hudi.common.table)
access$000:81, HoodieTableMetaClient (org.apache.hudi.common.table)
build:770, HoodieTableMetaClient$Builder (org.apache.hudi.common.table)
createMetaClient:277, StreamerUtil (org.apache.hudi.util)
<init>:118, WriteProfile (org.apache.hudi.sink.partitioner.profile)
<init>:44, DeltaWriteProfile (org.apache.hudi.sink.partitioner.profile)
getWriteProfile:75, WriteProfiles (org.apache.hudi.sink.partitioner.profile)
lambda$singleton$0:64, WriteProfiles (org.apache.hudi.sink.partitioner.profile)
apply:-1, 401283836 (org.apache.hudi.sink.partitioner.profile.WriteProfiles$$Lambda$3189)
computeIfAbsent:1134, HashMap (java.util)
singleton:63, WriteProfiles (org.apache.hudi.sink.partitioner.profile)
create:56, BucketAssigners (org.apache.hudi.sink.partitioner)
open:122, BucketAssignFunction (org.apache.hudi.sink.partitioner)
openFunction:34, FunctionUtils (org.apache.flink.api.common.functions.util)
open:100, AbstractUdfStreamOperator (org.apache.flink.streaming.api.operators)
open:55, KeyedProcessOperator (org.apache.flink.streaming.api.operators)
initializeStateAndOpenOperators:107, RegularOperatorChain (org.apache.flink.streaming.runtime.tasks)
restoreGates:726, StreamTask (org.apache.flink.streaming.runtime.tasks)
call:-1, 704715276 (org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$3036)
call:55, StreamTaskActionExecutor$1 (org.apache.flink.streaming.runtime.tasks)
restoreInternal:702, StreamTask (org.apache.flink.streaming.runtime.tasks)
restore:669, StreamTask (org.apache.flink.streaming.runtime.tasks)
run:-1, 619798876 (org.apache.flink.runtime.taskmanager.Task$$Lambda$2938)
runWithSystemExitMonitoring:935, Task (org.apache.flink.runtime.taskmanager)
restoreAndInvoke:904, Task (org.apache.flink.runtime.taskmanager)
doRun:728, Task (org.apache.flink.runtime.taskmanager)
run:550, Task (org.apache.flink.runtime.taskmanager)
run:829, Thread (java.lang)
```
```
leaseConnection:276, PoolingHttpClientConnectionManager (org.apache.hudi.org.apache.http.impl.conn)
get:263, PoolingHttpClientConnectionManager$1 (org.apache.hudi.org.apache.http.impl.conn)
execute:190, MainClientExec (org.apache.hudi.org.apache.http.impl.execchain)
execute:184, ProtocolExec (org.apache.hudi.org.apache.http.impl.execchain)
execute:88, RetryExec (org.apache.hudi.org.apache.http.impl.execchain)
execute:110, RedirectExec (org.apache.hudi.org.apache.http.impl.execchain)
doExecute:184, InternalHttpClient (org.apache.hudi.org.apache.http.impl.client)
execute:82, CloseableHttpClient (org.apache.hudi.org.apache.http.impl.client)
execute:107, CloseableHttpClient (org.apache.hudi.org.apache.http.impl.client)
execute:55, CloseableHttpClient (org.apache.hudi.org.apache.http.impl.client)
execute:151, Request (org.apache.hudi.org.apache.http.client.fluent)
get:545, RemoteHoodieTableFileSystemView (org.apache.hudi.common.table.view)
executeRequest:185, RemoteHoodieTableFileSystemView (org.apache.hudi.common.table.view)
getPendingCompactionOperations:458, RemoteHoodieTableFileSystemView (org.apache.hudi.common.table.view)
apply:-1, 2001188586 (org.apache.hudi.common.table.view.PriorityBasedFileSystemView$$Lambda$3253)
execute:69, PriorityBasedFileSystemView (org.apache.hudi.common.table.view)
getPendingCompactionOperations:231, PriorityBasedFileSystemView (org.apache.hudi.common.table.view)
<init>:96, CleanPlanner (org.apache.hudi.table.action.clean)
requestClean:105, CleanPlanActionExecutor (org.apache.hudi.table.action.clean)
requestClean:150, CleanPlanActionExecutor (org.apache.hudi.table.action.clean)
execute:176, CleanPlanActionExecutor (org.apache.hudi.table.action.clean)
scheduleCleaning:310, HoodieFlinkCopyOnWriteTable (org.apache.hudi.table)
scheduleTableServiceInternal:430, BaseHoodieTableServiceClient (org.apache.hudi.client)
clean:543, BaseHoodieTableServiceClient (org.apache.hudi.client)
clean:758, BaseHoodieWriteClient (org.apache.hudi.client)
clean:730, BaseHoodieWriteClient (org.apache.hudi.client)
lambda$open$0:67, CleanFunction (org.apache.hudi.sink)
run:-1, 1349800153 (org.apache.hudi.sink.CleanFunction$$Lambda$3117)
lambda$wrapAction$0:130, NonThrownExecutor (org.apache.hudi.sink.utils)
run:-1, 329014366 (org.apache.hudi.sink.utils.NonThrownExecutor$$Lambda$3118)
runWorker:1128, ThreadPoolExecutor (java.util.concurrent)
run:628, ThreadPoolExecutor$Worker (java.util.concurrent)
run:829, Thread (java.lang)
```
The one that caused our exception is the AWS shaded one. It's irrelevant to the `EXECUTOR` used by Hudi. I'll close this for now and get deeper into the connection pool's behaviour by collecting some debug logs. Really appreciate your insights!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org