[jira] [Commented] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

["Attila Zsolt Piros (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17181138#comment-17181138 ] 

Attila Zsolt Piros commented on SPARK-32663:
--------------------------------------------

You are right. The clients will be closed via [TransportClientFactory#close()|https://github.com/attilapiros/spark/blob/e6795cd3416bbe32505efd4c1fa3202f451bf74d/common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java#L324].



> TransportClient getting closed when there are outstanding requests to the server
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-32663
>                 URL: https://issues.apache.org/jira/browse/SPARK-32663
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 3.0.0
>            Reporter: Chandni Singh
>            Priority: Major
>
> The implementation of {{removeBlocks}} and {{getHostLocalDirs}} in {{ExternalBlockStoreClient}} closes the client after processing a response in the callback. 
> This is a cached client which will be re-used for other responses. There could be other outstanding request to the shuffle service, so it should not be closed after processing a response. 
> Seems like this is a bug introduced with SPARK-27651 and SPARK-27677. 
> The older methods  {{registerWithShuffleServer}} and {{fetchBlocks}} didn't close the client.
> cc [~attilapiros] [~vanzin] [~mridulm80]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org