[jira] [Created] (SPARK-45206) Shutting down SparkConnectClient can have outstanding ReleaseExecute

["Juliusz Sompolski (Jira)" <ji...@apache.org>]

Juliusz Sompolski created SPARK-45206:
-----------------------------------------

             Summary: Shutting down SparkConnectClient can have outstanding ReleaseExecute
                 Key: SPARK-45206
                 URL: https://issues.apache.org/jira/browse/SPARK-45206
             Project: Spark
          Issue Type: Improvement
          Components: Connect
    Affects Versions: 4.0.0
            Reporter: Juliusz Sompolski


In Spark Connect scala client, there can be outstanding asynchronous ReleaseExecute calls when the client is shutdown. 

In ExecutePlanResponseReattachableIterator we (ab)use a grpc thread in createRetryingReleaseExecuteResponseObserver to run it.

 
When we do {{{}SparkConnectClient.shutdown{}}}, which does {{channel.shutdownNow()}}
it kills it before it finishes the ReleaseExecute. Maybe a more graceful shutdown would work, or we should move to a more explicit threadpool that we manage, like in python?

See discussion in https://github.com/apache/spark/pull/42929/files#r1329071826
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org