You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2026/07/09 01:57:00 UTC
[jira] [Resolved] (SPARK-47819) Use asynchronous callback for execution cleanup
[ https://issues.apache.org/jira/browse/SPARK-47819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-47819.
----------------------------------
Resolution: Fixed
Resolving as Fixed: this change was merged to master. Implementing commit: 2bc9573e94f2 [SPARK-47819][CONNECT] Use asynchronous callback for execution cleanup. (Triaged as a stranded ticket — the fix landed but the JIRA was left unresolved. Verified present on current master and not reverted.)
> Use asynchronous callback for execution cleanup
> -----------------------------------------------
>
> Key: SPARK-47819
> URL: https://issues.apache.org/jira/browse/SPARK-47819
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 3.5.0, 3.5.1, 4.0.0
> Reporter: Xi Lyu
> Priority: Major
> Labels: pull-request-available
>
> Expired sessions are regularly checked and cleaned up by a maintenance thread. However, currently, this process is synchronous. Therefore, in rare cases, interrupting the execution thread of a query in a session can take hours, causing the entire maintenance process to stall, resulting in a large amount of memory not being cleared.
> We address this by introducing asynchronous callbacks for execution cleanup, avoiding synchronous joins of execution threads, and preventing the maintenance thread from stalling in the above scenarios. To be more specific, instead of calling {{runner.join()}} in ExecutorHolder.close(), we set a post-cleanup function as the callback through {{{}runner.processOnCompletion{}}}, which will be called asynchronously once the execution runner is completed or interrupted. In this way, the maintenance thread won't get blocked on {{{}join{}}}ing an execution thread.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org