You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Herman van Hövell (Jira)" <ji...@apache.org> on 2026/03/25 19:44:00 UTC
[jira] [Resolved] (SPARK-55448) Spark Connect query execution events dropped when session closes while query running
[ https://issues.apache.org/jira/browse/SPARK-55448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Herman van Hövell resolved SPARK-55448.
---------------------------------------
Fix Version/s: 4.2.0
Resolution: Fixed
Issue resolved by pull request 54225
[https://github.com/apache/spark/pull/54225]
> Spark Connect query execution events dropped when session closes while query running
> --------------------------------------------------------------------------------------
>
> Key: SPARK-55448
> URL: https://issues.apache.org/jira/browse/SPARK-55448
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 3.4.4, 4.0.2, 3.5.8, 4.1.1
> Reporter: Biruk Tesfaye
> Assignee: Biruk Tesfaye
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> There is a race condition in Spark Connect's execution lifecycle where SparkListenerBus query events (Cancelled/Closed) aren't sent if a session closes while a query is still running.
> The root cause is ExecuteEventsManager.assertStatus(), which validates that the session status is Started before allowing any event to be sent. When trying to close a session, the thread interrupts all execution threads for the session and proceeds to close the session, marking the session as closed. Each of the interrupted threads' cleanup path tries to send its terminal event which would fail if the main thread marked the session as closed before, causing an IllegalStateException that prevents the events from ever reaching the listener bus.
> The proposed fix allows ExecuteEventsManager.assertStatus() to accept SessionStatus.Closed alongside Started so terminal events can still be sent during execution cleanup. This is safe because the underlying SparkContext and its listener bus outlive the Spark Connect session, and all downstream operations after the assert are independent of session state. The assert continues to block SessionStatus.Pending as a guard against events on uninitialized sessions.
>
> A more robust solution might be to introduce a SessionStatus.Closing state that exists until all execution threads are cleaned up then transitions to SessionStatus.Closed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org