You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2026/08/26 04:09:00 UTC
[jira] [Resolved] (SPARK-58977) TransformWithState in PySpark state server reports spurious daemon crash during early shutdown
[ https://issues.apache.org/jira/browse/SPARK-58977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jungtaek Lim resolved SPARK-58977.
----------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 58217
[https://github.com/apache/spark/pull/58217]
> TransformWithState in PySpark state server reports spurious daemon crash during early shutdown
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-58977
> URL: https://issues.apache.org/jira/browse/SPARK-58977
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Jonathan Gao
> Assignee: Jonathan Gao
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> This is a follow-up to SPARK-58751 and https://github.com/apache/spark/pull/57941.
> TransformWithState in PySpark starts a JVM state-server listener that blocks in ServerSocketChannel.accept() while waiting for the Python worker to connect.
> If the runner is stopped before that connection is established, the shutdown path interrupts and closes the channel. Java NIO wakes accept() with ClosedByInterruptException or AsynchronousCloseException. These are expected shutdown signals, but the pre-connect path does not handle them, so the listener exception is wrapped as:
> TransformWithStateInPySpark state server daemon thread exited unexpectedly (crashed)
> Expected behavior:
> The listener should terminate cleanly, preserve interrupt status for interrupt-driven shutdown, and transition the processor handle to CLOSED.
> Proposed fix:
> - Interrupt the listener before closing the channel.
> - Handle the expected NIO interruption and closed-channel exceptions around accept().
> - Preserve interrupt status only for the interrupt-driven path.
> - Add mocked exception-hierarchy tests and real loopback ServerSocketChannel race tests.
> Pull request: https://github.com/apache/spark/pull/58217
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org