You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (Jira)" <ji...@apache.org> on 2026/07/15 03:39:00 UTC
[jira] [Resolved] (SPARK-58094) Blocking RPCs hang forever when the connection dies mid-wait (no gRPC keepalive / no client-side deadline)
[ https://issues.apache.org/jira/browse/SPARK-58094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-58094.
---------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57202
[https://github.com/apache/spark/pull/57202]
> Blocking RPCs hang forever when the connection dies mid-wait (no gRPC keepalive / no client-side deadline)
> ----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58094
> URL: https://issues.apache.org/jira/browse/SPARK-58094
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 4.2.0, 4.3.0
> Reporter: Vinod KC
> Assignee: Vinod KC
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
> Attachments: spark-connect-hang-demo.zip
>
>
> Any blocking call in Spark Connect can hang indefinitely and never throw if the network path between client and server goes silently dark while the RPC is waiting on a response (e.g. a NAT gateway, load balancer, or corporate proxy evicting an idle connection without sending a TCP RST/FIN). This is a transport-layer gap, not specific to any one API: Spark Connect configures no gRPC/HTTP2 keepalive on either side, and the client's blocking read path has no deadline by default. Any RPC whose response is delayed while its connection is silently dropped is exposed — most visibly `StreamingQuery.awaitTermination()` and other long-lived/idle-prone calls (long streaming queries, session status polling, gaps between reattach calls), since they hold a connection open and idle for long enough that a middlebox is likely to evict it — but the same failure mode applies to a regular (non-streaming) query's RPC if its connection happens to be silently dropped mid-wait; only the size of the exposure window differs, not the mechanism.
> *Impact:*
> A Mission Critical case surfaced via StreamingQuery.awaitTermination(): a streaming job stalled for hours after a single non-retryable source error.
> Symptoms:
> - StreamingQueryListener.onQueryTerminated() fired correctly with the terminal exception.
> - The client's blocked query.awaitTermination() call never returned and never threw.
> - The job run stayed "RUNNING" indefinitely; the driver process was healthy and idle the whole time (no CPU/network activity, no errors logged).
>
> *Root Cause:*
> Two independent gaps combine to produce the hang, both at the transport/channel level shared by all RPCs:
> 1. No gRPC/HTTP2 keepalive is configured on either the client channel (SparkConnectClient.scala) or the server (SparkConnectService.scala's NettyServerBuilder), so a silently-dropped connection is never detected.
> 2. The client's blocking read (ExecutePlanResponseReattachableIterator → ArrayBlockingQueue.take()) has no deadline, and the server's polling loops (e.g. handleStreamingAwaitTermination) only exit via Context.current.isCancelled, a signal that itself depends on the transport noticing the dead connection — which never happens without keepalive.
> *Issue repro:*
> Reproduced with two separate JVM processes (real Connect server + separate client), using a `frozen TCP proxy` to simulate a silently-dropped connection (no root/iptables needed).
> Confirmed via thread dump that the client is parked forever in ArrayBlockingQueue.take() while the server-side query has already terminated independently. awaitTermination() was used as the concrete repro vehicle because its long idle/blocking window makes the bug easy and reliable to trigger, not because the bug is scoped to it.
>
> Issue repro setup in zip file `spark-connect-hang-demo.zip`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org