You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gengliang Wang (Jira)" <ji...@apache.org> on 2026/10/14 22:03:00 UTC
[jira] [Resolved] (SPARK-60246) Failed spark:// fetch blocks all later file downloads on the executor
[ https://issues.apache.org/jira/browse/SPARK-60246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gengliang Wang resolved SPARK-60246.
------------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59472
[https://github.com/apache/spark/pull/59472]
> Failed spark:// fetch blocks all later file downloads on the executor
> ---------------------------------------------------------------------
>
> Key: SPARK-60246
> URL: https://issues.apache.org/jira/browse/SPARK-60246
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.3.0, 4.1.3
> Reporter: Chiran Ravani
> Assignee: Chiran Ravani
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> When `Utils.fetchFile` downloads a `spark://` URL, `doFetchFile` opens the RPC download channel and then calls `downloadFile`. `downloadFile` creates its temp file before its try block. If that fails, for example because the target directory was deleted, the method throws and the input stream is never closed.
> The driver keeps sending the file. On the receiving side, `FileDownloadCallback.onData` writes the bytes into a pipe from the file-download Netty thread. `spark.files.io.threads` defaults to 1, so there is one such thread. Nobody reads the pipe, so once it is full the write blocks forever.
> Every later download through that `RpcEnv` waits behind the blocked thread: added jars and files, and REPL classes loaded by `ExecutorClassLoader`. Tasks that need any of them hang. There is no timeout on this path.
> The stuck thread:
> {code}
> "files-client-..." RUNNABLE
> sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:273)
> org.apache.spark.rpc.netty.NettyRpcEnv$FileDownloadCallback.onData(NettyRpcEnv.scala)
> org.apache.spark.network.client.StreamInterceptor.handle(StreamInterceptor.java)
> org.apache.spark.network.util.TransportFrameDecoder.feedInterceptor(TransportFrameDecoder.java)
> {code}
> Closing the stream on failure is not enough by itself. The Netty thread's next write then fails with "IOException: Broken pipe". That closes the connection and fails every other download in flight on it.
> To reproduce: fetch a large file (16 MiB) from a `spark://` URL into a directory that does not exist, then fetch a small file through the same `RpcEnv`. The second fetch never completes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org