You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Gengliang Wang (Jira)" <ji...@apache.org> on 2026/10/14 22:03:00 UTC

[jira] [Resolved] (SPARK-60246) Failed spark:// fetch blocks all later file downloads on the executor

     [ https://issues.apache.org/jira/browse/SPARK-60246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gengliang Wang resolved SPARK-60246.
------------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 59472
[https://github.com/apache/spark/pull/59472]

> Failed spark:// fetch blocks all later file downloads on the executor
> ---------------------------------------------------------------------
>
>                 Key: SPARK-60246
>                 URL: https://issues.apache.org/jira/browse/SPARK-60246
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 4.3.0, 4.1.3
>            Reporter: Chiran Ravani
>            Assignee: Chiran Ravani
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> When `Utils.fetchFile` downloads a `spark://` URL, `doFetchFile` opens the RPC download channel and then calls `downloadFile`. `downloadFile` creates its temp file before its try block. If that fails, for example because the target directory was deleted, the method throws and the input stream is never closed.
> The driver keeps sending the file. On the receiving side, `FileDownloadCallback.onData` writes the bytes into a pipe from the file-download Netty thread. `spark.files.io.threads` defaults to 1, so there is one such thread. Nobody reads the pipe, so once it is full the write blocks forever.
> Every later download through that `RpcEnv` waits behind the blocked thread: added jars and files, and REPL classes loaded by `ExecutorClassLoader`. Tasks that need any of them hang. There is no timeout on this path.
> The stuck thread:
> {code}
> "files-client-..." RUNNABLE
>   sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>   sun.nio.ch.SinkChannelImpl.write(SinkChannelImpl.java:273)
>   org.apache.spark.rpc.netty.NettyRpcEnv$FileDownloadCallback.onData(NettyRpcEnv.scala)
>   org.apache.spark.network.client.StreamInterceptor.handle(StreamInterceptor.java)
>   org.apache.spark.network.util.TransportFrameDecoder.feedInterceptor(TransportFrameDecoder.java)
> {code}
> Closing the stream on failure is not enough by itself. The Netty thread's next write then fails with "IOException: Broken pipe". That closes the connection and fails every other download in flight on it.
> To reproduce: fetch a large file (16 MiB) from a `spark://` URL into a directory that does not exist, then fetch a small file through the same `RpcEnv`. The second fetch never completes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org