You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Enrico Minack (Jira)" <ji...@apache.org> on 2025/07/07 20:54:00 UTC

[jira] [Commented] (SPARK-52508) Read from fallback storage should consider replication delay

    [ https://issues.apache.org/jira/browse/SPARK-52508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18003620#comment-18003620 ] 

Enrico Minack commented on SPARK-52508:
---------------------------------------

This is fixed in https://github.com/apache/spark/pull/51200.

> Read from fallback storage should consider replication delay
> ------------------------------------------------------------
>
>                 Key: SPARK-52508
>                 URL: https://issues.apache.org/jira/browse/SPARK-52508
>             Project: Spark
>          Issue Type: Sub-task
>          Components: k8s, Kubernetes
>    Affects Versions: 4.1.0
>            Reporter: Enrico Minack
>            Priority: Major
>
> Using the storage decommissioning feature on Kubernetes with a distributed filesystem as the fallback storage might run into the situation where an executor cannot see the shuffle data on the distributed filesystem that has just been written by the decommissioned executor. This is caused by some replication delay. Given the dependent executor knows the location of the shuffle data is the fallback storage, it can defer reading on a {{FileNotFoundException}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org