You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2024/10/04 17:31:00 UTC

[jira] [Updated] (SPARK-47702) Shuffle service endpoint is not removed from the locations list when RDD block is removed form a node.

     [ https://issues.apache.org/jira/browse/SPARK-47702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-47702:
----------------------------------
    Fix Version/s:     (was: 3.5.4)

> Shuffle service endpoint is not removed from the locations list when RDD block is removed form a node.
> ------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-47702
>                 URL: https://issues.apache.org/jira/browse/SPARK-47702
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 3.5.1
>            Reporter: mahesh kumar behera
>            Assignee: Attila Zsolt Piros
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>
> If SHUFFLE_SERVICE_FETCH_RDD_ENABLED is set to true, driver stores both executor end point and the external shuffle end points for a RDD block. When the RDD is migrated, the location info is updated to add the end point corresponds to new location and the old end point is removed. But currently, only the executor end point is removed. The shuffle service end point is not removed. This cause failure during RDD read if the shuffle service end point is chosen due to task locality.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org