[jira] [Updated] (SPARK-47678) Got fetch failed exception when new executor reused same ip address from a previously killed executor

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-47678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-47678:
-----------------------------------
    Labels: pull-request-available  (was: )

> Got fetch failed exception when new executor reused same ip address from a previously killed executor
> -----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-47678
>                 URL: https://issues.apache.org/jira/browse/SPARK-47678
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 3.4.2, 3.4.0, 3.4.1, 3.5.0, 3.5.1
>         Environment: This only happens on Kubernetes, where same ip address can be re-used for new executor pod.
>            Reporter: BoYang
>            Priority: Critical
>              Labels: pull-request-available
>
> This is an edge case which caused Spark on Kubernetes getting fetch failed exception when new executor reused same ip address from a previously killed executor.
> The new executor checks shuffle block ip address and compares it with its own host address. If the two ip addresses are the same, the new executor will assume the block on its own local disk and try to read it locally. This causes failure since the block is actually on the previously killed executor which happened to have same ip address.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org