You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "zhuangxian (Jira)" <ji...@apache.org> on 2024/09/24 03:40:00 UTC
[jira] [Updated] (SPARK-49762) How to handling Task Timeouts and Placeholder Allocation in Spark Shuffle Write Phase
[ https://issues.apache.org/jira/browse/SPARK-49762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhuangxian updated SPARK-49762:
-------------------------------
Affects Version/s: 2.4.1
(was: 3.5.1)
> How to handling Task Timeouts and Placeholder Allocation in Spark Shuffle Write Phase
> -------------------------------------------------------------------------------------
>
> Key: SPARK-49762
> URL: https://issues.apache.org/jira/browse/SPARK-49762
> Project: Spark
> Issue Type: Wish
> Components: Spark Core
> Affects Versions: 2.4.1
> Reporter: zhuangxian
> Priority: Major
>
> During the Spark shuffle write phase, the driver initiates a task to write a partition and has allocated a placeholder for the commit to that task. However, when dealing with a large volume of data, the task may fail to complete the commit task due to network issues or disk failures. In such cases, how should the driver detect the task timeout and launch a new task to commit the task for the same partition? Additionally, starting a new task raises the following issues: 1.Since the placeholder is occupied by the old task, the new task cannot obtain the placeholder for submission. How should the new task be allocated a placeholder? 2.How can the old task exit safely to ensure it does not commit the same data as the new task?
> The commit protocol is the 2PC. The main process is placeholder->move->commit, [github.com/apache/spark/blob/master/core/src/main/scala/org/…|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/OutputCommitCoordinator.scala] here is the specific implementation.
> And the commit algorithm I used is v2.
> I tried searching in the history but could not find a solution to this problem. I look forward to discussing this issue with community members.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org