You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Li Yuanjian (JIRA)" <ji...@apache.org> on 2018/06/06 15:56:00 UTC
[jira] [Comment Edited] (SPARK-24375) Design sketch: support
barrier scheduling in Apache Spark
[ https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503486#comment-16503486 ]
Li Yuanjian edited comment on SPARK-24375 at 6/6/18 3:55 PM:
-------------------------------------------------------------
Hi [~cloud_fan] and [~jiangxb1987], just I tiny question here, I notice the discussion in SPARK-20928 mentioned the barrier scheduling which Continuous Processing will depend on.
{quote}
A barrier stage doesn’t launch any of its tasks until the available slots(free CPU cores can be used to launch pending tasks) satisfies the target to launch all the tasks at the same time, and always retry the whole stage when any task(s) fail.
{quote}
Why the task level retrying was forbidden here, is there any possible to achieve this? Thanks.
was (Author: xuanyuan):
Hi [~cloud_fan] and [~jiangxb1987], just I tiny question here, I notice the discussion in SPARK-20928 mentioned the barrier scheduling.
{quote}
A barrier stage doesn’t launch any of its tasks until the available slots(free CPU cores can be used to launch pending tasks) satisfies the target to launch all the tasks at the same time, and always retry the whole stage when any task(s) fail.
{quote}
Why the task level retrying was forbidden here, is there any possible to achieve this? Thanks.
> Design sketch: support barrier scheduling in Apache Spark
> ---------------------------------------------------------
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Assignee: Jiang Xingbo
> Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP discussion. It doesn't need to be a complete design before the vote. But it should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org