You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Takuya Ueshin (JIRA)" <ji...@apache.org> on 2016/03/15 09:50:33 UTC
[jira] [Updated] (SPARK-13902) Make
DAGScheduler.getAncestorShuffleDependencies() return in topological order
to ensure building ancestor stages first.
[ https://issues.apache.org/jira/browse/SPARK-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Takuya Ueshin updated SPARK-13902:
----------------------------------
Description:
{{DAGScheduler}} sometimes generate incorrect stage graph.
Some stages are generated for the same shuffleId twice or more and they are referenced by the child stages because the building order of the graph is not correct.
I added the sample RDD graph to show the illegal stage graph to {{DAGSchedulerSuite}} and then fixed it.
was:
{{DAGScheduler}} sometimes generate correct stage graph.
Some stages are generated for the same shuffleId twice or more and they are referenced by the child stages because the building order of the graph is not correct.
I added the sample RDD graph to show the illegal stage graph to {{DAGSchedulerSuite}} and then fixed it.
> Make DAGScheduler.getAncestorShuffleDependencies() return in topological order to ensure building ancestor stages first.
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-13902
> URL: https://issues.apache.org/jira/browse/SPARK-13902
> Project: Spark
> Issue Type: Bug
> Components: Scheduler
> Reporter: Takuya Ueshin
>
> {{DAGScheduler}} sometimes generate incorrect stage graph.
> Some stages are generated for the same shuffleId twice or more and they are referenced by the child stages because the building order of the graph is not correct.
> I added the sample RDD graph to show the illegal stage graph to {{DAGSchedulerSuite}} and then fixed it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org