You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Aaron Staple (JIRA)" <ji...@apache.org> on 2014/07/18 19:10:06 UTC
[jira] [Created] (SPARK-2581) complete or withdraw visitedStages optimization in DAGScheduler’s stageDependsOn
Aaron Staple created SPARK-2581:
-----------------------------------
Summary: complete or withdraw visitedStages optimization in DAGScheduler’s stageDependsOn
Key: SPARK-2581
URL: https://issues.apache.org/jira/browse/SPARK-2581
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Aaron Staple
Priority: Minor
Right now the visitedStages HashSet is populated with stages, but never queried to limit examination of previously visited stages. It may make sense to check whether a mapStage has been visited previously before visiting it again, as in the nearby visitedRdds check. Or it may be that the existing visitedRdds check sufficiently optimizes this function, and visitedStages can simply be removed.
See discussion here: https://github.com/apache/spark/pull/1362#discussion-diff-15018046L1107
--
This message was sent by Atlassian JIRA
(v6.2#6252)