You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Aaron Staple (JIRA)" <ji...@apache.org> on 2014/07/18 19:10:06 UTC

[jira] [Created] (SPARK-2581) complete or withdraw visitedStages optimization in DAGScheduler’s stageDependsOn

Aaron Staple created SPARK-2581:
-----------------------------------

             Summary: complete or withdraw visitedStages optimization in DAGScheduler’s stageDependsOn
                 Key: SPARK-2581
                 URL: https://issues.apache.org/jira/browse/SPARK-2581
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
            Reporter: Aaron Staple
            Priority: Minor


Right now the visitedStages HashSet is populated with stages, but never queried to limit examination of previously visited stages.  It may make sense to check whether a mapStage has been visited previously before visiting it again, as in the nearby visitedRdds check.  Or it may be that the existing visitedRdds check sufficiently optimizes this function, and visitedStages can simply be removed.

See discussion here: https://github.com/apache/spark/pull/1362#discussion-diff-15018046L1107



--
This message was sent by Atlassian JIRA
(v6.2#6252)