[jira] [Created] (SPARK-2299) Consolidate various stageIdTo* hash maps

["Reynold Xin (JIRA)" <ji...@apache.org>]

Reynold Xin created SPARK-2299:
----------------------------------

             Summary: Consolidate various stageIdTo* hash maps
                 Key: SPARK-2299
                 URL: https://issues.apache.org/jira/browse/SPARK-2299
             Project: Spark
          Issue Type: Improvement
          Components: Web UI
    Affects Versions: 1.0.0
            Reporter: Reynold Xin
            Assignee: Andrew Or


In JobProgressListener:

{code}
  val stageIdToTime = HashMap[Int, Long]()
  val stageIdToShuffleRead = HashMap[Int, Long]()
  val stageIdToShuffleWrite = HashMap[Int, Long]()
  val stageIdToMemoryBytesSpilled = HashMap[Int, Long]()
  val stageIdToDiskBytesSpilled = HashMap[Int, Long]()
  val stageIdToTasksActive = HashMap[Int, HashMap[Long, TaskInfo]]()
  val stageIdToTasksComplete = HashMap[Int, Int]()
  val stageIdToTasksFailed = HashMap[Int, Int]()
  val stageIdToTaskData = HashMap[Int, HashMap[Long, TaskUIData]]()
  val stageIdToExecutorSummaries = HashMap[Int, HashMap[String, ExecutorSummary]]()
  val stageIdToPool = HashMap[Int, String]()
  val stageIdToDescription = HashMap[Int, String]()
{code}

We should consolidate them to reduce memory & be less error prone. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)