You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liyezhang556520 <gi...@git.apache.org> on 2014/11/01 04:53:35 UTC

[GitHub] spark pull request: [SPARK-4168][WebUI] web statges number should ...

Github user liyezhang556520 commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3035#discussion_r19701557
  
    --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala ---
    @@ -59,6 +59,11 @@ class JobProgressListener(conf: SparkConf) extends SparkListener with Logging {
       val failedStages = ListBuffer[StageInfo]()
       val stageIdToData = new HashMap[(StageId, StageAttemptId), StageUIData]
       val stageIdToInfo = new HashMap[StageId, StageInfo]
    +  
    +  // Number of completed stages, not equal to completedStages.size
    +  var completedStagesNum = 0
    +  // number of failed stages, not equal to failedStagesNum.size
    --- End diff --
    
    Hi @andrewor14 , if stages are too large, may consume too much memory. So there is a mechanism to limit the buffer maintaining the `completedStages` and `failedStages`. And if the stage size is large than `retainedStages`, the starting part of the completed stages will be removed. See the code here ([trimIfNecessary](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/jobs/JobProgressListener.scala#L121)). So if the stage number is too large, `completedStages.size` is not the actually the real completedStages. Currently, the threshold is 1000. So the completed stages or failed stages showed on the WebUI will never higher than 1000, it will chages between 900~1000 if the stage number is more than 1000.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org