You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/09/02 09:26:44 UTC
spark git commit: [SPARK-17352][WEBUI] Executor computing time can be
negative-number because of calculation error
Repository: spark
Updated Branches:
refs/heads/master a3097e2b3 -> 7ee24dac8
[SPARK-17352][WEBUI] Executor computing time can be negative-number because of calculation error
## What changes were proposed in this pull request?
In StagePage, executor-computing-time is calculated but calculation error can occur potentially because it's calculated by subtraction of floating numbers.
Following capture is an example.
<img width="949" alt="capture-timeline" src="https://cloud.githubusercontent.com/assets/4736016/18152359/43f07a28-7030-11e6-8cbd-8e73bf4c4c67.png">
## How was this patch tested?
Manual tests.
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Closes #14908 from sarutak/SPARK-17352.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/7ee24dac
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/7ee24dac
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/7ee24dac
Branch: refs/heads/master
Commit: 7ee24dac8e779f6a9bf45371fdc2be83fb679cb2
Parents: a3097e2
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Authored: Fri Sep 2 10:26:43 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Fri Sep 2 10:26:43 2016 +0100
----------------------------------------------------------------------
core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/7ee24dac/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
index a266164..de787f2 100644
--- a/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
+++ b/core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
@@ -651,9 +651,9 @@ private[ui] class StagePage(parent: StagesTab) extends WebUIPage("stage") {
}
val executorComputingTime = executorRunTime - shuffleReadTime - shuffleWriteTime
val executorComputingTimeProportion =
- (100 - schedulerDelayProportion - shuffleReadTimeProportion -
+ math.max(100 - schedulerDelayProportion - shuffleReadTimeProportion -
shuffleWriteTimeProportion - serializationTimeProportion -
- deserializationTimeProportion - gettingResultTimeProportion)
+ deserializationTimeProportion - gettingResultTimeProportion, 0)
val schedulerDelayProportionPos = 0
val deserializationTimeProportionPos =
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org