You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2015/03/13 14:21:08 UTC
spark git commit: [SPARK-5845][Shuffle] Time to cleanup spilled
shuffle files not included in shuffle write time
Repository: spark
Updated Branches:
refs/heads/master 3980ebdf1 -> 0af9ea74a
[SPARK-5845][Shuffle] Time to cleanup spilled shuffle files not included in shuffle write time
I've added a timer in the right place to fix this inaccuracy.
Author: Ilya Ganelin <il...@capitalone.com>
Closes #4965 from ilganeli/SPARK-5845 and squashes the following commits:
bfabf88 [Ilya Ganelin] Changed to using a foreach vs. getorelse
3e059b0 [Ilya Ganelin] Switched to using getorelse
b946d08 [Ilya Ganelin] Fixed error with option
9434b50 [Ilya Ganelin] Merge remote-tracking branch 'upstream/master' into SPARK-5845
db8647e [Ilya Ganelin] Added update for shuffleWriteTime around spilled file cleanup in ExternalSorter
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0af9ea74
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0af9ea74
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0af9ea74
Branch: refs/heads/master
Commit: 0af9ea74a07ecdc08c43fa63cb9c9f0c57e3029b
Parents: 3980ebd
Author: Ilya Ganelin <il...@capitalone.com>
Authored: Fri Mar 13 13:21:04 2015 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Fri Mar 13 13:21:04 2015 +0000
----------------------------------------------------------------------
.../scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala | 3 +++
1 file changed, 3 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/0af9ea74/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala b/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
index 27496c5..fa2e617 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleWriter.scala
@@ -88,7 +88,10 @@ private[spark] class SortShuffleWriter[K, V, C](
} finally {
// Clean up our sorter, which may have its own intermediate files
if (sorter != null) {
+ val startTime = System.nanoTime()
sorter.stop()
+ context.taskMetrics.shuffleWriteMetrics.foreach(
+ _.incShuffleWriteTime(System.nanoTime - startTime))
sorter = null
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org