You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2016/10/13 03:43:18 UTC
spark git commit: [SPARK-16827][BRANCH-2.0] Avoid reporting spill
metrics as shuffle metrics
Repository: spark
Updated Branches:
refs/heads/branch-2.0 050b8177e -> 5903dabc5
[SPARK-16827][BRANCH-2.0] Avoid reporting spill metrics as shuffle metrics
## What changes were proposed in this pull request?
Fix a bug where spill metrics were being reported as shuffle metrics. Eventually these spill metrics should be reported (SPARK-3577), but separate from shuffle metrics. The fix itself basically reverts the line to what it was in 1.6.
## How was this patch tested?
Cherry-picked from master (#15347)
Author: Brian Cho <bc...@fb.com>
Closes #15455 from dafrista/shuffle-metrics-2.0.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5903dabc
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5903dabc
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5903dabc
Branch: refs/heads/branch-2.0
Commit: 5903dabc57c07310573babe94e4f205bdea6455f
Parents: 050b817
Author: Brian Cho <bc...@fb.com>
Authored: Wed Oct 12 20:43:18 2016 -0700
Committer: Reynold Xin <rx...@databricks.com>
Committed: Wed Oct 12 20:43:18 2016 -0700
----------------------------------------------------------------------
.../spark/util/collection/unsafe/sort/UnsafeExternalSorter.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/5903dabc/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
index 999ded4..6e03064 100644
--- a/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
+++ b/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java
@@ -144,7 +144,9 @@ public final class UnsafeExternalSorter extends MemoryConsumer {
// Use getSizeAsKb (not bytes) to maintain backwards compatibility for units
// this.fileBufferSizeBytes = (int) conf.getSizeAsKb("spark.shuffle.file.buffer", "32k") * 1024;
this.fileBufferSizeBytes = 32 * 1024;
- this.writeMetrics = taskContext.taskMetrics().shuffleWriteMetrics();
+ // The spill metrics are stored in a new ShuffleWriteMetrics, and then discarded (this fixes SPARK-16827).
+ // TODO: Instead, separate spill metrics should be stored and reported (tracked in SPARK-3577).
+ this.writeMetrics = new ShuffleWriteMetrics();
if (existingInMemorySorter == null) {
this.inMemSorter = new UnsafeInMemorySorter(
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org