You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jo...@apache.org on 2015/04/08 19:27:01 UTC
spark git commit: [SPARK-6753] Clone SparkConf in ShuffleSuite tests
Repository: spark
Updated Branches:
refs/heads/master f7e21dd1e -> 9d44ddce1
[SPARK-6753] Clone SparkConf in ShuffleSuite tests
Prior to this change, the unit test for SPARK-3426 did not clone the
original SparkConf, which meant that that test did not use the options
set by suites that subclass ShuffleSuite.scala. This commit fixes that
problem.
JoshRosen would be great if you could take a look at this, since you wrote this
test originally.
Author: Kay Ousterhout <ka...@gmail.com>
Closes #5401 from kayousterhout/SPARK-6753 and squashes the following commits:
368c540 [Kay Ousterhout] [SPARK-6753] Clone SparkConf in ShuffleSuite tests
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9d44ddce
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9d44ddce
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9d44ddce
Branch: refs/heads/master
Commit: 9d44ddce1d1e19011026605549c37d0db6d6afa1
Parents: f7e21dd
Author: Kay Ousterhout <ka...@gmail.com>
Authored: Wed Apr 8 10:26:45 2015 -0700
Committer: Josh Rosen <jo...@databricks.com>
Committed: Wed Apr 8 10:26:45 2015 -0700
----------------------------------------------------------------------
core/src/test/scala/org/apache/spark/ShuffleSuite.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/9d44ddce/core/src/test/scala/org/apache/spark/ShuffleSuite.scala
----------------------------------------------------------------------
diff --git a/core/src/test/scala/org/apache/spark/ShuffleSuite.scala b/core/src/test/scala/org/apache/spark/ShuffleSuite.scala
index f57921b..30b6184 100644
--- a/core/src/test/scala/org/apache/spark/ShuffleSuite.scala
+++ b/core/src/test/scala/org/apache/spark/ShuffleSuite.scala
@@ -242,14 +242,14 @@ abstract class ShuffleSuite extends FunSuite with Matchers with LocalSparkContex
shuffleSpillCompress <- Set(true, false);
shuffleCompress <- Set(true, false)
) {
- val conf = new SparkConf()
+ val myConf = conf.clone()
.setAppName("test")
.setMaster("local")
.set("spark.shuffle.spill.compress", shuffleSpillCompress.toString)
.set("spark.shuffle.compress", shuffleCompress.toString)
.set("spark.shuffle.memoryFraction", "0.001")
resetSparkContext()
- sc = new SparkContext(conf)
+ sc = new SparkContext(myConf)
try {
sc.parallelize(0 until 100000).map(i => (i / 4, i)).groupByKey().collect()
} catch {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org