You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/03/16 10:05:31 UTC
spark git commit: [SPARK-12653][SQL] Re-enable test "SPARK-8489:
MissingRequirementError during reflection"
Repository: spark
Updated Branches:
refs/heads/master 92024797a -> 431a3d04b
[SPARK-12653][SQL] Re-enable test "SPARK-8489: MissingRequirementError during reflection"
## What changes were proposed in this pull request?
The purpose of [SPARK-12653](https://issues.apache.org/jira/browse/SPARK-12653) is re-enabling a regression test.
Historically, the target regression test is added by [SPARK-8498](https://github.com/apache/spark/commit/093c34838d1db7a9375f36a9a2ab5d96a23ae683), but is temporarily disabled by [SPARK-12615](https://github.com/apache/spark/commit/8ce645d4eeda203cf5e100c4bdba2d71edd44e6a) due to binary compatibility error.
The following is the current error message at the submitting spark job with the pre-built `test.jar` file in the target regression test.
```
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.SparkContext$.$lessinit$greater$default$6()Lscala/collection/Map;
```
Simple rebuilding `test.jar` can not recover the purpose of testcase since we need to support both Scala 2.10 and 2.11 for a while. For example, we will face the following Scala 2.11 error if we use `test.jar` built by Scala 2.10.
```
Exception in thread "main" java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
```
This PR replace the existing `test.jar` with `test-2.10.jar` and `test-2.11.jar` and improve the regression test to use the suitable jar file.
## How was this patch tested?
Pass the existing Jenkins test.
Author: Dongjoon Hyun <do...@apache.org>
Closes #11744 from dongjoon-hyun/SPARK-12653.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/431a3d04
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/431a3d04
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/431a3d04
Branch: refs/heads/master
Commit: 431a3d04b437df4ab323fd925f57873aa6b8a0c8
Parents: 9202479
Author: Dongjoon Hyun <do...@apache.org>
Authored: Wed Mar 16 09:05:53 2016 +0000
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Mar 16 09:05:53 2016 +0000
----------------------------------------------------------------------
.../regression-test-SPARK-8489/test-2.10.jar | Bin 0 -> 6873 bytes
.../regression-test-SPARK-8489/test-2.11.jar | Bin 0 -> 7039 bytes
.../resources/regression-test-SPARK-8489/test.jar | Bin 6828 -> 0 bytes
.../spark/sql/hive/HiveSparkSubmitSuite.scala | 9 +++++++--
4 files changed, 7 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar
new file mode 100644
index 0000000..26d410f
Binary files /dev/null and b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.10.jar differ
http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar
new file mode 100644
index 0000000..f347847
Binary files /dev/null and b/sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar differ
http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar b/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar
deleted file mode 100644
index 5944aa6..0000000
Binary files a/sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar and /dev/null differ
http://git-wip-us.apache.org/repos/asf/spark/blob/431a3d04/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
----------------------------------------------------------------------
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
index 12a5542..c14c94f 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveSparkSubmitSuite.scala
@@ -22,6 +22,7 @@ import java.sql.Timestamp
import java.util.Date
import scala.collection.mutable.ArrayBuffer
+import scala.tools.nsc.Properties
import org.scalatest.{BeforeAndAfterEach, Matchers}
import org.scalatest.concurrent.Timeouts
@@ -87,13 +88,17 @@ class HiveSparkSubmitSuite
runSparkSubmit(args)
}
- ignore("SPARK-8489: MissingRequirementError during reflection") {
+ test("SPARK-8489: MissingRequirementError during reflection") {
// This test uses a pre-built jar to test SPARK-8489. In a nutshell, this test creates
// a HiveContext and uses it to create a data frame from an RDD using reflection.
// Before the fix in SPARK-8470, this results in a MissingRequirementError because
// the HiveContext code mistakenly overrides the class loader that contains user classes.
// For more detail, see sql/hive/src/test/resources/regression-test-SPARK-8489/*scala.
- val testJar = "sql/hive/src/test/resources/regression-test-SPARK-8489/test.jar"
+ val version = Properties.versionNumberString match {
+ case v if v.startsWith("2.10") || v.startsWith("2.11") => v.substring(0, 4)
+ case x => throw new Exception(s"Unsupported Scala Version: $x")
+ }
+ val testJar = s"sql/hive/src/test/resources/regression-test-SPARK-8489/test-$version.jar"
val args = Seq(
"--conf", "spark.ui.enabled=false",
"--conf", "spark.master.rest.enabled=false",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org