You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2018/02/15 11:52:48 UTC
spark git commit: [SPARK-23422][CORE] YarnShuffleIntegrationSuite fix when SPARK_PREPEN…
Repository: spark
Updated Branches:
refs/heads/master ed8647609 -> 44e20c422
[SPARK-23422][CORE] YarnShuffleIntegrationSuite fix when SPARK_PREPEN…
…D_CLASSES set to 1
## What changes were proposed in this pull request?
YarnShuffleIntegrationSuite fails when SPARK_PREPEND_CLASSES set to 1.
Normally mllib built before yarn module. When SPARK_PREPEND_CLASSES used mllib classes are on yarn test classpath.
Before 2.3 that did not cause issues. But 2.3 has SPARK-22450, which registered some mllib classes with the kryo serializer. Now it dies with the following error:
`
18/02/13 07:33:29 INFO SparkContext: Starting job: collect at YarnShuffleIntegrationSuite.scala:143
Exception in thread "dag-scheduler-event-loop" java.lang.NoClassDefFoundError: breeze/linalg/DenseMatrix
`
In this PR NoClassDefFoundError caught only in case of testing and then do nothing.
## How was this patch tested?
Automated: Pass the Jenkins.
Author: Gabor Somogyi <ga...@gmail.com>
Closes #20608 from gaborgsomogyi/SPARK-23422.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/44e20c42
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/44e20c42
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/44e20c42
Branch: refs/heads/master
Commit: 44e20c42254bc6591b594f54cd94ced5fcfadae3
Parents: ed86476
Author: Gabor Somogyi <ga...@gmail.com>
Authored: Thu Feb 15 03:52:40 2018 -0800
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Thu Feb 15 03:52:40 2018 -0800
----------------------------------------------------------------------
.../src/main/scala/org/apache/spark/serializer/KryoSerializer.scala | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/44e20c42/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index 538ae05..72427dd 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -206,6 +206,7 @@ class KryoSerializer(conf: SparkConf)
kryo.register(clazz)
} catch {
case NonFatal(_) => // do nothing
+ case _: NoClassDefFoundError if Utils.isTesting => // See SPARK-23422.
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org