You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/01/15 08:08:05 UTC
[3/3] git commit: Merge pull request #435 from tdas/filestream-fix
Merge pull request #435 from tdas/filestream-fix
Fixed the flaky tests by making SparkConf not serializable
SparkConf was being serialized with CoGroupedRDD and Aggregator, which somehow caused OptionalJavaException while being deserialized as part of a ShuffleMapTask. SparkConf should not even be serializable (according to conversation with Matei). This change fixes that.
@mateiz @pwendell
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/139c24ef
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/139c24ef
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/139c24ef
Branch: refs/heads/master
Commit: 139c24ef08e6ffb090975c9808a2cba304eb79e0
Parents: 087487e 0e15bd7
Author: Patrick Wendell <pw...@gmail.com>
Authored: Tue Jan 14 23:07:55 2014 -0800
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Tue Jan 14 23:07:55 2014 -0800
----------------------------------------------------------------------
.../scala/org/apache/spark/Aggregator.scala | 3 +--
.../main/scala/org/apache/spark/SparkConf.scala | 3 ++-
.../org/apache/spark/rdd/CoGroupedRDD.scala | 3 +--
.../flume/src/test/resources/log4j.properties | 2 +-
.../kafka/src/test/resources/log4j.properties | 2 +-
.../streaming/kafka/KafkaStreamSuite.scala | 1 +
.../mqtt/src/test/resources/log4j.properties | 2 +-
.../spark/streaming/mqtt/MQTTStreamSuite.scala | 1 +
.../twitter/src/test/resources/log4j.properties | 2 +-
.../streaming/twitter/TwitterStreamSuite.scala | 1 +
.../zeromq/src/test/resources/log4j.properties | 2 +-
.../streaming/zeromq/ZeroMQStreamSuite.scala | 1 +
.../org/apache/spark/streaming/Checkpoint.scala | 10 ++++---
.../apache/spark/streaming/DStreamGraph.scala | 2 ++
.../dstream/DStreamCheckpointData.scala | 26 +++++++++++++++++-
.../spark/streaming/CheckpointSuite.scala | 28 ++++++++++++++------
16 files changed, 66 insertions(+), 23 deletions(-)
----------------------------------------------------------------------