You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by zs...@apache.org on 2016/09/01 06:25:23 UTC
spark git commit: [SPARK-17318][TESTS] Fix ReplSuite replicating
blocks of object with class defined in repl again
Repository: spark
Updated Branches:
refs/heads/master aaf632b21 -> 21c0a4fe9
[SPARK-17318][TESTS] Fix ReplSuite replicating blocks of object with class defined in repl again
## What changes were proposed in this pull request?
After digging into the logs, I noticed the failure is because in this test, it starts a local cluster with 2 executors. However, when SparkContext is created, executors may be still not up. When one of the executor is not up during running the job, the blocks won't be replicated.
This PR just adds a wait loop before running the job to fix the flaky test.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <sh...@databricks.com>
Closes #14905 from zsxwing/SPARK-17318-2.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/21c0a4fe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/21c0a4fe
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/21c0a4fe
Branch: refs/heads/master
Commit: 21c0a4fe9d8e21819ba96e7dc2b1f2999d3299ae
Parents: aaf632b
Author: Shixiong Zhu <sh...@databricks.com>
Authored: Wed Aug 31 23:25:20 2016 -0700
Committer: Shixiong Zhu <sh...@databricks.com>
Committed: Wed Aug 31 23:25:20 2016 -0700
----------------------------------------------------------------------
.../src/test/scala/org/apache/spark/repl/ReplSuite.scala | 9 +++++++++
1 file changed, 9 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/21c0a4fe/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
----------------------------------------------------------------------
diff --git a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
index f1284b1..f7d7a4f 100644
--- a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
+++ b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
@@ -399,6 +399,15 @@ class ReplSuite extends SparkFunSuite {
test("replicating blocks of object with class defined in repl") {
val output = runInterpreter("local-cluster[2,1,1024]",
"""
+ |val timeout = 60000 // 60 seconds
+ |val start = System.currentTimeMillis
+ |while(sc.getExecutorStorageStatus.size != 3 &&
+ | (System.currentTimeMillis - start) < timeout) {
+ | Thread.sleep(10)
+ |}
+ |if (System.currentTimeMillis - start >= timeout) {
+ | throw new java.util.concurrent.TimeoutException("Executors were not up in 60 seconds")
+ |}
|import org.apache.spark.storage.StorageLevel._
|case class Foo(i: Int)
|val ret = sc.parallelize((1 to 100).map(Foo), 10).persist(MEMORY_AND_DISK_2)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org