You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by va...@apache.org on 2018/01/10 17:44:43 UTC
spark git commit: [SPARK-23019][CORE] Wait until SparkContext.stop()
finished in SparkLauncherSuite
Repository: spark
Updated Branches:
refs/heads/master f340b6b30 -> 344e3aab8
[SPARK-23019][CORE] Wait until SparkContext.stop() finished in SparkLauncherSuite
## What changes were proposed in this pull request?
In current code ,the function `waitFor` call https://github.com/apache/spark/blob/cfcd746689c2b84824745fa6d327ffb584c7a17d/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java#L155 only wait until DAGScheduler is stopped, while SparkContext.clearActiveContext may not be called yet.
https://github.com/apache/spark/blob/1c9f95cb771ac78775a77edd1abfeb2d8ae2a124/core/src/main/scala/org/apache/spark/SparkContext.scala#L1924
Thus, in the Jenkins test
https://amplab.cs.berkeley.edu/jenkins/job/spark-branch-2.3-test-maven-hadoop-2.6/ , `JdbcRDDSuite` failed because the previous test `SparkLauncherSuite` exit before SparkContext.stop() is finished.
To repo:
```
$ build/sbt
> project core
> testOnly *SparkLauncherSuite *JavaJdbcRDDSuite
```
To Fix:
Wait for a reasonable amount of time to avoid creating two active SparkContext in JVM in SparkLauncherSuite.
Can' come up with any better solution for now.
## How was this patch tested?
Unit test
Author: Wang Gengliang <lt...@gmail.com>
Closes #20221 from gengliangwang/SPARK-23019.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/344e3aab
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/344e3aab
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/344e3aab
Branch: refs/heads/master
Commit: 344e3aab87178e45957333479a07e07f202ca1fd
Parents: f340b6b
Author: Wang Gengliang <lt...@gmail.com>
Authored: Wed Jan 10 09:44:30 2018 -0800
Committer: Marcelo Vanzin <va...@cloudera.com>
Committed: Wed Jan 10 09:44:30 2018 -0800
----------------------------------------------------------------------
.../test/java/org/apache/spark/launcher/SparkLauncherSuite.java | 5 +++++
1 file changed, 5 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/344e3aab/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java
----------------------------------------------------------------------
diff --git a/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java b/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java
index c2261c2..9d2f563 100644
--- a/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java
+++ b/core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java
@@ -23,6 +23,7 @@ import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.concurrent.TimeUnit;
import org.junit.Test;
import static org.junit.Assert.*;
@@ -133,6 +134,10 @@ public class SparkLauncherSuite extends BaseSuite {
p.put(e.getKey(), e.getValue());
}
System.setProperties(p);
+ // Here DAGScheduler is stopped, while SparkContext.clearActiveContext may not be called yet.
+ // Wait for a reasonable amount of time to avoid creating two active SparkContext in JVM.
+ // See SPARK-23019 and SparkContext.stop() for details.
+ TimeUnit.MILLISECONDS.sleep(500);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org