You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2017/05/03 18:09:40 UTC
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/17844
[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases
## What changes were proposed in this pull request?
`ReplSuite.newProductSeqEncoder with REPL defined class` was flaky and throws OOM exception frequently. By analyzing the heap dump, we found the reason is that, in each test case of `ReplSuite`, we create a REPL instance, which creates a classloader and loads a lot of classes related to `SparkContext`. More details please see https://github.com/apache/spark/pull/17833#issuecomment-298711435.
In this PR, we create a new test suite, `SingletonReplSuite`, which shares one REPL instances among all the test cases. Then we move most of the tests from `ReplSuite` to `SingletonReplSuite`, to avoid creating a lot of REPL instances and reduce memory footprint.
## How was this patch tested?
test only change
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark flaky-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17844.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17844
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76600/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by sameeragarwal <gi...@git.apache.org>.
Github user sameeragarwal commented on the issue:
https://github.com/apache/spark/pull/17844
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76656 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76656/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76675/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17844
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76618/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115097789
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -373,52 +190,6 @@ class ReplSuite extends SparkFunSuite {
}
}
- test("collecting objects of class defined in repl") {
- val output = runInterpreter("local[2]",
- """
- |case class Foo(i: Int)
- |val ret = sc.parallelize((1 to 100).map(Foo), 10).collect()
- """.stripMargin)
- assertDoesNotContain("error:", output)
- assertDoesNotContain("Exception", output)
- assertContains("ret: Array[Foo] = Array(Foo(1),", output)
- }
-
- test("collecting objects of class defined in repl - shuffling") {
- val output = runInterpreter("local-cluster[1,1,1024]",
- """
- |case class Foo(i: Int)
- |val list = List((1, Foo(1)), (1, Foo(2)))
- |val ret = sc.parallelize(list).groupByKey().collect()
- """.stripMargin)
- assertDoesNotContain("error:", output)
- assertDoesNotContain("Exception", output)
- assertContains("ret: Array[(Int, Iterable[Foo])] = Array((1,", output)
- }
-
- test("replicating blocks of object with class defined in repl") {
- val output = runInterpreter("local-cluster[2,1,1024]",
- """
- |val timeout = 60000 // 60 seconds
- |val start = System.currentTimeMillis
- |while(sc.getExecutorStorageStatus.size != 3 &&
- | (System.currentTimeMillis - start) < timeout) {
- | Thread.sleep(10)
- |}
- |if (System.currentTimeMillis - start >= timeout) {
- | throw new java.util.concurrent.TimeoutException("Executors were not up in 60 seconds")
- |}
- |import org.apache.spark.storage.StorageLevel._
- |case class Foo(i: Int)
- |val ret = sc.parallelize((1 to 100).map(Foo), 10).persist(MEMORY_AND_DISK_2)
- |ret.count()
- |sc.getExecutorStorageStatus.map(s => s.rddBlocksById(ret.id).size).sum
- """.stripMargin)
- assertDoesNotContain("error:", output)
- assertDoesNotContain("Exception", output)
- assertContains(": Int = 20", output)
- }
-
test("line wrapper only initialized once when used as encoder outer scope") {
--- End diff --
Why you don't move this test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17844
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17844
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76582 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76582/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17844
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76423/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76675/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76656 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76656/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76618/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76471/testReport)** for PR 17844 at commit [`9248a5e`](https://github.com/apache/spark/commit/9248a5e005c000c42e5a233c9f3ca37b51b6c95d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76582 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76582/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76569/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
* This patch **fails from timeout after a configured wait of \`250m\`**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76471/testReport)** for PR 17844 at commit [`9248a5e`](https://github.com/apache/spark/commit/9248a5e005c000c42e5a233c9f3ca37b51b6c95d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76461/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76600/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76675/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76423 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76423/testReport)** for PR 17844 at commit [`1565a34`](https://github.com/apache/spark/commit/1565a34448cfa08f9afd06c38666ff65fafec7d4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76569/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76577/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115200884
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala ---
@@ -373,52 +190,6 @@ class ReplSuite extends SparkFunSuite {
}
}
- test("collecting objects of class defined in repl") {
- val output = runInterpreter("local[2]",
- """
- |case class Foo(i: Int)
- |val ret = sc.parallelize((1 to 100).map(Foo), 10).collect()
- """.stripMargin)
- assertDoesNotContain("error:", output)
- assertDoesNotContain("Exception", output)
- assertContains("ret: Array[Foo] = Array(Foo(1),", output)
- }
-
- test("collecting objects of class defined in repl - shuffling") {
- val output = runInterpreter("local-cluster[1,1,1024]",
- """
- |case class Foo(i: Int)
- |val list = List((1, Foo(1)), (1, Foo(2)))
- |val ret = sc.parallelize(list).groupByKey().collect()
- """.stripMargin)
- assertDoesNotContain("error:", output)
- assertDoesNotContain("Exception", output)
- assertContains("ret: Array[(Int, Iterable[Foo])] = Array((1,", output)
- }
-
- test("replicating blocks of object with class defined in repl") {
- val output = runInterpreter("local-cluster[2,1,1024]",
- """
- |val timeout = 60000 // 60 seconds
- |val start = System.currentTimeMillis
- |while(sc.getExecutorStorageStatus.size != 3 &&
- | (System.currentTimeMillis - start) < timeout) {
- | Thread.sleep(10)
- |}
- |if (System.currentTimeMillis - start >= timeout) {
- | throw new java.util.concurrent.TimeoutException("Executors were not up in 60 seconds")
- |}
- |import org.apache.spark.storage.StorageLevel._
- |case class Foo(i: Int)
- |val ret = sc.parallelize((1 to 100).map(Foo), 10).persist(MEMORY_AND_DISK_2)
- |ret.count()
- |sc.getExecutorStorageStatus.map(s => s.rddBlocksById(ret.id).size).sum
- """.stripMargin)
- assertDoesNotContain("error:", output)
- assertDoesNotContain("Exception", output)
- assertContains(": Int = 20", output)
- }
-
test("line wrapper only initialized once when used as encoder outer scope") {
--- End diff --
this test only works in local mode...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17844
thanks for the review, merging to maser/2.2!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115095994
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala ---
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.repl
+
+import java.io._
+import java.net.URLClassLoader
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.util.Utils
+
+/**
+ * A special test suite for REPL that all test cases share one REPL instance.
+ */
+class SingletonReplSuite extends SparkFunSuite {
+
+ private val out = new StringWriter()
+ private val in = new PipedOutputStream()
+ private var thread: Thread = _
+
+ private val CONF_EXECUTOR_CLASSPATH = "spark.executor.extraClassPath"
+ private val oldExecutorClasspath = System.getProperty(CONF_EXECUTOR_CLASSPATH)
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ val cl = getClass.getClassLoader
+ var paths = new ArrayBuffer[String]
+ if (cl.isInstanceOf[URLClassLoader]) {
+ val urlLoader = cl.asInstanceOf[URLClassLoader]
+ for (url <- urlLoader.getURLs) {
+ if (url.getProtocol == "file") {
+ paths += url.getFile
+ }
+ }
+ }
+ val classpath = paths.map(new File(_).getAbsolutePath).mkString(File.pathSeparator)
+
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, classpath)
+ Main.conf.set("spark.master", "local-cluster[2,4,4096]")
+ val interp = new SparkILoop(
+ new BufferedReader(new InputStreamReader(new PipedInputStream(in))),
+ new PrintWriter(out))
+
+ // Forces to create new SparkContext
+ Main.sparkContext = null
+ Main.sparkSession = null
+
+ // Starts a new thread to run the REPL interpreter, so that we won't block.
+ thread = new Thread(new Runnable {
+ override def run(): Unit = Main.doMain(Array("-classpath", classpath), interp)
+ })
+ thread.start()
+
+ waitUntil(() => out.toString.contains("Type :help for more information"))
+ }
+
+ override def afterAll(): Unit = {
+ in.close()
+ thread.join()
+ if (oldExecutorClasspath != null) {
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, oldExecutorClasspath)
+ } else {
+ System.clearProperty(CONF_EXECUTOR_CLASSPATH)
+ }
+ super.afterAll()
+ }
+
+ private def waitUntil(cond: () => Boolean): Unit = {
+ var i = 0
+ while (i < 100 && !cond()) {
+ Thread.sleep(500)
+ i += 1
+ }
+ if (i == 100) {
+ throw new IllegalStateException("timeout after 50 seconds, current output: " + out.toString)
+ }
+ }
+
+ def runInterpreter(input: String): String = {
--- End diff --
Could you add a comment mentioning that the codes in `input` should not throw an exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17844
cc @zsxwing @sameeragarwal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115098412
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala ---
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.repl
+
+import java.io._
+import java.net.URLClassLoader
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.util.Utils
+
+/**
+ * A special test suite for REPL that all test cases share one REPL instance.
+ */
+class SingletonReplSuite extends SparkFunSuite {
+
+ private val out = new StringWriter()
+ private val in = new PipedOutputStream()
+ private var thread: Thread = _
+
+ private val CONF_EXECUTOR_CLASSPATH = "spark.executor.extraClassPath"
+ private val oldExecutorClasspath = System.getProperty(CONF_EXECUTOR_CLASSPATH)
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ val cl = getClass.getClassLoader
+ var paths = new ArrayBuffer[String]
+ if (cl.isInstanceOf[URLClassLoader]) {
+ val urlLoader = cl.asInstanceOf[URLClassLoader]
+ for (url <- urlLoader.getURLs) {
+ if (url.getProtocol == "file") {
+ paths += url.getFile
+ }
+ }
+ }
+ val classpath = paths.map(new File(_).getAbsolutePath).mkString(File.pathSeparator)
+
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, classpath)
+ Main.conf.set("spark.master", "local-cluster[2,4,4096]")
+ val interp = new SparkILoop(
+ new BufferedReader(new InputStreamReader(new PipedInputStream(in))),
+ new PrintWriter(out))
+
+ // Forces to create new SparkContext
+ Main.sparkContext = null
+ Main.sparkSession = null
+
+ // Starts a new thread to run the REPL interpreter, so that we won't block.
+ thread = new Thread(new Runnable {
--- End diff --
nit: please call `Thread.setDaemon(true)` in case that something is wrong and it prevents the process from exiting.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76461/testReport)** for PR 17844 at commit [`1565a34`](https://github.com/apache/spark/commit/1565a34448cfa08f9afd06c38666ff65fafec7d4).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76582/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/17844
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76649 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76649/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76461/testReport)** for PR 17844 at commit [`1565a34`](https://github.com/apache/spark/commit/1565a34448cfa08f9afd06c38666ff65fafec7d4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76423 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76423/testReport)** for PR 17844 at commit [`1565a34`](https://github.com/apache/spark/commit/1565a34448cfa08f9afd06c38666ff65fafec7d4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76577/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76600 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76600/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76656/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76649/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76471/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76649 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76649/testReport)** for PR 17844 at commit [`7dde745`](https://github.com/apache/spark/commit/7dde745dbd5b629fb7cff598f417d00494bcbc00).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115098191
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala ---
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.repl
+
+import java.io._
+import java.net.URLClassLoader
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.util.Utils
+
+/**
+ * A special test suite for REPL that all test cases share one REPL instance.
+ */
+class SingletonReplSuite extends SparkFunSuite {
+
+ private val out = new StringWriter()
+ private val in = new PipedOutputStream()
+ private var thread: Thread = _
+
+ private val CONF_EXECUTOR_CLASSPATH = "spark.executor.extraClassPath"
+ private val oldExecutorClasspath = System.getProperty(CONF_EXECUTOR_CLASSPATH)
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ val cl = getClass.getClassLoader
+ var paths = new ArrayBuffer[String]
+ if (cl.isInstanceOf[URLClassLoader]) {
+ val urlLoader = cl.asInstanceOf[URLClassLoader]
+ for (url <- urlLoader.getURLs) {
+ if (url.getProtocol == "file") {
+ paths += url.getFile
+ }
+ }
+ }
+ val classpath = paths.map(new File(_).getAbsolutePath).mkString(File.pathSeparator)
+
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, classpath)
+ Main.conf.set("spark.master", "local-cluster[2,4,4096]")
--- End diff --
nit: I think it's safe to just use `local-cluster[2,1,1024]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76569/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115096431
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala ---
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.repl
+
+import java.io._
+import java.net.URLClassLoader
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.util.Utils
+
+/**
+ * A special test suite for REPL that all test cases share one REPL instance.
+ */
+class SingletonReplSuite extends SparkFunSuite {
+
+ private val out = new StringWriter()
+ private val in = new PipedOutputStream()
+ private var thread: Thread = _
+
+ private val CONF_EXECUTOR_CLASSPATH = "spark.executor.extraClassPath"
+ private val oldExecutorClasspath = System.getProperty(CONF_EXECUTOR_CLASSPATH)
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ val cl = getClass.getClassLoader
+ var paths = new ArrayBuffer[String]
+ if (cl.isInstanceOf[URLClassLoader]) {
+ val urlLoader = cl.asInstanceOf[URLClassLoader]
+ for (url <- urlLoader.getURLs) {
+ if (url.getProtocol == "file") {
+ paths += url.getFile
+ }
+ }
+ }
+ val classpath = paths.map(new File(_).getAbsolutePath).mkString(File.pathSeparator)
+
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, classpath)
+ Main.conf.set("spark.master", "local-cluster[2,4,4096]")
+ val interp = new SparkILoop(
+ new BufferedReader(new InputStreamReader(new PipedInputStream(in))),
+ new PrintWriter(out))
+
+ // Forces to create new SparkContext
+ Main.sparkContext = null
+ Main.sparkSession = null
+
+ // Starts a new thread to run the REPL interpreter, so that we won't block.
+ thread = new Thread(new Runnable {
+ override def run(): Unit = Main.doMain(Array("-classpath", classpath), interp)
+ })
+ thread.start()
+
+ waitUntil(() => out.toString.contains("Type :help for more information"))
+ }
+
+ override def afterAll(): Unit = {
+ in.close()
+ thread.join()
+ if (oldExecutorClasspath != null) {
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, oldExecutorClasspath)
+ } else {
+ System.clearProperty(CONF_EXECUTOR_CLASSPATH)
+ }
+ super.afterAll()
+ }
+
+ private def waitUntil(cond: () => Boolean): Unit = {
--- End diff --
You can use `eventually`
```
eventually(timeout(50.seconds), interval(500.millis)) {
assert(cond(), "current output: " + out.toString)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by zsxwing <gi...@git.apache.org>.
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/17844#discussion_r115060517
--- Diff: repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala ---
@@ -0,0 +1,404 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.repl
+
+import java.io._
+import java.net.URLClassLoader
+
+import scala.collection.mutable.ArrayBuffer
+
+import org.apache.commons.lang3.StringEscapeUtils
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.util.Utils
+
+/**
+ * A special test suite for REPL that all test cases share one REPL instance.
+ */
+class SingletonReplSuite extends SparkFunSuite {
+
+ private val out = new StringWriter()
+ private val in = new PipedOutputStream()
+ private var thread: Thread = _
+
+ private val CONF_EXECUTOR_CLASSPATH = "spark.executor.extraClassPath"
+ private val oldExecutorClasspath = System.getProperty(CONF_EXECUTOR_CLASSPATH)
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+
+ val cl = getClass.getClassLoader
+ var paths = new ArrayBuffer[String]
+ if (cl.isInstanceOf[URLClassLoader]) {
+ val urlLoader = cl.asInstanceOf[URLClassLoader]
+ for (url <- urlLoader.getURLs) {
+ if (url.getProtocol == "file") {
+ paths += url.getFile
+ }
+ }
+ }
+ val classpath = paths.map(new File(_).getAbsolutePath).mkString(File.pathSeparator)
+
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, classpath)
+ Main.conf.set("spark.master", "local-cluster[2,4,4096]")
+ val interp = new SparkILoop(
+ new BufferedReader(new InputStreamReader(new PipedInputStream(in))),
+ new PrintWriter(out))
+
+ // Forces to create new SparkContext
+ Main.sparkContext = null
+ Main.sparkSession = null
+
+ // Starts a new thread to run the REPL interpreter, so that we won't block.
+ thread = new Thread(new Runnable {
+ override def run(): Unit = Main.doMain(Array("-classpath", classpath), interp)
+ })
+ thread.start()
+
+ waitUntil(() => out.toString.contains("Type :help for more information"))
+ }
+
+ override def afterAll(): Unit = {
+ in.close()
+ thread.join()
+ if (oldExecutorClasspath != null) {
+ System.setProperty(CONF_EXECUTOR_CLASSPATH, oldExecutorClasspath)
+ } else {
+ System.clearProperty(CONF_EXECUTOR_CLASSPATH)
+ }
+ super.afterAll()
+ }
+
+ private def waitUntil(cond: () => Boolean): Unit = {
+ var i = 0
+ while (i < 100 && !cond()) {
+ Thread.sleep(500)
+ i += 1
+ }
+ if (i == 100) {
+ throw new IllegalStateException("timeout after 50 seconds, current output: " + out.toString)
+ }
+ }
+
+ def runInterpreter(input: String): String = {
+ val currentOffset = out.getBuffer.length()
+ // append a `val _result = 1` statement to the end of the given code, so that we can know what's
+ // the final output of this code snippet and rely on it to wait until the output is ready.
+ in.write((input + s"\nval _result = 1\n").getBytes)
--- End diff --
nit: it's better to use some more special string. E.g.,
```
val timestamp = System.currentTimeMillis()
in.write((input + s"\nval _result_$timestamp = 1\n").getBytes)
in.flush()
val stopMessage = s"_result_$timestamp: Int = 1"
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17844
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance among ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17844
**[Test build #76577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76577/testReport)** for PR 17844 at commit [`fd5b43b`](https://github.com/apache/spark/commit/fd5b43b7ca7ba03e96a64537f5f015341af4793c).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17844: [SPARK-20548][FLAKY-TEST] share one REPL instance...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17844
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org