You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by ilganeli <gi...@git.apache.org> on 2015/04/14 01:53:45 UTC
[GitHub] spark pull request: [SPARK-6703][Core][WIP] Provide a way to disco...
GitHub user ilganeli opened a pull request:
https://github.com/apache/spark/pull/5501
[SPARK-6703][Core][WIP] Provide a way to discover existing SparkContext's
I've added a static getOrCreate method to the static SparkContext object that allows one to either retrieve a previously created SparkContext or to instantiate a new one with the provided config. The method accepts an optional SparkConf to make usage intuitive.
Still working on a test for this, basically want to create a new context from scratch, then ensure that subsequent calls don't overwrite that.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ilganeli/spark SPARK-6703
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5501.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5501
----
commit d7a06b8b773da4923f5504ee4387906a53249fa3
Author: Ilya Ganelin <il...@capitalone.com>
Date: 2015-04-13T23:50:41Z
Updated SparkConf class to add getOrCreate method. Started test suite implementation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94092324
Not sure why the above is failing, the test passes locally. Could it be a transient failure?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92992965
In this case it probably doesn't matter whether it is an atomic reference or not given the lock (I missed this earlier). However, it still makes it more explicit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92990066
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30256/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94111988
It's OK, it's a streaming test. I'm merging this to master thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94080174
Alright LGTM now. Let's wait till Jenkins is happy.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93022043
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30263/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93017775
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30267/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93051231
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30269/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5501
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94079804
[Test build #30501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30501/consoleFull) for PR 5501 at commit [`db9a963`](https://github.com/apache/spark/commit/db9a9631415d86ee36add876770fbde510530ead).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core][WIP] Provide a way to disco...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92547808
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30207/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28355263
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -67,6 +67,23 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
}
}
+ test("Test getOrCreateContext") {
+ var sc2: SparkContext = null
+ SparkContext.clearActiveContext()
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.driver.allowMultipleContexts", "true")
--- End diff --
I don't think we need to set this here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93018258
@pwendell and @JoshRosen should take a look at this too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28353698
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
--- End diff --
SparkContext
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28355032
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
+ */
+ def getOrCreate(config: SparkConf): SparkContext = {
+ if (activeContext.get() != null) {
+ setActiveContext(new SparkContext(config),
+ config.getBoolean("spark.driver.allowMultipleContexts", false))
--- End diff --
I believe this can just be `false`, because if we create a new `SparkContext` here then it already means that there is not already an existing one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28353720
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
--- End diff --
let's be consistent and use `SparkContext` everywhere
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core][WIP] Provide a way to disco...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92541131
[Test build #30207 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30207/consoleFull) for PR 5501 at commit [`d7a06b8`](https://github.com/apache/spark/commit/d7a06b8b773da4923f5504ee4387906a53249fa3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94077334
It looks good to me --- other than that test leaking resource.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core][WIP] Provide a way to disco...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28301261
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1855,19 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ */
+ private[spark] def getOrCreate(config: SparkConf = new SparkConf()): SparkContext = {
+ activeContext match {
--- End diff --
this is not your change, but can we change activeContext to an AtomicReference so it is thread safe?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92985944
[Test build #30256 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30256/consoleFull) for PR 5501 at commit [`dfec4da`](https://github.com/apache/spark/commit/dfec4dae763e888a2e8ae267b1a2e21180c9decc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core][WIP] Provide a way to disco...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92547798
[Test build #30207 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30207/consoleFull) for PR 5501 at commit [`d7a06b8`](https://github.com/apache/spark/commit/d7a06b8b773da4923f5504ee4387906a53249fa3).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93000992
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93017819
LGTM, I believe this does what we want.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94090273
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30501/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94106790
[Test build #30506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30506/consoleFull) for PR 5501 at commit [`db9a963`](https://github.com/apache/spark/commit/db9a9631415d86ee36add876770fbde510530ead).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92990049
[Test build #30256 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30256/consoleFull) for PR 5501 at commit [`dfec4da`](https://github.com/apache/spark/commit/dfec4dae763e888a2e8ae267b1a2e21180c9decc).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93016764
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30254/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93051220
[Test build #30269 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30269/consoleFull) for PR 5501 at commit [`1dc0444`](https://github.com/apache/spark/commit/1dc044452faf11062aa6f54f01c5d29094ebc113).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `commons-math3-3.4.1.jar`
* This patch **removes the following dependencies:**
* `commons-math3-3.1.1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28630870
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -67,6 +67,24 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
}
}
+ test("Test getOrCreate") {
+ var sc2: SparkContext = null
+ SparkContext.clearActiveContext()
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+
+ sc = SparkContext.getOrCreate(conf)
+
+ assert(sc.getConf.get("spark.app.name").equals("test"))
+ sc2 = SparkContext.getOrCreate(new SparkConf().setAppName("test2").setMaster("local"))
+ assert(sc2.getConf.get("spark.app.name").equals("test"))
+ assert(sc === sc2)
+ assert(sc eq sc2)
+
+ // Try creating second context to confirm that it's still possible, if desired
+ sc2 = new SparkContext(new SparkConf().setAppName("test3").setMaster("local")
--- End diff --
this is leaking a spark context right? shouldn't we shut it down?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28355155
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -67,6 +67,23 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
}
}
+ test("Test getOrCreateContext") {
--- End diff --
That's technically not what you called the method! Can you just make this `test("getOrCreate")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94075884
@pwendell @JoshRosen @andrewor14 Does this need anything more? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93001036
[Test build #30263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30263/consoleFull) for PR 5501 at commit [`15e8dea`](https://github.com/apache/spark/commit/15e8dea2ba4dc8965fe2da8cf88714021e217d19).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28353812
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
--- End diff --
I would just leave out this line because we actually discourage users from creating multiple SparkContexts in the same JVM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core][WIP] Provide a way to disco...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28301202
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1855,19 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ */
+ private[spark] def getOrCreate(config: SparkConf = new SparkConf()): SparkContext = {
--- End diff --
this should be public, shouldn't it?
also would be better to avoid using default value. just overload the function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93023726
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30264/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93003725
[Test build #30264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30264/consoleFull) for PR 5501 at commit [`15e8dea`](https://github.com/apache/spark/commit/15e8dea2ba4dc8965fe2da8cf88714021e217d19).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93016636
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30266/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28355633
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
+ */
+ def getOrCreate(config: SparkConf): SparkContext = {
+ if (activeContext.get() != null) {
+ setActiveContext(new SparkContext(config),
--- End diff --
BTW, on top of Andrew's comments, this method is not thread-safe. If two threads call it, you may have both trying to create a new SparkContext, and one of them will end up throwing.
So you probably want to make this method synchronized.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94092045
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92986595
@rxin Does it really make sense to have this be an ```AtomicReference``` if access to ```activeContext``` is already protected by the ```SPARK_CONTEXT_CONSTRUCTOR_LOCK``` lock? Moreover, even if ```activeContext``` becomes an ```AtomicReference```, I don't think we can remove the existing constructor lock since creating a ```SparkContext``` will call the constructor. Thus, I don't necessarily see an advantage to making this an AtomicReference - please let me know if you disagree.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92998098
Reynold - does it make sense to update contextBeingConstructed at the same time? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28347135
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1855,19 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ */
+ private[spark] def getOrCreate(config: SparkConf = new SparkConf()): SparkContext = {
+ activeContext match {
--- End diff --
Reynold - if we make it an atomic reference can we get rid of the SPARK_CONTEXT_CONSTRUCTOR locks around it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28354916
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
+ */
+ def getOrCreate(config: SparkConf): SparkContext = {
+ if (activeContext.get() != null) {
--- End diff --
Totally right, got mixed up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92979921
[Test build #30254 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30254/consoleFull) for PR 5501 at commit [`733ec9f`](https://github.com/apache/spark/commit/733ec9ff69dbc4dde5b9c6809133308248d16657).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93016755
[Test build #30254 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30254/consoleFull) for PR 5501 at commit [`733ec9f`](https://github.com/apache/spark/commit/733ec9ff69dbc4dde5b9c6809133308248d16657).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch **adds the following new dependencies:**
* `commons-math3-3.4.1.jar`
* This patch **removes the following dependencies:**
* `commons-math3-3.1.1.jar`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92993118
this is the mima error
```
[error] * synthetic method org$apache$spark$SparkContext$$activeContext()scala.Option in object org.apache.spark.SparkContext has now a different result type; was: scala.Option, is now: java.util.concurrent.atomic.AtomicReference
[error] filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.SparkContext.org$apache$spark$SparkContext$$activeContext")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28350387
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1804,7 +1804,8 @@ object SparkContext extends Logging {
*
* Access to this field is guarded by SPARK_CONTEXT_CONSTRUCTOR_LOCK
*/
- private var activeContext: Option[SparkContext] = None
+ private val activeContext: AtomicReference[Option[SparkContext]] =
--- End diff --
you don't need the Option wrapper anymore for atomic reference. just set it to null when it is empty.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93022017
[Test build #30263 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30263/consoleFull) for PR 5501 at commit [`15e8dea`](https://github.com/apache/spark/commit/15e8dea2ba4dc8965fe2da8cf88714021e217d19).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93014367
[Test build #30269 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30269/consoleFull) for PR 5501 at commit [`1dc0444`](https://github.com/apache/spark/commit/1dc044452faf11062aa6f54f01c5d29094ebc113).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28355471
--- Diff: core/src/test/scala/org/apache/spark/SparkContextSuite.scala ---
@@ -67,6 +67,23 @@ class SparkContextSuite extends FunSuite with LocalSparkContext {
}
}
+ test("Test getOrCreateContext") {
+ var sc2: SparkContext = null
+ SparkContext.clearActiveContext()
+ val conf = new SparkConf().setAppName("test").setMaster("local")
+ .set("spark.driver.allowMultipleContexts", "true")
+
+ sc = SparkContext.getOrCreate(conf)
+
+ assert(sc.getConf.get("spark.app.name").equals("test"))
+ sc2 = SparkContext.getOrCreate(new SparkConf().setAppName("test2").setMaster("local"))
+ assert(sc2.getConf.get("spark.app.name").equals("test"))
--- End diff --
can you add an equality check here to be more explicit?
```
assert(sc === sc2)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28356144
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
+ */
+ def getOrCreate(config: SparkConf): SparkContext = {
+ if (activeContext.get() != null) {
+ setActiveContext(new SparkContext(config),
--- End diff --
Good point - I was looking to use the setActiveContext lock but as you rightly point out, this will trigger an exception if both attempt to create.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94093004
[Test build #30506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30506/consoleFull) for PR 5501 at commit [`db9a963`](https://github.com/apache/spark/commit/db9a9631415d86ee36add876770fbde510530ead).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-93023709
[Test build #30264 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30264/consoleFull) for PR 5501 at commit [`15e8dea`](https://github.com/apache/spark/commit/15e8dea2ba4dc8965fe2da8cf88714021e217d19).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by ilganeli <gi...@git.apache.org>.
Github user ilganeli commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-92992147
Any clues as to what MiMa exclude is necessary? I don't see an explicit failure in the log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94090243
[Test build #30501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30501/consoleFull) for PR 5501 at commit [`db9a963`](https://github.com/apache/spark/commit/db9a9631415d86ee36add876770fbde510530ead).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5501#issuecomment-94106794
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30506/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28354727
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
+ */
+ def getOrCreate(config: SparkConf): SparkContext = {
+ if (activeContext.get() != null) {
--- End diff --
is this correct? Shouldn't we create a `SparkContext` when there is not already an active one (i.e. when `activeContext.get() == null`)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6703][Core] Provide a way to discover e...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/5501#discussion_r28354450
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1855,6 +1857,28 @@ object SparkContext extends Logging {
}
/**
+ * Because we can only have one active Spark Context per JVM and there are times when multiple
+ * applications may wish to share a SparkContext, this function may be used to get or instantiate
+ * a SparkContext and register it as a singleton object.
+ *
+ * Note: This function cannot be used to create multiple spark contexts even if multiple contexts
+ * are allowed. Multiple contexts will still need to be created with explicit calls to the
+ * SparkContext constructor.
+ */
--- End diff --
@ilganeli can you make this more like a java doc, where the first sentence describes what this function does? Same in the other `getOrCreate`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org