You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by YanTangZhai <gi...@git.apache.org> on 2014/08/20 16:13:06 UTC
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
GitHub user YanTangZhai opened a pull request:
https://github.com/apache/spark/pull/2059
[SPARK-3148] Update global variables of HttpBroadcast so that multiple SparkContexts can coexist
Update global variables of HttpBroadcast so that multiple SparkContexts can coexist
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/YanTangZhai/spark SPARK-3148
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2059.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2059
----
commit cdef539abc5d2d42d4661373939bdd52ca8ee8e6
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-06T13:07:08Z
Merge pull request #1 from apache/master
update
commit cbcba66ad77b96720e58f9d893e87ae5f13b2a95
Author: YanTangZhai <ha...@tencent.com>
Date: 2014-08-20T13:14:08Z
Merge pull request #3 from apache/master
Update
commit e751ebd22c0683746b8e13c48570bf22b4de45db
Author: yantangzhai <ty...@163.com>
Date: 2014-08-20T14:07:57Z
[SPARK-3148] Update global variables of HttpBroadcast so that multiple SparkContexts can coexist
commit 97b34079b4af178ff2bca42c314aeb0e51687167
Author: yantangzhai <ty...@163.com>
Date: 2014-08-20T14:11:34Z
[SPARK-3148] Update global variables of HttpBroadcast so that multiple SparkContexts can coexist
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by YanTangZhai <gi...@git.apache.org>.
Github user YanTangZhai commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-52884506
Hi @JoshRosen SparkContext1 creates broadcastManager and initializes HttpBroadcast object. HttpBroadcast creates httpserver and broadcastDir and so on. However SparkContext2 in the same process won't initialize HttpBroadcast object when creating broadcastManager. Since HttpBroadcast object is marked initialized and will not be initialized any more. SparkContext1 and SparkContext2 will share the same HttpBroadcast object. When SparkContext1 stops HttpBroadcast, HttpBroadcast in SparkContext2 actually is stopped. When HttpBroadcast1 cleans up files, some files owned by SparkContext2 may be removed. Since they are the same one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-52844784
Could you provide some more background in order to help us review this PR? Did older Spark versions support multiple SparkContexts? Is this fixing a regression from an earlier release?
Please add this information to the pull request description so that it's incorporated in the final commit message if we merge this. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-52783760
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18958/consoleFull) for PR 2059 at commit [`97b3407`](https://github.com/apache/spark/commit/97b34079b4af178ff2bca42c314aeb0e51687167).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2059
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-54695001
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19869/consoleFull) for PR 2059 at commit [`a5987a5`](https://github.com/apache/spark/commit/a5987a5633b9c000dbc0d95eb013b7cb7044aa37).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-52785971
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18961/consoleFull) for PR 2059 at commit [`a5987a5`](https://github.com/apache/spark/commit/a5987a5633b9c000dbc0d95eb013b7cb7044aa37).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-52783775
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18958/consoleFull) for PR 2059 at commit [`97b3407`](https://github.com/apache/spark/commit/97b34079b4af178ff2bca42c314aeb0e51687167).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-67579613
I think that we should close this issue for now, since there are other blockers to multiple SparkContexts in the same JVM (namely, the global SparkEnv). We'll consider this fix when addressing the larger "multiple SparkContexts issue", though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-52785960
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18961/consoleFull) for PR 2059 at commit [`a5987a5`](https://github.com/apache/spark/commit/a5987a5633b9c000dbc0d95eb013b7cb7044aa37).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3148] Update global variables of HttpBr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2059#issuecomment-54695048
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/19869/consoleFull) for PR 2059 at commit [`a5987a5`](https://github.com/apache/spark/commit/a5987a5633b9c000dbc0d95eb013b7cb7044aa37).
* This patch **fails** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org