You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/27 09:50:50 UTC
[GitHub] [spark] Ngone51 opened a new pull request #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Ngone51 opened a new pull request #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Remove automatically resource coordination support from Standalone.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Resource coordination is mainly designed for the scenario where multiple workers launched on the same host. However, it's, actually, a non-existed scenario for today's Spark. Because, Spark now can start multiple executors in a single Worker, while it only allow one executor per Worker at very beginning. So, now, it really help nothing for user to launch multiple workers on the same host. Thus, it's not worth for us to bring over complicated implementation and potential high maintain cost for such an impossible scenario.
### Does this PR introduce any user-facing change?
<!--
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If no, write 'No'.
-->
No, it's Spark 3.0 feature.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Pass Jenkins.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591880658
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591952747
**[Test build #119029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119029/testReport)** for PR 27722 at commit [`0103ace`](https://github.com/apache/spark/commit/0103ace33c1bdb7f983fb687dd28300667d79b7b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 edited a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592031322
> Its just running discovery script and assuming this owns all of them, correct?
Yea, I think this is the only difference. And it really break local-cluster mode. Though, I haven't made change in `LocalClusterSpark` to handle it and only update one related test to workaround it. And there's no more local-cluster mode tests rely on **coordination**.
And, of cause, user should still be aware of resource conflict between multi drivers. So I still keep the warning in the doc. Same node driver and worker mode, actually, is not recommended when driver submitted in client mode. Ideally, it should be an edge node out of cluster for the driver and assumed not to be on the same of host of worker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591950091
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592633532
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592397820
**[Test build #119076 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119076/testReport)** for PR 27722 at commit [`134abfc`](https://github.com/apache/spark/commit/134abfc6e5d5ed618329e4aa964ab8c004cfd589).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591950105
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23775/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592390382
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591952747
**[Test build #119029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119029/testReport)** for PR 27722 at commit [`0103ace`](https://github.com/apache/spark/commit/0103ace33c1bdb7f983fb687dd28300667d79b7b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593356099
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592389931
**[Test build #119076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119076/testReport)** for PR 27722 at commit [`134abfc`](https://github.com/apache/spark/commit/134abfc6e5d5ed618329e4aa964ab8c004cfd589).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591880682
cc @mengxr @tgravescs @jiangxb1987 @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592390392
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23820/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591940661
**[Test build #119023 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119023/testReport)** for PR 27722 at commit [`0103ace`](https://github.com/apache/spark/commit/0103ace33c1bdb7f983fb687dd28300667d79b7b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591941006
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119023/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592161154
Ok, after thinking about this some more, I'm ok with removing. its a new feature and if we are really pushing to support only one worker per node it makes sense to not add it in. I think you deal with standalone mode in production more then me and most of the cases I'm thinking of are dev/testing type cases. It can be worked around by specifying different resource files for different workers (except for local-cluster) so shouldn't be an issue with a little extra setup if people do want to run multiple workers per node.
We need to explicitly state that local-cluster mode is not supported with custom resources.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593573221
Thanks, merged to master/3.0 !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592542588
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593273792
**[Test build #119156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119156/testReport)** for PR 27722 at commit [`1b44596`](https://github.com/apache/spark/commit/1b4459671781ed3325e10900754eb2161c62cdd0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592633532
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592633564
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119094/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592633564
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119094/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
jiangxb1987 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592145041
> I know of people running multiple workers per node
@tgravescs Could you please share details about those workloads? Based on our experience it's not necessarily needed to deploy multiple workers on the same host.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591940996
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591880670
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23769/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] jiangxb1987 closed pull request #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
jiangxb1987 closed pull request #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592039488
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119029/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591880670
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23769/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592389931
**[Test build #119076 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119076/testReport)** for PR 27722 at commit [`134abfc`](https://github.com/apache/spark/commit/134abfc6e5d5ed618329e4aa964ab8c004cfd589).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592042018
> Ideally, it should be an edge node out of cluster for the driver and assumed not to be on the same of host of worker.
This may be ideal but not true in many cases, especially in just test/dev setups.
For me this comes down to making things convenient for the user and doesn't seem like much overhead on the Spark side, especially since its already implemented/tested. If you aren't using custom resources, the code has no affect (or shouldn't anyway - if it does we should fix). Its configurable off if people don't want it or see some overhead from the file locking pieces. The code for this in my opinion is not that much and pretty isolated. I don't see much downside here.
I guess the spark.resources.dir could be left around in worker crash type cases. Though again could be turned off.
The only way I see to fix the local-cluster mode is leave this code or somehow allow users to specify different resource files or script per worker, which seems like a hard/weird api.
I would like more justification here? why is this a high maintenance cost? the code is done and tested and I don't envision it needed more feature work. local-cluster mode unit tests should be enough to test it pretty thoroughly (perhaps we need more).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593722722
Thanks all!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593271459
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591949730
Jenkins, retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592397852
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592031322
> Its just running discovery script and assuming this owns all of them, correct?
Yea, I think this is the only difference. And it really break local-cluster mode. Though, I haven't made change in `LocalClusterSpark` to handle it and only update one related test to workaround it. And there's no more local-cluster mode tests rely on ***coordination**.
And, of cause, user should still be aware of resource conflict between multi drivers. So I still keep the warning in the doc. Same node driver and worker mode, actually, is not recommended when driver submitted in client mode. Ideally, it should be an edge node out of cluster for the driver and assumed not to be on the same of host of worker.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593356108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119156/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592390392
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23820/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592542594
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23837/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593356099
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592545280
**[Test build #119094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119094/testReport)** for PR 27722 at commit [`134abfc`](https://github.com/apache/spark/commit/134abfc6e5d5ed618329e4aa964ab8c004cfd589).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592390382
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591883799
**[Test build #119023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119023/testReport)** for PR 27722 at commit [`0103ace`](https://github.com/apache/spark/commit/0103ace33c1bdb7f983fb687dd28300667d79b7b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mengxr commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
mengxr commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592057231
@tgravescs I suggested removal of this resource coordination code in an offline 3.0 API audit discussion to keep Spark codebase simple. Scheduling is already an area that lacks of maintainers. Increasing its complexity would keep potential maintainers away. This is why we have been very careful when we introduced the resource-aware scheduling feature. Here are the reasons to remove this feature before Spark 3.0 release:
* In a real standalone deployment, having multiple workers on the same host is not longer needed. As @Ngone51 mentioned, the feature was introduced to keep worker JVM small to avoid long GC pause when worker and executors share the same JVM. Now executors run in a different JVM, so having one worker per host is sufficient. Correct me if I'm wrong since you mentioned several deployment using multiple workers on the same host.
* Due to the reason above, instead of supporting GPU scheduling for multiple workers on the same host, we should deprecate entirely the support of multiple workers on the same host in 3.0 and remove it in a future release.
* The local-cluster mode is not a public feature, which should only be used in Spark tests. Actually this change is the first to mention "local-cluster" in the user guide, which makes it "public". I don't think we want to add (even localized) complexity just for this mode. In a test setup, we can use one worker process and separate driver/worker scripts to simplify the resource allocation. Again, using multiple workers in local-cluster mode is to simulate a real cluster, not a "production" setup. If this is for simulation, we just need fake discovery scripts.
* For worker recovery, my understanding is that there shouldn't be a case that the old worker and the recovery worker are running at the same time. Because the recovery is usually done by monit which monitors the worker process. cc: @jiangxb1987
cc: @squito
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592016691
Can you please clarify the overall behavior changes? Its just running discovery script and assuming this owns all of them, correct?
I believe this will break local-cluster mode unless you have handled in your change? local cluster mode you can’t specify different scripts/resource files per worker so how is this now handled?
This also handled coordination between multiple drivers (if you gave drivers gpus) and workers on the same node.
I know of people running multiple workers per node - so please provide more justification for this - is it causing performance issue? It seems like this provided useful feature for so unless its causing some issue why remove.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593273792
**[Test build #119156 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119156/testReport)** for PR 27722 at commit [`1b44596`](https://github.com/apache/spark/commit/1b4459671781ed3325e10900754eb2161c62cdd0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592397861
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119076/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#discussion_r385356809
##########
File path: docs/configuration.md
##########
@@ -2782,5 +2762,4 @@ There are configurations available to request resources for the driver: <code>sp
Spark will use the configurations specified to first request containers with the corresponding resources from the cluster manager. Once it gets the container, Spark launches an Executor in that container which will discover what resources the container has and the addresses associated with each resource. The Executor will register with the Driver and report back the resources available to that Executor. The Spark scheduler can then schedule tasks to each Executor and assign specific resource addresses based on the resource requirements the user specified. The user can see the resources assigned to a task using the <code>TaskContext.get().resources</code> api. On the driver, the user can see the resources assigned with the SparkContext <code>resources</code> call. It's then up to the user to use the assignedaddresses to do the processing they want or pass those into the ML/AI framework they are using.
-See your cluster manager specific page for requirements and details on each of - [YARN](running-on-yarn.html#resource-allocation-and-configuration-overview), [Kubernetes](running-on-kubernetes.html#resource-allocation-and-configuration-overview) and [Standalone Mode](spark-standalone.html#resource-allocation-and-configuration-overview). It is currently not available with Mesos or local mode. If using local-cluster mode see the Spark Standalone documentation but be aware only a single worker resources file or discovery script can be specified the is shared by all the Workers so you should enable resource coordination (see <code>spark.resources.coordinateResourcesInStandalone</code>).
-
+See your cluster manager specific page for requirements and details on each of - [YARN](running-on-yarn.html#resource-allocation-and-configuration-overview), [Kubernetes](running-on-kubernetes.html#resource-allocation-and-configuration-overview) and [Standalone Mode](spark-standalone.html#resource-allocation-and-configuration-overview). It is currently not available with Mesos or local mode. If using local-cluster mode please see the Spark Standalone documentation.
Review comment:
we should modify to say local-cluster mode not supported - I guess if we want to be very specific we could state say with multiple workers (but I'm fine just saying not supported in general).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593272606
@tgravescs thanks for the review. I manually tested with resources file and script locally and checked in UI. It looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592039476
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592397852
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mengxr edited a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
mengxr edited a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592057231
@tgravescs I suggested removal of this resource coordination code in an offline 3.0 API audit discussion to keep Spark codebase simple. Scheduling is already an area that lacks of maintainers. Increasing its complexity would keep potential maintainers away. This is why we have been very careful when we introduced the resource-aware scheduling feature. Here are the reasons to remove this feature before Spark 3.0 release:
* In a real standalone deployment, having multiple workers on the same host is not longer needed. As @Ngone51 mentioned, the feature was introduced to keep worker JVM small to avoid long GC pause when worker and executors share the same JVM. Now executors run in a different JVM, so having one worker per host is sufficient. Correct me if I'm wrong since you mentioned several deployment using multiple workers on the same host.
* Due to the reason above, instead of supporting GPU scheduling for multiple workers on the same host, we should deprecate entirely the support of multiple workers on the same host in 3.0 and remove it in a future release, to further simplify the codebase.
* The local-cluster mode is not a public feature, which should only be used in Spark tests. Actually this change is the first to mention "local-cluster" in the user guide, which makes it "public". I don't think we want to add (even localized) complexity just for this mode. In a test setup, we can use one worker process and separate driver/worker scripts to simplify the resource allocation. Again, using multiple workers in local-cluster mode is to simulate a real cluster, not a "production" setup. If this is for simulation, we just need fake discovery scripts.
* For worker recovery, my understanding is that there shouldn't be a case that the old worker and the recovery worker are running at the same time. Because the recovery is usually done by monit which monitors the worker process. cc: @jiangxb1987
cc: @squito
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs edited a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs edited a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592042018
> Ideally, it should be an edge node out of cluster for the driver and assumed not to be on the same of host of worker.
This may be ideal but not true in many cases, especially in just test/dev setups, but its no different from yarn case with multiple drivers so can see this being ok.
For me this comes down to making things convenient for the user and doesn't seem like much overhead on the Spark side, especially since its already implemented/tested. If you aren't using custom resources, the code has no affect (or shouldn't anyway - if it does we should fix). Its configurable off if people don't want it or see some overhead from the file locking pieces. The code for this in my opinion is not that much and pretty isolated. I don't see much downside here.
I guess the spark.resources.dir could be left around in worker crash type cases. Though again could be turned off.
The only way I see to fix the local-cluster mode is leave this code or somehow allow users to specify different resource files or script per worker, which seems like a hard/weird api.
I would like more justification here? why is this a high maintenance cost? the code is done and tested and I don't envision it needed more feature work. local-cluster mode unit tests should be enough to test it pretty thoroughly (perhaps we need more).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592397861
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119076/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591950105
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23775/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591883799
**[Test build #119023 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119023/testReport)** for PR 27722 at commit [`0103ace`](https://github.com/apache/spark/commit/0103ace33c1bdb7f983fb687dd28300667d79b7b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592541996
test this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592039488
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119029/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592038347
**[Test build #119029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119029/testReport)** for PR 27722 at commit [`0103ace`](https://github.com/apache/spark/commit/0103ace33c1bdb7f983fb687dd28300667d79b7b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
Ngone51 edited a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592031322
> Its just running discovery script and assuming this owns all of them, correct?
Yea, I think this is the only difference. And it really break local-cluster mode. Though, I haven't made change in `LocalClusterSpark` to handle it and only update one related test to workaround it. And there's no more local-cluster mode tests rely on **coordination**.
And, of cause, user should still be aware of resource conflict between multi drivers. So I still keep the warning in the doc. Same node driver and worker mode, actually, is not recommended when driver submitted in client mode. Ideally, it should be an edge node out of cluster for the driver and assumed not to be on the same of host of worker.
I think the main concern to remove is this may introduce potential high maintain cost due to its complicate implementation compares to its low benefit we could have.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592542588
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592071235
I'm not very familiar with the whole story, but the recent discussion of backward compatibility makes me worry about this one.
If we release a very complicated API for 1% of the user cases, it may not worth the cost to maintain that API. Already implemented/tested doesn't count the future maintenance cost. Recently I'm suffered from keeping the support of some legacy behavior/API and I hope we can avoid such problems at the beginning.
Are there any valid use cases for multi-worker per host? If not or only a little, I'd suggest we at least don't support it in the new GPU scheduling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592542594
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23837/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593355123
**[Test build #119156 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119156/testReport)** for PR 27722 at commit [`1b44596`](https://github.com/apache/spark/commit/1b4459671781ed3325e10900754eb2161c62cdd0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592545280
**[Test build #119094 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119094/testReport)** for PR 27722 at commit [`134abfc`](https://github.com/apache/spark/commit/134abfc6e5d5ed618329e4aa964ab8c004cfd589).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27722: [SPARK-30969][CORE]
Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592629692
**[Test build #119094 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/119094/testReport)** for PR 27722 at commit [`134abfc`](https://github.com/apache/spark/commit/134abfc6e5d5ed618329e4aa964ab8c004cfd589).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593271459
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593356108
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119156/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591941006
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119023/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-592039476
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593271462
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23898/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591880658
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-593271462
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/23898/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591950091
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tgravescs commented on a change in pull request #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#discussion_r385815769
##########
File path: core/src/main/scala/org/apache/spark/deploy/StandaloneResourceUtils.scala
##########
@@ -17,30 +17,21 @@
package org.apache.spark.deploy
-import java.io.{File, RandomAccessFile}
-import java.nio.channels.{FileLock, OverlappingFileLockException}
+import java.io.File
import java.nio.file.Files
import scala.collection.mutable
-import scala.util.Random
import scala.util.control.NonFatal
import org.json4s.{DefaultFormats, Extraction}
-import org.json4s.jackson.JsonMethods.{compact, parse, render}
+import org.json4s.jackson.JsonMethods.{compact, render}
import org.apache.spark.{SparkConf, SparkException}
Review comment:
SparkConf no longer needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27722:
[SPARK-30969][CORE] Remove resource coordination support from Standalone
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27722: [SPARK-30969][CORE] Remove resource coordination support from Standalone
URL: https://github.com/apache/spark/pull/27722#issuecomment-591940996
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org