You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nkronenfeld <gi...@git.apache.org> on 2014/12/15 08:13:29 UTC
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
GitHub user nkronenfeld opened a pull request:
https://github.com/apache/spark/pull/3699
[SPARK-4848] Stand-alone cluster: Allow differences between workers with multiple instances
I've changed the stand-alone cluster run scripts to allow different workers to have different numbers of instances, and base webui ports.
I did this by moving the loop over instances from start-slaves to start-slave.
In order to stop things properly, I had to make similar changes in stop-slaves (and introduce stop-slave).
While I was at it, I changed SPARK_WORKER_PORT to work the same way as SPARK_WORKER_UI_PORT, since the new methods works fine for both.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nkronenfeld/spark-1 startup-scripts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3699.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3699
----
commit 479c31c9d3e580879d76146e2a687b5235c87b33
Author: Nathan Kronenfeld <nk...@oculusinfo.com>
Date: 2014-12-15T06:58:08Z
Move looping through instances from the master to the workers, so that each worker respects its own number of instances and web-ui port.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-66961638
[Test build #24453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24453/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79376311
jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by shaneknapp <gi...@git.apache.org>.
Github user shaneknapp commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79274011
jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79383124
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28590/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by nkronenfeld <gi...@git.apache.org>.
Github user nkronenfeld closed the pull request at:
https://github.com/apache/spark/pull/3699
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-66957276
[Test build #24453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24453/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-67007651
[Test build #24460 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24460/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79381251
[Test build #28590 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28590/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-66961646
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24453/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79383088
[Test build #28590 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28590/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-67007668
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24460/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79373195
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28578/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-66998218
[Test build #24460 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/24460/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by nkronenfeld <gi...@git.apache.org>.
Github user nkronenfeld commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-85112967
I'm redoing this in the latest code, remaking the PR from scratch, to alleviate merge issues. I'll post the new PR here when it's made.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by nkronenfeld <gi...@git.apache.org>.
Github user nkronenfeld commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-66957137
I'm not sure how mesos and yarn clusters are started/stopped (nor do I have such clusters on which to test), so I'm not sure how this will affect them. I think the way I did this should be safe - it's mostly just moving code around - but I could use a knowledgeable set of eyes to be sure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79166806
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28573/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by nkronenfeld <gi...@git.apache.org>.
Github user nkronenfeld commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-66998011
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-4848] Stand-alone cluster: Allow differ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3699#issuecomment-79277030
[Test build #28578 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28578/consoleFull) for PR 3699 at commit [`479c31c`](https://github.com/apache/spark/commit/479c31c9d3e580879d76146e2a687b5235c87b33).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org