You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sarutak <gi...@git.apache.org> on 2014/09/18 11:16:26 UTC
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/2444
[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark slaves-scripts-modification
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2444.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2444
----
commit 297e75d1b10c1b9979b943ad822386b50bda6299
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2014-09-18T09:07:39Z
Modified sbin/slaves not to export HOSTLIST
Modified sbin/slaves to choose localhost as a default host list
Renamed conf/slaves to conf/slaves.template
Added entries about slaves and slaves.template to .rat-excludes
Added entries about slaves to .gitignore
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r18016579
--- Diff: sbin/slaves.sh ---
@@ -67,23 +69,34 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
fi
fi
+
+
# By default disable strict host key checking
if [ "$SPARK_SSH_OPTS" = "" ]; then
SPARK_SSH_OPTS="-o StrictHostKeyChecking=no"
fi
-for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
- ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
- 2>&1 | sed "s/^/$slave: /" &
- if [ "$SPARK_SLAVE_SLEEP" != "" ]; then
- sleep $SPARK_SLAVE_SLEEP
- fi
+for slave in `echo "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
+ if [ "${SPARK_SSH_FOREGROUND}" = "y" ] || [ "${SPARK_SSH_FOREGROUND}" = "yes" ]; then
--- End diff --
Typically for these types of options we just check whether it's defined or not. For example elsewhere we do:
```
if [ -n "$SPARK_PRINT_LAUNCH_COMMAND" ]; then
```
Can you make it consistent with that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by jameszhouyi <gi...@git.apache.org>.
Github user jameszhouyi commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-57071944
Hi @pwendell ,
After this commit, for spark-perf will complain 'not found slaves' when run ./bin/run... so have to modify from slaves.template to slaves manually ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17823625
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
--- End diff --
O.K, I'll add SSH_FOREGROUND variable and add description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56801957
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20799/consoleFull) for PR 2444 at commit [`7858225`](https://github.com/apache/spark/commit/7858225029bae9991a5d3e861f58fa0bd529e3a3).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56331158
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20646/consoleFull) for PR 2444 at commit [`e570431`](https://github.com/apache/spark/commit/e5704313997efff4a9df28ef46f1872cd1fb36d5).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17743713
--- Diff: .gitignore ---
@@ -19,6 +19,7 @@ conf/*.sh
conf/*.properties
conf/*.conf
conf/*.xml
+conf/slaves
--- End diff --
For this one I think we intentionally have a simple slaves file in the repo so that people can start a local cluster by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56330484
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20645/consoleFull) for PR 2444 at commit [`7120a0c`](https://github.com/apache/spark/commit/7120a0cba77a6a832f674aba20dda7d0cc98c541).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56807831
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20799/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56212658
Made some comments. We need to guard this with a config parameter because otherwise it will regress behavior on large clusters where serial vs parallel ssh makes a big difference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17799143
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
--- End diff --
We should change the docs in `spark-standalone.md` to explain two new features:
1. You can set SSH_FOREGROUND if you cannot use paswordless SSH (currently, it says this is required).
2. If there is no `slaves` file in existence, it will launch a single slave at `localhost` by default.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by mattf <gi...@git.apache.org>.
Github user mattf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17815117
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
--- End diff --
yes, i was moving too quickly this morning. definitely need something to allow for background ssh.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-57699224
I also noticed that `make-distribution.sh` fails when trying to copy the now-nonexistent `conf/slaves` file. I'm going to push a hotfix commit to fix that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17798769
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
fi
fi
+
+
# By default disable strict host key checking
if [ "$SPARK_SSH_OPTS" = "" ]; then
SPARK_SSH_OPTS="-o StrictHostKeyChecking=no"
fi
-for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
+for slave in `echo "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
- 2>&1 | sed "s/^/$slave: /" &
+ 2>&1 | sed "s/^/$slave: /"
--- End diff --
I agree with matt - this will regress behavior for other users. Can we have a flag called `SSH_FOREGROUND` that turns this on?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56020878
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20534/consoleFull) for PR 2444 at commit [`297e75d`](https://github.com/apache/spark/commit/297e75d1b10c1b9979b943ad822386b50bda6299).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56331729
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20643/consoleFull) for PR 2444 at commit [`1bba8a9`](https://github.com/apache/spark/commit/1bba8a959ea9ecf21eb0783d3ce302065bf1a154).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56014482
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20534/consoleFull) for PR 2444 at commit [`297e75d`](https://github.com/apache/spark/commit/297e75d1b10c1b9979b943ad822386b50bda6299).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56778519
This looks good, just had a minor comment, then I think it's ready to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56820515
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20805/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-57699480
Actually, nevermind: this was fixed in #2549.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17823617
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
fi
fi
+
+
# By default disable strict host key checking
if [ "$SPARK_SSH_OPTS" = "" ]; then
SPARK_SSH_OPTS="-o StrictHostKeyChecking=no"
fi
-for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
+for slave in `echo "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
- 2>&1 | sed "s/^/$slave: /" &
+ 2>&1 | sed "s/^/$slave: /"
--- End diff --
O.K, I'll try it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17769855
--- Diff: .gitignore ---
@@ -19,6 +19,7 @@ conf/*.sh
conf/*.properties
conf/*.conf
conf/*.xml
+conf/slaves
--- End diff --
So, this file will not to be edited? User should use another slave list file by SPARK_SLAVES variable right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56807825
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20799/consoleFull) for PR 2444 at commit [`7858225`](https://github.com/apache/spark/commit/7858225029bae9991a5d3e861f58fa0bd529e3a3).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17799162
--- Diff: .gitignore ---
@@ -19,6 +19,7 @@ conf/*.sh
conf/*.properties
conf/*.conf
conf/*.xml
+conf/slaves
--- End diff --
Okay this is fine actually, given that we preserve the "deafult" behavior due to your edits below (of starting at localhost).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by mattf <gi...@git.apache.org>.
Github user mattf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17730829
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
fi
fi
+
+
# By default disable strict host key checking
if [ "$SPARK_SSH_OPTS" = "" ]; then
SPARK_SSH_OPTS="-o StrictHostKeyChecking=no"
fi
-for slave in `cat "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
+for slave in `echo "$HOSTLIST"|sed "s/#.*$//;/^$/d"`; do
ssh $SPARK_SSH_OPTS "$slave" $"${@// /\\ }" \
- 2>&1 | sed "s/^/$slave: /" &
+ 2>&1 | sed "s/^/$slave: /"
--- End diff --
doing this in parallel is a feature, and passwordless access is common. i'd suggest adding a flag to indicate ssh should be run in the foreground.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56812364
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20805/consoleFull) for PR 2444 at commit [`eff7394`](https://github.com/apache/spark/commit/eff73948d44a1067498033605ce9158becacedf6).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2444
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56902257
Thanks @sarutak and @mattf for the review. I'll pull this in.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by mattf <gi...@git.apache.org>.
Github user mattf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17782052
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
--- End diff --
thanks for pointing that out. i didn't read closely enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56801729
Thanks @pwendell , I've modified what you mentioned.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by mattf <gi...@git.apache.org>.
Github user mattf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r18027788
--- Diff: docs/spark-standalone.md ---
@@ -62,7 +62,12 @@ Finally, the following configuration options can be passed to the master and wor
# Cluster Launch Scripts
-To launch a Spark standalone cluster with the launch scripts, you need to create a file called `conf/slaves` in your Spark directory, which should contain the hostnames of all the machines where you would like to start Spark workers, one per line. The master machine must be able to access each of the slave machines via password-less `ssh` (using a private key). For testing, you can just put `localhost` in this file.
+To launch a Spark standalone cluster with the launch scripts, you need to create a file called `conf/slaves` in your Spark directory,
+which should contain the hostnames of all the machines where you would like to start Spark workers, one per line. If `conf/slaves`
+does not exist, the launch scripts use a list which contains single hostname `localhost`. This can be used for testing.
+The master machine must be able to access each of the slave machines via `ssh`. By default, `ssh` is executed in the background for parallel execution for each slave machine.
+If you would like to use password authentication instead of password-less(using a private key) for `ssh`, `ssh` does not work well in the background.
+To avoid this, you can set a environment variable `SPARK_SSH_FOREGROUND` to something like `yes` or `y` to execute `ssh` in the foreground.
--- End diff --
what about -
To launch a Spark standalone cluster with the launch scripts, you should create a file called `conf/slaves` in your Spark directory, which must contain the hostnames of all the machines where you intend to start Spark workers, one per line. If `conf/slaves` does not exist, the launch scripts defaults to a single machine (`localhost`), which is useful for testing. Note, the master machine accesses each of the worker machines via `ssh`. By default, `ssh` is run in parallel and requires password-less (using a private key) access to be setup. If you do not have a password-less setup, you can set the environment variable `SPARK_SSH_FOREGROUND` and serially provide a password for each worker.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56328753
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20643/consoleFull) for PR 2444 at commit [`1bba8a9`](https://github.com/apache/spark/commit/1bba8a959ea9ecf21eb0783d3ce302065bf1a154).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r18028634
--- Diff: docs/spark-standalone.md ---
@@ -62,7 +62,12 @@ Finally, the following configuration options can be passed to the master and wor
# Cluster Launch Scripts
-To launch a Spark standalone cluster with the launch scripts, you need to create a file called `conf/slaves` in your Spark directory, which should contain the hostnames of all the machines where you would like to start Spark workers, one per line. The master machine must be able to access each of the slave machines via password-less `ssh` (using a private key). For testing, you can just put `localhost` in this file.
+To launch a Spark standalone cluster with the launch scripts, you need to create a file called `conf/slaves` in your Spark directory,
+which should contain the hostnames of all the machines where you would like to start Spark workers, one per line. If `conf/slaves`
+does not exist, the launch scripts use a list which contains single hostname `localhost`. This can be used for testing.
+The master machine must be able to access each of the slave machines via `ssh`. By default, `ssh` is executed in the background for parallel execution for each slave machine.
+If you would like to use password authentication instead of password-less(using a private key) for `ssh`, `ssh` does not work well in the background.
+To avoid this, you can set a environment variable `SPARK_SSH_FOREGROUND` to something like `yes` or `y` to execute `ssh` in the foreground.
--- End diff --
@mattf Thank you for reviewing. It makes sense.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by mattf <gi...@git.apache.org>.
Github user mattf commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56172067
+1 lgtm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by mattf <gi...@git.apache.org>.
Github user mattf commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17730762
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
--- End diff --
why cat here and echo later?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56820508
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20805/consoleFull) for PR 2444 at commit [`eff7394`](https://github.com/apache/spark/commit/eff73948d44a1067498033605ce9158becacedf6).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56335094
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20646/consoleFull) for PR 2444 at commit [`e570431`](https://github.com/apache/spark/commit/e5704313997efff4a9df28ef46f1872cd1fb36d5).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2444#issuecomment-56334081
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20645/consoleFull) for PR 2444 at commit [`7120a0c`](https://github.com/apache/spark/commit/7120a0cba77a6a832f674aba20dda7d0cc98c541).
* This patch **passes** unit tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3584] sbin/slaves doesn't work when we ...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/2444#discussion_r17769822
--- Diff: sbin/slaves.sh ---
@@ -67,20 +69,26 @@ fi
if [ "$HOSTLIST" = "" ]; then
if [ "$SPARK_SLAVES" = "" ]; then
- export HOSTLIST="${SPARK_CONF_DIR}/slaves"
+ if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+ HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+ else
+ HOSTLIST=localhost
+ fi
else
- export HOSTLIST="${SPARK_SLAVES}"
+ HOSTLIST=`cat "${SPARK_SLAVES}"`
--- End diff --
This is to use HOSTLIST as List of Host, not file.
It's to use localhost as a default host list entry.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org