You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jimlohse <gi...@git.apache.org> on 2016/01/24 20:08:09 UTC
[GitHub] spark pull request: Provide same info as in spark-submit --help
GitHub user jimlohse opened a pull request:
https://github.com/apache/spark/pull/10890
Provide same info as in spark-submit --help
this is stated for --packages and --repositories. Without stating it for --jars, people expect a standard java classpath to work, with expansion and using a different delimiter than a comma. Currently this is only state in the --help for spark-submit "Comma-separated list of local jars to include on the driver and executor classpaths."
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jimlohse/spark patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10890.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10890
----
commit 9a832da21917091ab4e7b7e8de1fd049f84fd9d0
Author: James Lohse <ji...@users.noreply.github.com>
Date: 2016-01-24T19:07:53Z
Provide same info as in spark-submit --help
this is stated for --packages and --repositories. Without stating it for --jars, people expect a standard java classpath to work, with expansion and using a different delimiter than a comma. Currently this is only state in the --help for spark-submit "Comma-separated list of local jars to include on the driver and executor classpaths."
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-176115335
**[Test build #50271 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50271/consoleFull)** for PR 10890 at commit [`73bf7e9`](https://github.com/apache/spark/commit/73bf7e965a4f60be282f1a6e97c62114f28846d3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-174331990
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by jimlohse <gi...@git.apache.org>.
Github user jimlohse commented on a diff in the pull request:
https://github.com/apache/spark/pull/10890#discussion_r50653981
--- Diff: docs/submitting-applications.md ---
@@ -177,8 +177,9 @@ debugging information by running `spark-submit` with the `--verbose` option.
# Advanced Dependency Management
When using `spark-submit`, the application jar along with any jars included with the `--jars` option
-will be automatically transferred to the cluster. Spark uses the following URL scheme to allow
-different strategies for disseminating jars:
+will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. Each entry points to a specific jar file, resulting in a comma-separated list of local jars. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars.
--- End diff --
Oh I didn't catch the first point of your question til now, I think they
are URLS because I think they need the file:// before them, that's an
URL too?
On 01/24/2016 12:24 PM, Sean Owen wrote:
>
> In docs/submitting-applications.md
> <https://github.com/apache/spark/pull/10890#discussion_r50640513>:
>
> > @@ -177,8 +177,9 @@ debugging information by running `spark-submit` with the `--verbose` option.
> >
> > # Advanced Dependency Management
> > When using `spark-submit`, the application jar along with any jars included with the `--jars` option
> > -will be automatically transferred to the cluster. Spark uses the following URL scheme to allow
> > -different strategies for disseminating jars:
> > +will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. Each entry points to a specific jar file, resulting in a comma-separated list of local jars. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars.
>
> This seems OK but do they have to be local JARs (I think so)? in which
> case are they really URLs? The second sentence you added seems to say
> the same thing as the first then. You could back-tick |--jars| too for
> consistency.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/10890/files#r50640513>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by jimlohse <gi...@git.apache.org>.
GitHub user jimlohse reopened a pull request:
https://github.com/apache/spark/pull/10890
Provide same info as in spark-submit --help
this is stated for --packages and --repositories. Without stating it for --jars, people expect a standard java classpath to work, with expansion and using a different delimiter than a comma. Currently this is only state in the --help for spark-submit "Comma-separated list of local jars to include on the driver and executor classpaths."
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jimlohse/spark patch-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10890.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10890
----
commit 9a832da21917091ab4e7b7e8de1fd049f84fd9d0
Author: James Lohse <ji...@users.noreply.github.com>
Date: 2016-01-24T19:07:53Z
Provide same info as in spark-submit --help
this is stated for --packages and --repositories. Without stating it for --jars, people expect a standard java classpath to work, with expansion and using a different delimiter than a comma. Currently this is only state in the --help for spark-submit "Comma-separated list of local jars to include on the driver and executor classpaths."
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-176115460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50271/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-176109173
**[Test build #50271 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50271/consoleFull)** for PR 10890 at commit [`73bf7e9`](https://github.com/apache/spark/commit/73bf7e965a4f60be282f1a6e97c62114f28846d3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-175508971
Jenkins, test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-175517012
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50186/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-176104639
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-175509029
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10890#discussion_r50640513
--- Diff: docs/submitting-applications.md ---
@@ -177,8 +177,9 @@ debugging information by running `spark-submit` with the `--verbose` option.
# Advanced Dependency Management
When using `spark-submit`, the application jar along with any jars included with the `--jars` option
-will be automatically transferred to the cluster. Spark uses the following URL scheme to allow
-different strategies for disseminating jars:
+will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. Each entry points to a specific jar file, resulting in a comma-separated list of local jars. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars.
--- End diff --
This seems OK but do they have to be local JARs (I think so)? in which case are they really URLs? The second sentence you added seems to say the same thing as the first then. You could back-tick `--jars` too for consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-175517006
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-176115457
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by jimlohse <gi...@git.apache.org>.
Github user jimlohse commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-174331816
I don't know if I have write access, about to find out, I suspect I don't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by jimlohse <gi...@git.apache.org>.
Github user jimlohse closed the pull request at:
https://github.com/apache/spark/pull/10890
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10890
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/10890#issuecomment-176121885
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Provide same info as in spark-submit --help
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/10890#discussion_r50671889
--- Diff: docs/submitting-applications.md ---
@@ -177,8 +177,9 @@ debugging information by running `spark-submit` with the `--verbose` option.
# Advanced Dependency Management
When using `spark-submit`, the application jar along with any jars included with the `--jars` option
-will be automatically transferred to the cluster. Spark uses the following URL scheme to allow
-different strategies for disseminating jars:
+will be automatically transferred to the cluster. URLs supplied after --jars must be separated by commas. Each entry points to a specific jar file, resulting in a comma-separated list of local jars. That list is included on the driver and executor classpaths. Directory expansion does not work with --jars.
--- End diff --
It was a dumb question since it's answered just below, yes. I didn't even realize this. They're really URIs not URLs but this is minor.
Actually, `spark-submit --help` refers to local JARs, but yeah they're not necessarily local? I think it'd be fine to also fix the help text on all of the spark-* scripts while we're here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org