You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pwendell <gi...@git.apache.org> on 2014/05/01 07:26:27 UTC
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
GitHub user pwendell opened a pull request:
https://github.com/apache/spark/pull/609
SPARK-1691: Support quoted arguments inside of spark-submit.
This is a fairly straightforward fix. The bug was reported by @vanzin and the fix was proposed by @deanwampler and myself. Please take a look!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwendell/spark quotes
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/609.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #609
----
commit 8bed76793f2a15dc924e67c3ef8150f1e26998ad
Author: Patrick Wendell <pw...@gmail.com>
Date: 2014-05-01T00:47:11Z
SPARK-1691: Support quoted arguments inside of spark-submit.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by koertkuipers <gi...@git.apache.org>.
Github user koertkuipers commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-49559153
on the command line i can get this to work now, but its still way beyond my bash skills to use "exec spark-submit" inside a script with multiple java options. this is a fairly normal use case i imagine if you want to launch an application using spark-submit.
for example my attempt to launch SparkPi with multiple java options is below. when i run it i get:
$ ./pi
spark-submit --class org.apache.spark.examples.SparkPi --driver-java-options "-Dfoo=a -Dbar=b" /usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
Error: Could not find or load main class "-Dfoo=a
i have no doubt its me that is doing something wrong here, but it shouldnt be this hard to use... perhaps a different syntax would make life easier?
best, koert
$ more pi
#!/bin/bash -e
DRIVER_JAVA_OPTS="-Dfoo=a -Dbar=b"
SPARK_SUBMIT="spark-submit"
OPTS=""
OPTS+=" --class org.apache.spark.examples.SparkPi"
OPTS+=" --driver-java-options \"${DRIVER_JAVA_OPTS}\""
echo $SPARK_SUBMIT $OPTS /usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
exec $SPARK_SUBMIT $OPTS /usr/lib/spark/examples/lib/spark-examples_2.10-1.0.0-cdh5.1.0.jar
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-41882158
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-41883608
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark issue #609: SPARK-1691: Support quoted arguments inside of spark-submi...
Posted by koertkuipers <gi...@git.apache.org>.
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/609
```OPTS+=" --driver-java-options \"-Da=b -Dx=y\""```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #609: SPARK-1691: Support quoted arguments inside of spark-submi...
Posted by koertkuipers <gi...@git.apache.org>.
Github user koertkuipers commented on the issue:
https://github.com/apache/spark/pull/609
@ganeshm25 it seems to work in newer spark versions. i havent tried in spark 1.4.2. however its still very tricky to get it right and i would prefer a simpler solution.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-41883609
All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14603/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by cosmicangler <gi...@git.apache.org>.
Github user cosmicangler commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47289248
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-41889925
Since this was already vetted by a few folks on the mailing list I'm gonna go ahead an merge it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47287791
Being the person who reported the original bug :-), the issue for me was not quotes inside --driver-java-options, but providing several arguments (e.g. --driver-java-options "-Dfoo -Dbar" would not work, as the example in the jira issue shows). So maybe the commit description is a little misleading.
That issue is definitely fixed, so it sounds like your problem is different (e.g. sounds like you want quotes to be propagated to the driver but that's not happening).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by cosmicangler <gi...@git.apache.org>.
Github user cosmicangler commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47288923
That is correct. That is precisely what is not working for me int he 1.0 distribution. I am looking to do a --driver-java-options "-XX:+UseParallelOldGC -XX:parallelGCThreads=8"
I get -XX:parallelGCThreads=8 is not a class error as it doesn't appear to be respecting the " ". Maybe there is an issue the version our ops folks used in our clusters. Was this fixed in 1.0 master?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47289177
That is working fine for me. Looks like an issue with your deployment. The fix is both in branch-1.0 and master (there's no such thing as "1.0 master").
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by cosmicangler <gi...@git.apache.org>.
Github user cosmicangler commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47286611
Is this fix in 1.0 as I am still getting this error when submitting with yarn?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47287123
@cosmicangler yes:
https://github.com/apache/spark/commit/dd601bf92fe23ccc181ce0cbafe9412aa1fbb6de
If you're running into some issue with submitting jobs, probably better to ask on the list or file a bug.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by cosmicangler <gi...@git.apache.org>.
Github user cosmicangler commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-47287482
Thank you for hte I can file a bug but my problem is this bug that marked fixed but not corrected in the 1.0 distribution I am using. I am unable to use quoted arguments for --driver-java-options as discussed in this bug.
Should a submit another identical bug to this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/609
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark issue #609: SPARK-1691: Support quoted arguments inside of spark-submi...
Posted by ganeshm25 <gi...@git.apache.org>.
Github user ganeshm25 commented on the issue:
https://github.com/apache/spark/pull/609
@koertkuipers i am trying to do achieve running the multiple driver-java-options with Spark 1.4.2 inside a bash script? is there a solution you found for this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-1691: Support quoted arguments inside of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/609#issuecomment-41882150
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark issue #609: SPARK-1691: Support quoted arguments inside of spark-submi...
Posted by ganeshm25 <gi...@git.apache.org>.
Github user ganeshm25 commented on the issue:
https://github.com/apache/spark/pull/609
@koertkuipers can you the most recent syntax that you used to achieve the same?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org