You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by pankajarora12 <gi...@git.apache.org> on 2014/05/31 13:15:11 UTC
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
GitHub user pankajarora12 opened a pull request:
https://github.com/apache/spark/pull/930
[SPARK-1979] Added Error Handling if user passes application params with...
Added error message to user when used --arg for passing application parameters.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pankajarora12/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/930.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #930
----
commit d6bfba3d7b9236a02a7e91233f8e512bea761af0
Author: pankaj.arora <pa...@guavus.com>
Date: 2014-05-31T11:11:05Z
[SPARK-1979] Added Error Handling if user passes application params with --arg
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-54694684
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-45053339
What if the user application legitimately has a flag called --arg? Won't this cause it to break?
Also (just wondering) - why do you think users would be setting `--arg`... just from previous submissions scripts in Spark?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-54365418
Hi @pankajarora12, this issue is outdated since #1801 went in. Would you mind closing this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-45053395
I think right now of the user uses `--arg` before specifying the app jar it will say "unrecognized option --arg". This might be the best behavior we can achieve.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by pankajarora12 <gi...@git.apache.org>.
Github user pankajarora12 commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-45191676
@pwendell
/bin/spark-submit --name "My app" --master "local[4]" myJar.jar --name "This goes to the app"
If I pass --master twice, earlier --master argument passed gets overridden by the one following it. That means you cannot use --master as an application argument.
The command i fired was
/opt/spark-1.0.0/bin/spark-submit --master yarn /opt/tms/java/app-1.0-SNAPSHOT-jar-with-dependencies.jar --master yarn1 --deploy-mode cluster --verbose --class com.guavus.scala.MainClass yarn-cluster /data/output /data/collector/1/output/http/2014/05/11/22/50/72/ VS-NN --driver-memory 9099M --executor-memory 9099M --num-executors 1 --executor-cores 1
and master field took the value as yarn1 also --master yarn1 did not get passed to our App.
BTW I upgraded from tag v1.0.0-rc2 to v1.0.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by pankajarora12 <gi...@git.apache.org>.
Github user pankajarora12 commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-45066860
1. What if the user application legitimately has a flag called --arg? Won't this cause it to break?
So there are already reserved keywords like --
--name
--master ....
which user apps cannot use because spark considers them as special words. So asking the user to use different name rather than using --arg would be a good idea .Also in case spark starts using --arg for some special meaning user applications using --arg as application parameters will break.
2. Also (just wondering) - why do you think users would be setting --arg... just from previous submissions scripts in Spark?
I have scheduling framework written for firing jobs to spark. After upgrading to spark 1.0.0 my application was broken and it took me quite a lot time to find out why it was not taking parameters. So any one upgrading from previous version to 1.0.0 will face the same problem and they have to debug to find out whats failing. So an error message would be very useful.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-45132358
@pankajarora12 no, those aren't reserved keywords. For instance:
```
./bin/spark-submit --name "My app" --master "local[4]" myJar.jar --name "This goes to the app"
```
So just to be clear you were upgrading from `master --> 1.0`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/930
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-45166972
> What if the user application legitimately has a flag called --arg?
Perhaps the command line parser should add a case for "--", meaning "stop parsing arguments", at the same time making it more compatible with what people are used to. :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-1979] Added Error Handling if user pass...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/930#issuecomment-44745616
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---