You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by chutium <te...@gmail.com> on 2014/08/08 23:03:17 UTC
spark-shell is broken! (bad option: '--master')
git clone https://github.com/apache/spark.git
mvn -Pyarn -Phive -Phive-thriftserver -Dhadoop.version=2.3.0-cdh5.1.0
-DskipTests clean package
./bin/spark-shell --master local[2]
then i get
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
bad option: '--master'
unbelievable...
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
Posted by chutium <te...@gmail.com>.
an issue 3 - 4 PR, spark dev community is really active :)
it seems currently spark-shell takes only some SUBMISSION_OPTS, but no
APPLICATION_OPTS
do you have plan to add some APPLICATION_OPTS or CLI_OPTS like
hive -e
hive -f
hive -hivevar
then we can use our scala code as scripts, run them direktly via
spark-shell, without compiling, building, packing and so on...
those APPLICATION_OPTS should be some augments for
org.apache.spark.repl.Main right?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7798.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
Posted by chutium <te...@gmail.com>.
maybe this commit is the reason?
https://github.com/apache/spark/commit/a6cd31108f0d73ce6823daafe8447677e03cfd13
i fand some discuss in its PR: https://github.com/apache/spark/pull/1801
important is what vanzin said:
https://github.com/apache/spark/pull/1801#issuecomment-51545117
i tried to use this:
bin/spark-class org.apache.spark.deploy.SparkSubmit --class
org.apache.spark.repl.Main spark-shell --master local[2]
also
bad option: '--master'
but this works
bin/spark-class org.apache.spark.deploy.SparkSubmit --master local[2]
--class org.apache.spark.repl.Main spark-shell
and some more try out:
~/tachyon/spark$ bin/spark-class org.apache.spark.deploy.SparkSubmit
--master local[2] --class org.apache.spark.repl.Main spark-shell --foo bar
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
bad option: '--foo'
~/tachyon/spark$ bin/spark-class org.apache.spark.deploy.SparkSubmit
--master local[2] --foo bar --class org.apache.spark.repl.Main spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on
classpath
Error: Unrecognized option '--foo'.
Run with --help for usage help or --verbose for debug output
god... no test suite for those shell scripts?
such a issue is really awful~
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7779.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
Posted by Cheng Lian <li...@gmail.com>.
Just opened a PR based on the branch Patrick mentioned for this issue
https://github.com/apache/spark/pull/1864
On Sat, Aug 9, 2014 at 6:48 AM, Patrick Wendell <pw...@gmail.com> wrote:
> Cheng Lian also has a fix for this. I've asked him to make a PR - he
> is on China time so it probably won't come until tonight:
>
>
> https://github.com/liancheng/spark/compare/apache:master...liancheng:spark-2894
>
> On Fri, Aug 8, 2014 at 3:46 PM, Sandy Ryza <sa...@cloudera.com>
> wrote:
> > Hi Chutium,
> >
> > This is currently being addressed in
> > https://github.com/apache/spark/pull/1825
> >
> > -Sandy
> >
> >
> > On Fri, Aug 8, 2014 at 2:26 PM, chutium <te...@gmail.com> wrote:
> >
> >> no one use spark-shell in master branch?
> >>
> >> i created a PR as follow up commit of SPARK-2678 and PR #1801:
> >>
> >> https://github.com/apache/spark/pull/1861
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html
> >> Sent from the Apache Spark Developers List mailing list archive at
> >> Nabble.com.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> For additional commands, e-mail: dev-help@spark.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: spark-shell is broken! (bad option: '--master')
Posted by Patrick Wendell <pw...@gmail.com>.
Cheng Lian also has a fix for this. I've asked him to make a PR - he
is on China time so it probably won't come until tonight:
https://github.com/liancheng/spark/compare/apache:master...liancheng:spark-2894
On Fri, Aug 8, 2014 at 3:46 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Chutium,
>
> This is currently being addressed in
> https://github.com/apache/spark/pull/1825
>
> -Sandy
>
>
> On Fri, Aug 8, 2014 at 2:26 PM, chutium <te...@gmail.com> wrote:
>
>> no one use spark-shell in master branch?
>>
>> i created a PR as follow up commit of SPARK-2678 and PR #1801:
>>
>> https://github.com/apache/spark/pull/1861
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-shell is broken! (bad option: '--master')
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Chutium,
This is currently being addressed in
https://github.com/apache/spark/pull/1825
-Sandy
On Fri, Aug 8, 2014 at 2:26 PM, chutium <te...@gmail.com> wrote:
> no one use spark-shell in master branch?
>
> i created a PR as follow up commit of SPARK-2678 and PR #1801:
>
> https://github.com/apache/spark/pull/1861
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: spark-shell is broken! (bad option: '--master')
Posted by chutium <te...@gmail.com>.
no one use spark-shell in master branch?
i created a PR as follow up commit of SPARK-2678 and PR #1801:
https://github.com/apache/spark/pull/1861
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-shell-is-broken-bad-option-master-tp7778p7780.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org