You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by freedafeng <fr...@yahoo.com> on 2015/01/15 00:14:37 UTC

how to run python app in yarn?

A cdh5.3.0 with spark is set up. just wondering how to run a python
application on it. 

I used 'spark-submit --master yarn-cluster ./loadsessions.py' but got the
error,

Error: Cluster deploy mode is currently not supported for python
applications.
Run with --help for usage help or --verbose for debug output

Thanks.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-run-python-app-in-yarn-tp21141.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: how to run python app in yarn?

Posted by Jian Feng <fr...@yahoo.com.INVALID>.
Thanks Josh and Marcelo! It now works!
BTW, just wondering, is there any perf difference between running spark in standalone mode and under yarn? The only goal that I created this cluster is to run spark jobs. So I can set up spark in standalone mode if it runs slow in yarn.
best.
      From: Josh Rosen <ro...@gmail.com>
 To: Marcelo Vanzin <va...@cloudera.com> 
Cc: freedafeng <fr...@yahoo.com>; "user@spark.apache.org" <us...@spark.apache.org> 
 Sent: Wednesday, January 14, 2015 3:20 PM
 Subject: Re: how to run python app in yarn?
   
There's an open PR for supporting yarn-cluster mode in PySpark: https://github.com/apache/spark/pull/3976 (currently blocked on reviewer attention / time)


On Wed, Jan 14, 2015 at 3:16 PM, Marcelo Vanzin <va...@cloudera.com> wrote:

As the error message says...

On Wed, Jan 14, 2015 at 3:14 PM, freedafeng <fr...@yahoo.com> wrote:
> Error: Cluster deploy mode is currently not supported for python
> applications.

Use "yarn-client" instead of "yarn-cluster" for pyspark apps.


--
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org





  

Re: how to run python app in yarn?

Posted by Josh Rosen <ro...@gmail.com>.
There's an open PR for supporting yarn-cluster mode in PySpark:
https://github.com/apache/spark/pull/3976 (currently blocked on reviewer
attention / time)

On Wed, Jan 14, 2015 at 3:16 PM, Marcelo Vanzin <va...@cloudera.com> wrote:

> As the error message says...
>
> On Wed, Jan 14, 2015 at 3:14 PM, freedafeng <fr...@yahoo.com> wrote:
> > Error: Cluster deploy mode is currently not supported for python
> > applications.
>
> Use "yarn-client" instead of "yarn-cluster" for pyspark apps.
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: how to run python app in yarn?

Posted by Marcelo Vanzin <va...@cloudera.com>.
As the error message says...

On Wed, Jan 14, 2015 at 3:14 PM, freedafeng <fr...@yahoo.com> wrote:
> Error: Cluster deploy mode is currently not supported for python
> applications.

Use "yarn-client" instead of "yarn-cluster" for pyspark apps.


-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: how to run python app in yarn?

Posted by freedafeng <fr...@yahoo.com>.
Got help from Marcelo and Josh. Now it is running smoothly. In case you need
this info - Just use "yarn-client" instead of "yarn-cluster" 

Thanks folks!



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-run-python-app-in-yarn-tp21141p21142.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org