You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Guanhua Yan <gh...@lanl.gov> on 2014/04/29 18:51:18 UTC

Python Spark on YARN

Hi all:

Is it possible to develop Spark programs in Python and run them on YARN?
>From the Python SparkContext class, it doesn't seem to have such an option.

Thank you,
- Guanhua

===============================
Guanhua Yan, Ph.D.
Information Sciences Group (CCS-3)
Los Alamos National Laboratory
Tel: +1-505-667-0176
Email: ghyan@lanl.gov
Web: http://ghyan.weebly.com/
===============================

Re: Python Spark on YARN

Posted by Guanhua Yan <gh...@lanl.gov>.

Thanks, Matei. Will take a look at it.

Best regards,
Guanhua

From:  Matei Zaharia <ma...@gmail.com>
Reply-To:  <us...@spark.apache.org>
Date:  Tue, 29 Apr 2014 14:19:30 -0700
To:  <us...@spark.apache.org>
Subject:  Re: Python Spark on YARN

This will be possible in 1.0 after this pull request:
https://github.com/apache/spark/pull/30

Matei

On Apr 29, 2014, at 9:51 AM, Guanhua Yan <gh...@lanl.gov> wrote:

> Hi all:
> 
> Is it possible to develop Spark programs in Python and run them on YARN? From
> the Python SparkContext class, it doesn't seem to have such an option.
> 
> Thank you,
> - Guanhua
> 
> ===============================
> Guanhua Yan, Ph.D.
> Information Sciences Group (CCS-3)
> Los Alamos National Laboratory
> Tel: +1-505-667-0176
> Email: ghyan@lanl.gov
> Web: http://ghyan.weebly.com <http://ghyan.weebly.com/> /
> ===============================

Re: Python Spark on YARN

Posted by Matei Zaharia <ma...@gmail.com>.

This will be possible in 1.0 after this pull request: https://github.com/apache/spark/pull/30

Matei

On Apr 29, 2014, at 9:51 AM, Guanhua Yan <gh...@lanl.gov> wrote:

> Hi all:
> 
> Is it possible to develop Spark programs in Python and run them on YARN? From the Python SparkContext class, it doesn't seem to have such an option.
> 
> Thank you,
> - Guanhua
> 
> ===============================
> Guanhua Yan, Ph.D.
> Information Sciences Group (CCS-3)
> Los Alamos National Laboratory
> Tel: +1-505-667-0176
> Email: ghyan@lanl.gov
> Web: http://ghyan.weebly.com/
> ===============================