You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anahita Talebi <an...@gmail.com> on 2017/03/03 14:43:40 UTC
How to run a spark on Pycharm
Hi everyone,
I am trying to run a spark code on Pycharm. I tried to give the path of
spark as a environment variable to the configuration of Pycharm.
Unfortunately, I get the error. Does anyone know how I can run the spark
code on Pycharm?
It shouldn't be necessarily on Pycharm. if you know any other software, It
would be nice to tell me.
Thanks a lot,
Anahita
Re: How to run a spark on Pycharm
Posted by "Pushkar.Gujar" <pu...@gmail.com>.
There are lot of articles available online which guide you thru setting up
jupyter notebooks to run spark program. For e.g -
http://blog.insightdatalabs.com/jupyter-on-apache-spark-step-by-step/
https://www.cloudera.com/documentation/enterprise/5-6-x/topics/spark_ipython.html
https://gist.github.com/tommycarpi/f5a67c66a8f2170e263c
Thank you,
*Pushkar Gujar*
On Fri, Mar 3, 2017 at 10:05 AM, Anahita Talebi <an...@gmail.com>
wrote:
> Hi,
>
> Thanks for your answer.
>
> Sorry, I am completely beginner in running the code in spark.
>
> Could you please tell me a bit more in details how to do that?
> I installed ipython and Jupyter notebook on my local machine. But how can
> I run the code using them? Before, I tried to run the code with Pycharm
> that I was failed.
>
> Thanks,
> Anahita
>
> On Fri, Mar 3, 2017 at 3:48 PM, Pushkar.Gujar <pu...@gmail.com>
> wrote:
>
>> Jupyter notebook/ipython can be connected to apache spark
>>
>>
>> Thank you,
>> *Pushkar Gujar*
>>
>>
>> On Fri, Mar 3, 2017 at 9:43 AM, Anahita Talebi <anahita.t.amiri@gmail.com
>> > wrote:
>>
>>> Hi everyone,
>>>
>>> I am trying to run a spark code on Pycharm. I tried to give the path of
>>> spark as a environment variable to the configuration of Pycharm.
>>> Unfortunately, I get the error. Does anyone know how I can run the spark
>>> code on Pycharm?
>>> It shouldn't be necessarily on Pycharm. if you know any other software,
>>> It would be nice to tell me.
>>>
>>> Thanks a lot,
>>> Anahita
>>>
>>>
>>>
>>
>
Re: How to run a spark on Pycharm
Posted by "TheGeorge1918 ." <zh...@gmail.com>.
Hey,
Depends on your configuration. I configure my dockerfile with spark2.0
installed and in pycharm, properly configure the interpreter using docker
and add following env in your script configuration. You can check the
dockerfile here: https://github.com/zhangxuan1918/spark2.0
PYSPARK_PYTHON /usr/bin/python
PYSPARK_DRIVER_PYTHON /usr/bin/python
PYTHONPATH /usr/spark/python:/usr/spark/python/lib/py4j-0.10.1-src.zip:$PYTHONPATH
SPARK_HOME /usr/spark
On 3 Mar 2017, at 16:11, Sidney Feiner <si...@startapp.com> wrote:
Hey,
I once found an article about that:
https://mengdong.github.io/2016/08/08/fully-armed-pyspark-with-ipython-and-
jupyter/
And I once managed to set it up on Pycharm as well. What I had to do was to
add /path/to/spark to a system variable called "PYTHTONPATH".
Try that one, it might help J
*From:* Anahita Talebi [mailto:anahita.t.amiri@gmail.com]
*Sent:* Friday, March 3, 2017 5:05 PM
*To:* Pushkar.Gujar <pu...@gmail.com>
*Cc:* User <us...@spark.apache.org>
*Subject:* Re: How to run a spark on Pycharm
Hi,
Thanks for your answer.
Sorry, I am completely beginner in running the code in spark.
Could you please tell me a bit more in details how to do that?
I installed ipython and Jupyter notebook on my local machine. But how can I
run the code using them? Before, I tried to run the code with Pycharm that
I was failed.
Thanks,
Anahita
On Fri, Mar 3, 2017 at 3:48 PM, Pushkar.Gujar <pu...@gmail.com>
wrote:
Jupyter notebook/ipython can be connected to apache spark
Thank you,
*Pushkar Gujar*
On Fri, Mar 3, 2017 at 9:43 AM, Anahita Talebi <an...@gmail.com>
wrote:
Hi everyone,
I am trying to run a spark code on Pycharm. I tried to give the path of
spark as a environment variable to the configuration of Pycharm.
Unfortunately, I get the error. Does anyone know how I can run the spark
code on Pycharm?
It shouldn't be necessarily on Pycharm. if you know any other software, It
would be nice to tell me.
Thanks a lot,
Anahita
RE: How to run a spark on Pycharm
Posted by Sidney Feiner <si...@startapp.com>.
Hey,
I once found an article about that:
https://mengdong.github.io/2016/08/08/fully-armed-pyspark-with-ipython-and-jupyter/
And I once managed to set it up on Pycharm as well. What I had to do was to add /path/to/spark to a system variable called "PYTHTONPATH".
Try that one, it might help ☺
From: Anahita Talebi [mailto:anahita.t.amiri@gmail.com]
Sent: Friday, March 3, 2017 5:05 PM
To: Pushkar.Gujar <pu...@gmail.com>
Cc: User <us...@spark.apache.org>
Subject: Re: How to run a spark on Pycharm
Hi,
Thanks for your answer.
Sorry, I am completely beginner in running the code in spark.
Could you please tell me a bit more in details how to do that?
I installed ipython and Jupyter notebook on my local machine. But how can I run the code using them? Before, I tried to run the code with Pycharm that I was failed.
Thanks,
Anahita
On Fri, Mar 3, 2017 at 3:48 PM, Pushkar.Gujar <pu...@gmail.com>> wrote:
Jupyter notebook/ipython can be connected to apache spark
Thank you,
Pushkar Gujar
On Fri, Mar 3, 2017 at 9:43 AM, Anahita Talebi <an...@gmail.com>> wrote:
Hi everyone,
I am trying to run a spark code on Pycharm. I tried to give the path of spark as a environment variable to the configuration of Pycharm. Unfortunately, I get the error. Does anyone know how I can run the spark code on Pycharm?
It shouldn't be necessarily on Pycharm. if you know any other software, It would be nice to tell me.
Thanks a lot,
Anahita
Re: How to run a spark on Pycharm
Posted by Anahita Talebi <an...@gmail.com>.
Hi,
Thanks for your answer.
Sorry, I am completely beginner in running the code in spark.
Could you please tell me a bit more in details how to do that?
I installed ipython and Jupyter notebook on my local machine. But how can I
run the code using them? Before, I tried to run the code with Pycharm that
I was failed.
Thanks,
Anahita
On Fri, Mar 3, 2017 at 3:48 PM, Pushkar.Gujar <pu...@gmail.com>
wrote:
> Jupyter notebook/ipython can be connected to apache spark
>
>
> Thank you,
> *Pushkar Gujar*
>
>
> On Fri, Mar 3, 2017 at 9:43 AM, Anahita Talebi <an...@gmail.com>
> wrote:
>
>> Hi everyone,
>>
>> I am trying to run a spark code on Pycharm. I tried to give the path of
>> spark as a environment variable to the configuration of Pycharm.
>> Unfortunately, I get the error. Does anyone know how I can run the spark
>> code on Pycharm?
>> It shouldn't be necessarily on Pycharm. if you know any other software,
>> It would be nice to tell me.
>>
>> Thanks a lot,
>> Anahita
>>
>>
>>
>
Re: How to run a spark on Pycharm
Posted by "Pushkar.Gujar" <pu...@gmail.com>.
Jupyter notebook/ipython can be connected to apache spark
Thank you,
*Pushkar Gujar*
On Fri, Mar 3, 2017 at 9:43 AM, Anahita Talebi <an...@gmail.com>
wrote:
> Hi everyone,
>
> I am trying to run a spark code on Pycharm. I tried to give the path of
> spark as a environment variable to the configuration of Pycharm.
> Unfortunately, I get the error. Does anyone know how I can run the spark
> code on Pycharm?
> It shouldn't be necessarily on Pycharm. if you know any other software, It
> would be nice to tell me.
>
> Thanks a lot,
> Anahita
>
>
>