You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Marcel Hofmann <mh...@gmail.com> on 2015/06/11 21:13:40 UTC
PySpark Interpretor not working
Hey everybody,
I'm currently testing Zeppelin, but unfortunatley, I can't really get it up
and running. The example notebook is running just fine, and everything
works there, but a simple pyspark statement like:
%pyspark
list = range(1,4)
print(list)
will not execute. Looking at the interpretor-spark logs the only thing
happening is:
INFO [2015-06-11 12:05:21,664] ({pool-2-thread-3}
SchedulerFactory.java[jobStarted]:132) - Job
remoteInterpretJob_1434049521664 started by scheduler interpreter_836706052
INFO [2015-06-11 12:05:21,667] ({pool-2-thread-3}
PySparkInterpreter.java[createPythonScript]:123) - File
/tmp/zeppelin_pyspark.py created
I have set the following env-variables:
export SPARK_HOME=/home/biadmin/spark-1.3.0/spark-1.3.0-bin-hadoop2.4
export PYSPARK_PYTHON=python
export
PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip
Thanks for any advice!
Marcel
Re: PySpark Interpretor not working
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Looks like your configuration okay, if you're not using yarn-client mode.
Could you make sure your interpreter setting have spark.home property set
to /home/biadmin/spark-1.3.0/spark-1.3.0-bin-hadoop2.4 ?
Thanks,
moon
On Thu, Jun 11, 2015 at 12:14 PM Marcel Hofmann <mh...@gmail.com>
wrote:
> Hey everybody,
>
> I'm currently testing Zeppelin, but unfortunatley, I can't really get it
> up and running. The example notebook is running just fine, and everything
> works there, but a simple pyspark statement like:
>
> %pyspark
> list = range(1,4)
> print(list)
>
> will not execute. Looking at the interpretor-spark logs the only thing
> happening is:
>
> INFO [2015-06-11 12:05:21,664] ({pool-2-thread-3}
> SchedulerFactory.java[jobStarted]:132) - Job
> remoteInterpretJob_1434049521664 started by scheduler interpreter_836706052
> INFO [2015-06-11 12:05:21,667] ({pool-2-thread-3}
> PySparkInterpreter.java[createPythonScript]:123) - File
> /tmp/zeppelin_pyspark.py created
>
> I have set the following env-variables:
> export SPARK_HOME=/home/biadmin/spark-1.3.0/spark-1.3.0-bin-hadoop2.4
> export PYSPARK_PYTHON=python
> export
> PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip
>
> Thanks for any advice!
>
> Marcel
>
>
Re: PySpark Interpretor not working
Posted by moon soo Lee <mo...@apache.org>.
Hi,
https://github.com/apache/incubator-zeppelin/pull/118
is recently merged and i think it helps configuring pyspark with Yarn.
Please try latest branch and let me know if it helps.
Thanks,
moon
On Tue, Jun 30, 2015 at 12:10 AM IT CTO <go...@gmail.com> wrote:
> I have the same configuration but when I run the program the process never
> returns. status remain on running
> log say SEND >> PROGRESS
> forever....
>
> On Sun, Jun 14, 2015 at 2:34 AM MrAsanjar . <af...@gmail.com> wrote:
>
>> hi
>> I had a similar issue, try these:
>> 1) add following settings to zeppelin-env.sh ( it must be added there at
>> this time)
>> export MASTER=yarn-client
>> export HADOOP_CONF_DIR=/etc/hadoop/conf
>> export
>> PYTHONPATH=/usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip
>> export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"
>> export SPARK_HOME=/usr/lib/spark
>> 2) Make sure /tmp has 1777 access permission
>> 3) restart the daemon "zeppelin-daemon.sh restart"
>>
>> On Thu, Jun 11, 2015 at 2:13 PM, Marcel Hofmann <mh...@gmail.com>
>> wrote:
>>
>>> Hey everybody,
>>>
>>> I'm currently testing Zeppelin, but unfortunatley, I can't really get it
>>> up and running. The example notebook is running just fine, and everything
>>> works there, but a simple pyspark statement like:
>>>
>>> %pyspark
>>> list = range(1,4)
>>> print(list)
>>>
>>> will not execute. Looking at the interpretor-spark logs the only thing
>>> happening is:
>>>
>>> INFO [2015-06-11 12:05:21,664] ({pool-2-thread-3}
>>> SchedulerFactory.java[jobStarted]:132) - Job
>>> remoteInterpretJob_1434049521664 started by scheduler interpreter_836706052
>>> INFO [2015-06-11 12:05:21,667] ({pool-2-thread-3}
>>> PySparkInterpreter.java[createPythonScript]:123) - File
>>> /tmp/zeppelin_pyspark.py created
>>>
>>> I have set the following env-variables:
>>> export SPARK_HOME=/home/biadmin/spark-1.3.0/spark-1.3.0-bin-hadoop2.4
>>> export PYSPARK_PYTHON=python
>>> export
>>> PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip
>>>
>>> Thanks for any advice!
>>>
>>> Marcel
>>>
>>>
>>
Re: PySpark Interpretor not working
Posted by IT CTO <go...@gmail.com>.
I have the same configuration but when I run the program the process never
returns. status remain on running
log say SEND >> PROGRESS
forever....
On Sun, Jun 14, 2015 at 2:34 AM MrAsanjar . <af...@gmail.com> wrote:
> hi
> I had a similar issue, try these:
> 1) add following settings to zeppelin-env.sh ( it must be added there at
> this time)
> export MASTER=yarn-client
> export HADOOP_CONF_DIR=/etc/hadoop/conf
> export
> PYTHONPATH=/usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip
> export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"
> export SPARK_HOME=/usr/lib/spark
> 2) Make sure /tmp has 1777 access permission
> 3) restart the daemon "zeppelin-daemon.sh restart"
>
> On Thu, Jun 11, 2015 at 2:13 PM, Marcel Hofmann <mh...@gmail.com>
> wrote:
>
>> Hey everybody,
>>
>> I'm currently testing Zeppelin, but unfortunatley, I can't really get it
>> up and running. The example notebook is running just fine, and everything
>> works there, but a simple pyspark statement like:
>>
>> %pyspark
>> list = range(1,4)
>> print(list)
>>
>> will not execute. Looking at the interpretor-spark logs the only thing
>> happening is:
>>
>> INFO [2015-06-11 12:05:21,664] ({pool-2-thread-3}
>> SchedulerFactory.java[jobStarted]:132) - Job
>> remoteInterpretJob_1434049521664 started by scheduler interpreter_836706052
>> INFO [2015-06-11 12:05:21,667] ({pool-2-thread-3}
>> PySparkInterpreter.java[createPythonScript]:123) - File
>> /tmp/zeppelin_pyspark.py created
>>
>> I have set the following env-variables:
>> export SPARK_HOME=/home/biadmin/spark-1.3.0/spark-1.3.0-bin-hadoop2.4
>> export PYSPARK_PYTHON=python
>> export
>> PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip
>>
>> Thanks for any advice!
>>
>> Marcel
>>
>>
>
Re: PySpark Interpretor not working
Posted by "MrAsanjar ." <af...@gmail.com>.
hi
I had a similar issue, try these:
1) add following settings to zeppelin-env.sh ( it must be added there at
this time)
export MASTER=yarn-client
export HADOOP_CONF_DIR=/etc/hadoop/conf
export
PYTHONPATH=/usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip
export SPARK_YARN_USER_ENV="PYTHONPATH=${PYTHONPATH}"
export SPARK_HOME=/usr/lib/spark
2) Make sure /tmp has 1777 access permission
3) restart the daemon "zeppelin-daemon.sh restart"
On Thu, Jun 11, 2015 at 2:13 PM, Marcel Hofmann <mh...@gmail.com>
wrote:
> Hey everybody,
>
> I'm currently testing Zeppelin, but unfortunatley, I can't really get it
> up and running. The example notebook is running just fine, and everything
> works there, but a simple pyspark statement like:
>
> %pyspark
> list = range(1,4)
> print(list)
>
> will not execute. Looking at the interpretor-spark logs the only thing
> happening is:
>
> INFO [2015-06-11 12:05:21,664] ({pool-2-thread-3}
> SchedulerFactory.java[jobStarted]:132) - Job
> remoteInterpretJob_1434049521664 started by scheduler interpreter_836706052
> INFO [2015-06-11 12:05:21,667] ({pool-2-thread-3}
> PySparkInterpreter.java[createPythonScript]:123) - File
> /tmp/zeppelin_pyspark.py created
>
> I have set the following env-variables:
> export SPARK_HOME=/home/biadmin/spark-1.3.0/spark-1.3.0-bin-hadoop2.4
> export PYSPARK_PYTHON=python
> export
> PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip
>
> Thanks for any advice!
>
> Marcel
>
>