pyspark interpreter not found

[Hammad <ha...@flexilogix.com>]

Using

- zeppeling-0.6-SNAPSHOT
[mvn clean package -Pspark-1.5 -Phadoop-2.6 -Pyarn -Ppyspark -DskipTests]

- Spark 1.5.2

1. Everything worked on zeppelin other than pyspark
when tried to execute as simple as below;
%pyspark
from os import getcwd

giving this error:
pyspark interpreter not found

2. Googling, I found out that below two environment variables need setting,
so did it
SPARK_HOME="/home/incubator/myspark"
PYTHONPATH="/home/incubator/myspark/python:/home/incubator/myspark/python/lib/py4j-0.8.2.1-src.zip"

after this,even below command wouldnt work;
sc.version

giving this error:
java.lang.ClassCastException: scala.None$ cannot be cast to java.util.Map

pyspark problem also stays there.

Please help with any thoughts on whats the proper way to make pyspark work
after fresh build of zeppelin and spark?

Regards,
Hmad

Re: pyspark interpreter not found

[Hammad <ha...@flexilogix.com>]

[SOLVED]
Following is summary of configuration in order to make zeppelin run with
pyspark

1. Set environment variables in /etc/environment

SPARK_HOME="/home/of/spark"
PYTHONPATH="/home/of/spark/python:/home/of/spark/python/lib/py4j-0.8.2.1-src.zip"


2. In zeppelin-env.sh

#directory of spark installation; should be same as spark.home in above(2)
export SPARK_HOME='/home/of/spark'

#important:python directory under spark installation to identify pyspark
export PYSPARK_PYTHON='/home/of/spark/python'

#primarily its for py4j distro and pyspark
export
PYTHONPATH='/home/of/spark/python:/home/of/spark/python/lib/py4j-0.8.2.1-src.zip'



3. After starting zeppelin, in UI -> Interpretter: add belows properties in
spark configuration section

a. Add below property that should be same as SPARK_HOME in zeppelin-env.sh
spark*.*home /home/of/spark

b. Edit below property to reflect location of 'python' command
zeppelin.pyspark.python /usr/bin/python


Thanks,
Hmad



On Thu, Jan 7, 2016 at 10:43 AM, Hammad <ha...@flexilogix.com> wrote:

> Using
>
> - zeppeling-0.6-SNAPSHOT
> [mvn clean package -Pspark-1.5 -Phadoop-2.6 -Pyarn -Ppyspark -DskipTests]
>
> - Spark 1.5.2
>
> 1. Everything worked on zeppelin other than pyspark
> when tried to execute as simple as below;
> %pyspark
> from os import getcwd
>
> giving this error:
> pyspark interpreter not found
>
> 2. Googling, I found out that below two environment variables need
> setting, so did it
> SPARK_HOME="/home/incubator/myspark"
>
> PYTHONPATH="/home/incubator/myspark/python:/home/incubator/myspark/python/lib/py4j-0.8.2.1-src.zip"
>
> after this,even below command wouldnt work;
> sc.version
>
> giving this error:
> java.lang.ClassCastException: scala.None$ cannot be cast to java.util.Map
>
> pyspark problem also stays there.
>
> Please help with any thoughts on whats the proper way to make pyspark work
> after fresh build of zeppelin and spark?
>
> Regards,
> Hmad
>



-- 
Flexilogix
Ph: +92 618090374
Fax: +92 612011810
http://www.flexilogix.com
info@flexilogix.com

Disclaimer:  This transmission (including any attachments) may contain
confidential information, privileged material or constitute non-public
information. Any use of this information by anyone other than the intended
recipient is prohibited. If you have received this transmission in error,
please immediately reply to the sender and delete this information from
your system.