You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "guoqing0629@yahoo.com.hk" <gu...@yahoo.com.hk> on 2016/03/07 10:28:09 UTC

PYSPARK_PYTHON doesn't work in spark worker

Hi all
I had following configuration in spark worker (spark-env.sh)
export PYTHON_HOME=/opt/soft/anaconda2
export PYSPARK_PYTHON=$PYTHON_HOME/bin/python

I'm try to run a simple test script on  pyspark --master yarn --queue spark --executor-cores 1  --num-executors 10
from pyspark import SparkContext, SparkConf
df = sc.textFile("/user/hive/warehouse/hdptest.db/111");
df.collect();
sortedCount = df.map(lambda x: (float(x), 1)).sortByKey();
sortedCount.collect();

However , i get following error  : 
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 7, datanode-8): org.apache.spark.SparkException: 
Error from python worker:
  /usr/bin/python: module pyspark.daemon not found
PYTHONPATH was:
  /home/hadoop/spark/python/lib/pyspark.zip:/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/data/hadoop_local/usercache/hdptest/filecache/38/spark-assembly-1.4.1-hadoop2.5.2.jar:pyspark.zip:py4j-0.8.2.1-src.zip

It seem that PYSPARK_PYTHON doesn't work in spark worker  ,  can someone please help me to solve it ? 

Thanks~


guoqing0629@yahoo.com.hk

Re: PYSPARK_PYTHON doesn't work in spark worker

Posted by Gourav Sengupta <go...@gmail.com>.
hi,

how are you running your SPARK cluster (is it in local mode or distributed
mode).  Do you have pyspark installed in anaconda?


Regards,
Gourav Sengupta

On Mon, Mar 7, 2016 at 9:28 AM, guoqing0629@yahoo.com.hk <
guoqing0629@yahoo.com.hk> wrote:

> Hi all
> I had following configuration in spark worker (spark-env.sh)
> export PYTHON_HOME=/opt/soft/anaconda2
> export PYSPARK_PYTHON=$PYTHON_HOME/bin/python
>
> I'm try to run a simple test script on
> pyspark --master yarn --queue spark --executor-cores 1  --num-executors 10
> from pyspark import SparkContext, SparkConf
> df = sc.textFile("/user/hive/warehouse/hdptest.db/111");
> df.collect();
> sortedCount = df.map(lambda x: (float(x), 1)).sortByKey();
> sortedCount.collect();
>
> However , i get following error  :
>
>
>
>
>
> *org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 4 times, most recent failure: Lost task 1.3 in stage 1.0 (TID 7, datanode-8): org.apache.spark.SparkException: Error from python worker:  /usr/bin/python: module pyspark.daemon not foundPYTHONPATH was:  /home/hadoop/spark/python/lib/pyspark.zip:/home/hadoop/spark/python/lib/py4j-0.8.2.1-src.zip:/data/hadoop_local/usercache/hdptest/filecache/38/spark-assembly-1.4.1-hadoop2.5.2.jar:pyspark.zip:py4j-0.8.2.1-src.zip*
>
> It seem that PYSPARK_PYTHON doesn't work in spark worker  ,  can someone
> please help me to solve it ?
>
> Thanks~
> ------------------------------
> guoqing0629@yahoo.com.hk
>