Fwd: May I ask a question about Toree?

[Russell Luo <rl...@mz.com.INVALID>]

Hello,

My name is Russell. My company is currently using Toree pyspark. However, I
encounter a problem that I couldn't figure it out:

here is the kernel setting:

{
  "language": "python",
  "display_name": "Apache Toree - PySpark",
  "env": {
    "__TOREE_SPARK_OPTS__": "--master yarn-client",
    "SPARK_HOME": "/usr/hdp/current/spark-client",
    "__TOREE_OPTS__": "",
    "DEFAULT_INTERPRETER": "PySpark",
    "PYTHONPATH": "/usr/hdp/current/spark-client/python:/usr/hdp/
current/spark-client/python/lib/py4j-0.9-src.zip",
    "PYTHON_EXEC": "python"
  },
  "argv": [
    "/usr/local/share/jupyter/kernels/apache_toree_pyspark/bin/run.sh",
    "--profile",
    "{connection_file}"
  ]
}

When I run a small pyspark program on jupyter note book, I got these error
message:

Error from python worker:
  /usr/bin/python: No module named pyspark
PYTHONPATH was:
  /data18/hadoop/yarn/local/filecache/10/spark-hdp-assembly.jar
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(
PythonWorkerFactory.scala:164)
at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(
PythonWorkerFactory.scala:87)
at org.apache.spark.api.python.PythonWorkerFactory.create(
PythonWorkerFactory.scala:63)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:134)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:101)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:70)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:313)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:277)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


Could you give me some hints what would be the possible cause? I have tried
submitting python job using the command line in yarn-client mode, and there
is no problem getting the result. I think there must be some setting
problems.

Any help is greatly appreciated. Thanks ahead!

Regards,
Russell