You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yesha Vora (JIRA)" <ji...@apache.org> on 2017/01/06 02:04:58 UTC
[jira] [Created] (SPARK-19097) virtualenv example failed with conda
due to ImportError: No module named ruamel.yaml.comments
Yesha Vora created SPARK-19097:
----------------------------------
Summary: virtualenv example failed with conda due to ImportError: No module named ruamel.yaml.comments
Key: SPARK-19097
URL: https://issues.apache.org/jira/browse/SPARK-19097
Project: Spark
Issue Type: Sub-task
Components: PySpark
Reporter: Yesha Vora
Spark version : 2
Steps:
* install conda on all nodes (python2.7) ( pip install conda )
* create requirement1.txt with "numpy > requirement1.txt "
* Run kmeans.py application in yarn-client mode.
{code}
spark-submit --master yarn --deploy-mode client --conf "spark.pyspark.virtualenv.enabled=true" --conf "spark.pyspark.virtualenv.type=conda" --conf "spark.pyspark.virtualenv.requirements=/tmp/requirements1.txt" --conf "spark.pyspark.virtualenv.bin.path=/usr/bin/conda" --jars /usr/hadoop-client/lib/hadoop-lzo.jar kmeans.py /tmp/in/kmeans_data.txt 3{code}
{code:title=app log}
17/01/06 01:39:25 DEBUG PythonWorkerFactory: user.home=/home/yarn
17/01/06 01:39:25 DEBUG PythonWorkerFactory: Running command:/usr/bin/conda create --prefix /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1483592608863_0017/container_1483592608863_0017_01_000003/virtualenv_application_1483592608863_0017_0 --file requirements1.txt -y
Traceback (most recent call last):
File "/usr/bin/conda", line 11, in <module>
load_entry_point('conda==4.2.7', 'console_scripts', 'conda')()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 561, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2291, in load
return self.resolve()
File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2297, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/lib/python2.7/site-packages/conda/cli/__init__.py", line 8, in <module>
from .main import main # NOQA
File "/usr/lib/python2.7/site-packages/conda/cli/main.py", line 46, in <module>
from ..base.context import context
File "/usr/lib/python2.7/site-packages/conda/base/context.py", line 18, in <module>
from ..common.configuration import (Configuration, MapParameter, PrimitiveParameter,
File "/usr/lib/python2.7/site-packages/conda/common/configuration.py", line 40, in <module>
from ruamel.yaml.comments import CommentedSeq, CommentedMap # pragma: no cover
ImportError: No module named ruamel.yaml.comments
17/01/06 01:39:26 WARN BlockManager: Putting block rdd_3_0 failed due to an exception
17/01/06 01:39:26 WARN BlockManager: Block rdd_3_0 could not be removed as it was not found on disk or in memory
17/01/06 01:39:26 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.RuntimeException: Fail to run command: /usr/bin/conda create --prefix /grid/0/hadoop/yarn/local/usercache/hrt_qa/appcache/application_1483592608863_0017/container_1483592608863_0017_01_000003/virtualenv_application_1483592608863_0017_0 --file requirements1.txt -y
at org.apache.spark.api.python.PythonWorkerFactory.execCommand(PythonWorkerFactory.scala:142)
at org.apache.spark.api.python.PythonWorkerFactory.setupVirtualEnv(PythonWorkerFactory.scala:124)
at org.apache.spark.api.python.PythonWorkerFactory.<init>(PythonWorkerFactory.scala:70)
at org.apache.spark.SparkEnv$$anonfun$createPythonWorker$1.apply(SparkEnv.scala:117)
at org.apache.spark.SparkEnv$$anonfun$createPythonWorker$1.apply(SparkEnv.scala:117)
at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116)
at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:957)
at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287){code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org