python dependencies loaded but not on PYTHONPATH

[Dominik Hübner <co...@dhuebner.com>]

Hey,
I just tried to submit a task to my spark cluster using the following command 

./spark/bin/spark-submit --py-files file:///root/abc.zip --master spark://xxx.xxx.xxx.xxx:7077 test.py

It seems like the dependency I’ve added gets loaded:
14/08/05 23:07:00 INFO spark.SparkContext: Added file file:///root/abc.zip at http://xxx.xxx.xxx.xxx:40346/files/abc.zip with timestamp 1407280020217

However, my python script can’t find the module in this zip file. I already verified if this zip file is not corrupt by install it with “pip install abc.zip” (works fine). 
Any ideas how to get the content of the archive to the PYTHONPATH on my master and slaves?

Traceback (most recent call last):
  File "/root/test.py", line 7, in <module>
    import abc
ImportError: No module named abc

Maybe, it’s just the master complaining since and it only transfers the archive to the slaves (and adds it to the PYTHONPATH)?
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org