You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Chengi Liu <ch...@gmail.com> on 2014/08/28 07:58:52 UTC

Submitting multiple files pyspark

Hi,
  I have two files..

main_app.py and helper.py
main_app.py calls some functions in helper.py.
I want to use spark-submit to submit a job but how do i specify helper.py?
Basically, how do i specify multiple files in spark?
Thanks

Re: Submitting multiple files pyspark

Posted by Andrew Or <an...@databricks.com>.

Hi Cheng,

You specify extra python files through --py-files. For example:

bin/spark-submit [your other options] --py-files helper.py main_app.py

-Andrew


2014-08-27 22:58 GMT-07:00 Chengi Liu <ch...@gmail.com>:

> Hi,
>   I have two files..
>
> main_app.py and helper.py
> main_app.py calls some functions in helper.py.
> I want to use spark-submit to submit a job but how do i specify helper.py?
> Basically, how do i specify multiple files in spark?
> Thanks
>