You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by poiuytrez <gu...@databerries.com> on 2015/03/17 11:29:35 UTC
Spark-submit and multiple files
Hello guys,
I am having a hard time to understand how spark-submit behave with multiple
files. I have created two code snippets. Each code snippet is composed of a
main.py and work.py. The code works if I paste work.py then main.py in a
pyspark shell. However both snippets do not work when using spark submit and
generate different errors.
Function add_1 definition outside
http://www.codeshare.io/4ao8B
https://justpaste.it/jzvj
Embedded add_1 function definition
http://www.codeshare.io/OQJxq
https://justpaste.it/jzvn
I am trying a way to make it work.
Thank you for your support.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark-submit and multiple files
Posted by Petar Zecevic <pe...@gmail.com>.
I tried your program in yarn-client mode and it worked with no
exception. This is the command I used:
spark-submit --master yarn-client --py-files work.py main.py
(Spark 1.2.1)
On 20.3.2015. 9:47, Guillaume Charhon wrote:
> Hi Davies,
>
> I am already using --py-files. The system does use the other file. The
> error I am getting is not trivial. Please check the error log.
>
>
>
> On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <davies@databricks.com
> <ma...@databricks.com>> wrote:
>
> You could submit additional Python source via --py-files , for
> example:
>
> $ bin/spark-submit --py-files work.py main.py
>
> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez
> <guillaume@databerries.com <ma...@databerries.com>> wrote:
> > Hello guys,
> >
> > I am having a hard time to understand how spark-submit behave
> with multiple
> > files. I have created two code snippets. Each code snippet is
> composed of a
> > main.py and work.py. The code works if I paste work.py then
> main.py in a
> > pyspark shell. However both snippets do not work when using
> spark submit and
> > generate different errors.
> >
> > Function add_1 definition outside
> > http://www.codeshare.io/4ao8B
> > https://justpaste.it/jzvj
> >
> > Embedded add_1 function definition
> > http://www.codeshare.io/OQJxq
> > https://justpaste.it/jzvn
> >
> > I am trying a way to make it work.
> >
> > Thank you for your support.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
> > Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> > For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
> >
>
>
Re: Spark-submit and multiple files
Posted by Davies Liu <da...@databricks.com>.
You MUST put --py-files BEFORE main.py, as mentioned in another threads.
On Fri, Mar 20, 2015 at 1:47 AM, Guillaume Charhon
<gu...@databerries.com> wrote:
> Hi Davies,
>
> I am already using --py-files. The system does use the other file. The error
> I am getting is not trivial. Please check the error log.
>
>
>
> On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <da...@databricks.com> wrote:
>>
>> You could submit additional Python source via --py-files , for example:
>>
>> $ bin/spark-submit --py-files work.py main.py
>>
>> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez <gu...@databerries.com>
>> wrote:
>> > Hello guys,
>> >
>> > I am having a hard time to understand how spark-submit behave with
>> > multiple
>> > files. I have created two code snippets. Each code snippet is composed
>> > of a
>> > main.py and work.py. The code works if I paste work.py then main.py in a
>> > pyspark shell. However both snippets do not work when using spark submit
>> > and
>> > generate different errors.
>> >
>> > Function add_1 definition outside
>> > http://www.codeshare.io/4ao8B
>> > https://justpaste.it/jzvj
>> >
>> > Embedded add_1 function definition
>> > http://www.codeshare.io/OQJxq
>> > https://justpaste.it/jzvn
>> >
>> > I am trying a way to make it work.
>> >
>> > Thank you for your support.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark-submit and multiple files
Posted by Guillaume Charhon <gu...@databerries.com>.
Hi Davies,
I am already using --py-files. The system does use the other file. The
error I am getting is not trivial. Please check the error log.
On Thu, Mar 19, 2015 at 8:03 PM, Davies Liu <da...@databricks.com> wrote:
> You could submit additional Python source via --py-files , for example:
>
> $ bin/spark-submit --py-files work.py main.py
>
> On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez <gu...@databerries.com>
> wrote:
> > Hello guys,
> >
> > I am having a hard time to understand how spark-submit behave with
> multiple
> > files. I have created two code snippets. Each code snippet is composed
> of a
> > main.py and work.py. The code works if I paste work.py then main.py in a
> > pyspark shell. However both snippets do not work when using spark submit
> and
> > generate different errors.
> >
> > Function add_1 definition outside
> > http://www.codeshare.io/4ao8B
> > https://justpaste.it/jzvj
> >
> > Embedded add_1 function definition
> > http://www.codeshare.io/OQJxq
> > https://justpaste.it/jzvn
> >
> > I am trying a way to make it work.
> >
> > Thank you for your support.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
Re: Spark-submit and multiple files
Posted by Davies Liu <da...@databricks.com>.
You could submit additional Python source via --py-files , for example:
$ bin/spark-submit --py-files work.py main.py
On Tue, Mar 17, 2015 at 3:29 AM, poiuytrez <gu...@databerries.com> wrote:
> Hello guys,
>
> I am having a hard time to understand how spark-submit behave with multiple
> files. I have created two code snippets. Each code snippet is composed of a
> main.py and work.py. The code works if I paste work.py then main.py in a
> pyspark shell. However both snippets do not work when using spark submit and
> generate different errors.
>
> Function add_1 definition outside
> http://www.codeshare.io/4ao8B
> https://justpaste.it/jzvj
>
> Embedded add_1 function definition
> http://www.codeshare.io/OQJxq
> https://justpaste.it/jzvn
>
> I am trying a way to make it work.
>
> Thank you for your support.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-submit-and-multiple-files-tp22097.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org