You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Benjamin Zaitlen <qu...@gmail.com> on 2016/04/14 23:14:45 UTC
Error with --files
Hi All,
I'm trying to use the --files option with yarn:
spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
> /home/ubuntu/localtest.txt#appSees.txt
I never see the file in HDFS or in the yarn containers. Am I doing
something incorrect ?
I'm running spark 1.6.0
Thanks,
--Ben
Re: Error with --files
Posted by Ted Yu <yu...@gmail.com>.
bq. localtest.txt#appSees.txt
Which file did you want to pass ?
Thanks
On Thu, Apr 14, 2016 at 2:14 PM, Benjamin Zaitlen <qu...@gmail.com>
wrote:
> Hi All,
>
> I'm trying to use the --files option with yarn:
>
> spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
>> /home/ubuntu/localtest.txt#appSees.txt
>
>
> I never see the file in HDFS or in the yarn containers. Am I doing
> something incorrect ?
>
> I'm running spark 1.6.0
>
>
> Thanks,
> --Ben
>
Re: Error with --files
Posted by Benjamin Zaitlen <qu...@gmail.com>.
That fixed it!
Thank you!
--Ben
On Thu, Apr 14, 2016 at 5:53 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> On Thu, Apr 14, 2016 at 2:14 PM, Benjamin Zaitlen <qu...@gmail.com>
> wrote:
> >> spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
> >> /home/ubuntu/localtest.txt#appSees.txt
>
> --files should come before the path to your python script. Otherwise
> it's just passed as arguments to your script when it's run.
>
> --
> Marcelo
>
Re: Error with --files
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Thu, Apr 14, 2016 at 2:14 PM, Benjamin Zaitlen <qu...@gmail.com> wrote:
>> spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
>> /home/ubuntu/localtest.txt#appSees.txt
--files should come before the path to your python script. Otherwise
it's just passed as arguments to your script when it's run.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org