You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Benjamin Zaitlen <qu...@gmail.com> on 2016/04/14 23:14:45 UTC

Error with --files

Hi All,

I'm trying to use the --files option with yarn:

spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
> /home/ubuntu/localtest.txt#appSees.txt


I never see the file in HDFS or in the yarn containers.  Am I doing
something incorrect ?

I'm running spark 1.6.0


Thanks,
--Ben

Re: Error with --files

Posted by Ted Yu <yu...@gmail.com>.
bq. localtest.txt#appSees.txt

Which file did you want to pass ?

Thanks

On Thu, Apr 14, 2016 at 2:14 PM, Benjamin Zaitlen <qu...@gmail.com>
wrote:

> Hi All,
>
> I'm trying to use the --files option with yarn:
>
> spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
>> /home/ubuntu/localtest.txt#appSees.txt
>
>
> I never see the file in HDFS or in the yarn containers.  Am I doing
> something incorrect ?
>
> I'm running spark 1.6.0
>
>
> Thanks,
> --Ben
>

Re: Error with --files

Posted by Benjamin Zaitlen <qu...@gmail.com>.
That fixed it!

Thank you!

--Ben

On Thu, Apr 14, 2016 at 5:53 PM, Marcelo Vanzin <va...@cloudera.com> wrote:

> On Thu, Apr 14, 2016 at 2:14 PM, Benjamin Zaitlen <qu...@gmail.com>
> wrote:
> >> spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
> >> /home/ubuntu/localtest.txt#appSees.txt
>
> --files should come before the path to your python script. Otherwise
> it's just passed as arguments to your script when it's run.
>
> --
> Marcelo
>

Re: Error with --files

Posted by Marcelo Vanzin <va...@cloudera.com>.
On Thu, Apr 14, 2016 at 2:14 PM, Benjamin Zaitlen <qu...@gmail.com> wrote:
>> spark-submit --master yarn-cluster /home/ubuntu/test_spark.py --files
>> /home/ubuntu/localtest.txt#appSees.txt

--files should come before the path to your python script. Otherwise
it's just passed as arguments to your script when it's run.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org