You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anh Hong <ho...@yahoo.com.INVALID> on 2015/07/29 06:02:54 UTC

Does spark-submit support file transfering from local to cluster?

Hi, 
   I'm using spark-submit cluster mode to submit a job from local to spark cluster. There are input files, output files, and job log files that I need to transfer in and out between local machine and spark cluster.Any recommendation methods to use file transferring. Is there any future plan that spark will support file transferring from cluster to local and vice versa. Any help is appreciated.
Thanks.

Re: Does spark-submit support file transfering from local to cluster?

Posted by Jeff Zhang <zj...@gmail.com>.
Spark is computing engine, I don't think it is suitable for spark to do the
input large file transferring.



On Thu, Jul 30, 2015 at 4:18 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:

> You can achieve this with a Network File System, create an NFS on the
> cluster and mount it to your local machine.
>
> Thanks
> Best Regards
>
> On Wed, Jul 29, 2015 at 9:32 AM, Anh Hong <ho...@yahoo.com.invalid>
> wrote:
>
>> Hi,
>>
>>   I'm using spark-submit cluster mode to submit a job from local to
>> spark cluster. There are input files, output files, and job log files that
>> I need to transfer in and out between local machine and spark cluster.
>> Any recommendation methods to use file transferring. Is there any future
>> plan that spark will support file transferring from cluster to local and
>> vice versa.
>> Any help is appreciated.
>>
>> Thanks.
>>
>
>


-- 
Best Regards

Jeff Zhang

Re: Does spark-submit support file transfering from local to cluster?

Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You can achieve this with a Network File System, create an NFS on the
cluster and mount it to your local machine.

Thanks
Best Regards

On Wed, Jul 29, 2015 at 9:32 AM, Anh Hong <ho...@yahoo.com.invalid>
wrote:

> Hi,
>
>   I'm using spark-submit cluster mode to submit a job from local to spark
> cluster. There are input files, output files, and job log files that I need
> to transfer in and out between local machine and spark cluster.
> Any recommendation methods to use file transferring. Is there any future
> plan that spark will support file transferring from cluster to local and
> vice versa.
> Any help is appreciated.
>
> Thanks.
>