You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Julien Beaudan <jb...@stottlerhenke.com> on 2015/07/16 02:20:52 UTC

Spark cluster read local files

Hi all,

Is it possible to use Spark to assign each machine in a cluster the same 
task, but on files in each machine's local file system, and then have 
the results sent back to the driver program?

Thank you in advance!

Julien

Re: Spark cluster read local files

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Yes you can do that, just make sure you rsync the same file to the same
location on every machine.

Thanks
Best Regards

On Thu, Jul 16, 2015 at 5:50 AM, Julien Beaudan <jb...@stottlerhenke.com>
wrote:

> Hi all,
>
> Is it possible to use Spark to assign each machine in a cluster the same
> task, but on files in each machine's local file system, and then have the
> results sent back to the driver program?
>
> Thank you in advance!
>
> Julien
>
>