You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by srujana <sr...@persistent.co.in> on 2014/06/24 08:21:07 UTC

How data is distributed while processing in spark cluster?

Hi,

I am working on auto scaling spark cluster. I would like to know how master
distributes the data to the slaves for processing in detail.

Any information on this would be helpful.

Thanks,
Srujana




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-data-is-distributed-while-processing-in-spark-cluster-tp8160.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How data is distributed while processing in spark cluster?

Posted by srujana <sr...@persistent.co.in>.

Thanks for the response.

I would also like to know, What happens if a slave node is removed while it
is processing some data. Does master send that data for
re-processing/resume-process to other slave nodes ? And does it happen with
the help of HDFS?

Thanks,
Srujana



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-data-is-distributed-while-processing-in-spark-cluster-tp8160p8235.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: How data is distributed while processing in spark cluster?

Posted by Mayur Rustagi <ma...@gmail.com>.

Using HDFS locality. The workers call for the data from hdfs/queue etc.
Unless you use parallelize then its sent from driver (typically on the
master) to the worker nodes.

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Tue, Jun 24, 2014 at 11:51 AM, srujana <sr...@persistent.co.in>
wrote:

> Hi,
>
> I am working on auto scaling spark cluster. I would like to know how master
> distributes the data to the slaves for processing in detail.
>
> Any information on this would be helpful.
>
> Thanks,
> Srujana
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-data-is-distributed-while-processing-in-spark-cluster-tp8160.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>