You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Soumya Simanta <so...@gmail.com> on 2014/02/16 23:33:25 UTC
Basic Spark terms - Slaves, workers, executors
Can someone explain what each of these terms means in terms of Spark? I've
a confusion between what's the difference between Slaves, workers,
executors.
My understanding is the slaves and workers are interchangeable ?
Thanks.
Re: Basic Spark terms - Slaves, workers, executors
Posted by Soumya Simanta <so...@gmail.com>.
Thanks Matei and Dachuan.
On Sun, Feb 16, 2014 at 6:03 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Hi Soumya,
>
> You can find an official description of the terms here:
> http://spark.incubator.apache.org/docs/latest/cluster-overview.html.
> "Slave" is sometimes used interchangeably for a worker, but most of the
> code and log messages uses the terms in that doc.
>
> Matei
>
> On Feb 16, 2014, at 2:44 PM, dachuan <hd...@gmail.com> wrote:
>
> I think so, slave = worker. one worker = multiple executors.
>
> One spark cluster consists of one master and multiple workers.
>
> One worker can have configured cores and memory.
>
> One worker's memory size is specified in conf/spark-env.sh, locally
> visible.
>
> One app = multiple executors
>
> One executor manages many cores (calculated in Master.schedule() function)
>
> One executor's memory size is specified in app's SparkConf.
>
>
> On Sun, Feb 16, 2014 at 5:33 PM, Soumya Simanta <so...@gmail.com>wrote:
>
>> Can someone explain what each of these terms means in terms of Spark?
>> I've a confusion between what's the difference between Slaves, workers,
>> executors.
>>
>> My understanding is the slaves and workers are interchangeable ?
>>
>> Thanks.
>>
>
>
>
> --
> Dachuan Huang
> Cellphone: 614-390-7234
> 2015 Neil Avenue
> Ohio State University
> Columbus, Ohio
> U.S.A.
> 43210
>
>
>
Re: Basic Spark terms - Slaves, workers, executors
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Soumya,
You can find an official description of the terms here: http://spark.incubator.apache.org/docs/latest/cluster-overview.html. “Slave” is sometimes used interchangeably for a worker, but most of the code and log messages uses the terms in that doc.
Matei
On Feb 16, 2014, at 2:44 PM, dachuan <hd...@gmail.com> wrote:
> I think so, slave = worker. one worker = multiple executors.
>
> One spark cluster consists of one master and multiple workers.
>
> One worker can have configured cores and memory.
>
> One worker's memory size is specified in conf/spark-env.sh, locally visible.
>
> One app = multiple executors
>
> One executor manages many cores (calculated in Master.schedule() function)
>
> One executor's memory size is specified in app's SparkConf.
>
>
> On Sun, Feb 16, 2014 at 5:33 PM, Soumya Simanta <so...@gmail.com> wrote:
> Can someone explain what each of these terms means in terms of Spark? I've a confusion between what's the difference between Slaves, workers, executors.
>
> My understanding is the slaves and workers are interchangeable ?
>
> Thanks.
>
>
>
> --
> Dachuan Huang
> Cellphone: 614-390-7234
> 2015 Neil Avenue
> Ohio State University
> Columbus, Ohio
> U.S.A.
> 43210
Re: Basic Spark terms - Slaves, workers, executors
Posted by dachuan <hd...@gmail.com>.
I think so, slave = worker. one worker = multiple executors.
One spark cluster consists of one master and multiple workers.
One worker can have configured cores and memory.
One worker's memory size is specified in conf/spark-env.sh, locally visible.
One app = multiple executors
One executor manages many cores (calculated in Master.schedule() function)
One executor's memory size is specified in app's SparkConf.
On Sun, Feb 16, 2014 at 5:33 PM, Soumya Simanta <so...@gmail.com>wrote:
> Can someone explain what each of these terms means in terms of Spark? I've
> a confusion between what's the difference between Slaves, workers,
> executors.
>
> My understanding is the slaves and workers are interchangeable ?
>
> Thanks.
>
--
Dachuan Huang
Cellphone: 614-390-7234
2015 Neil Avenue
Ohio State University
Columbus, Ohio
U.S.A.
43210