Executors on multiple nodes

[Mohit Anchlia <mo...@gmail.com>]

I am running on Yarn and do have a question on how spark runs executors on
different data nodes. Is that primarily decided based on number of
receivers?

What do I need to do to ensure that multiple nodes are being used for data
processing?

Re: Executors on multiple nodes

[Sandy Ryza <sa...@cloudera.com>]

Hi Mohit,

It depends on whether dynamic allocation is turned on.  If not, the number
of executors is specified by the user with the --num-executors option.  If
dynamic allocation is turned on, refer to the doc for details:
https://spark.apache.org/docs/1.4.0/job-scheduling.html#dynamic-resource-allocation
.

-Sandy


On Sat, Aug 15, 2015 at 6:40 AM, Mohit Anchlia <mo...@gmail.com>
wrote:

> I am running on Yarn and do have a question on how spark runs executors on
> different data nodes. Is that primarily decided based on number of
> receivers?
>
> What do I need to do to ensure that multiple nodes are being used for data
> processing?
>