Resource allocations

[rapelly kartheek <ka...@gmail.com>]

Hi,

I am trying to understand how the resource allocation happens in spark. I
understand the resourceOffer method in taskScheduler. This method takes
care of locality factor while allocating the resources. This resourceOffer
method gets invoked by the corresponding cluster manager.

I am working on stand-alone spark cluster. But I am not able to locate
starting point of the resource allocation. I want to understand from end to
end how exactly resource allocations happens, given a new Application to
the Spark. Also, apart from mesos, there is a folder called Local in the
Scheduler. Can someone tell me which all files should I look into to
understand the resource allocation in stand-alone spark cluster and what is
this "Local" for?

Thanks in advance!!
Karthik.

Re: Resource allocations

[Kay Ousterhout <ke...@eecs.berkeley.edu>]

Hi Karthik,

The resourceOffer() method is invoked from a class implementing the
SchedulerBackend interface; in the case of a standalone cluster, it's
invoked from a CoarseGrainedSchedulerBackend (in the makeOffers() method).
 If you look in TaskSchedulerImpl.submitTasks(), it calls
backend.reviveOffers() at the end -- which signals to the SchedulerBackend
that it should offer some resources.  This somewhat confusing interface
exists for historical reasons: Spark originally used only Mesos for
scheduling, which relies on resource offers for scheduling, and so
standalone mode was done in a similar way to maximize code re-use.

The scheduler/local folder contains code specific to running in local mode.
 With local mode, all code runs in a single JVM (as opposed to having one
JVM for the driver, one for the cluster master, and one for each worker).
 This mode is typically used for testing.

-Kay

On Wed, Jul 16, 2014 at 8:52 AM, rapelly kartheek <ka...@gmail.com>
wrote:

> Hi,
>
> I am trying to understand how the resource allocation happens in spark. I
> understand the resourceOffer method in taskScheduler. This method takes
> care of locality factor while allocating the resources. This resourceOffer
> method gets invoked by the corresponding cluster manager.
>
> I am working on stand-alone spark cluster. But I am not able to locate
> starting point of the resource allocation. I want to understand from end to
> end how exactly resource allocations happens, given a new Application to
> the Spark. Also, apart from mesos, there is a folder called Local in the
> Scheduler. Can someone tell me which all files should I look into to
> understand the resource allocation in stand-alone spark cluster and what is
> this "Local" for?
>
> Thanks in advance!!
> Karthik.
>