You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sai Prasanna <an...@gmail.com> on 2014/01/24 10:15:47 UTC
Lazy Scheduling
Hi, please clarify this,
In lazy scheduling that spark had implemented, it is given that tasks are
sorted when a slot becomes free at a node and the job with least tasks is
scheduled first or made to wait maximum till time D if locality of data is
not there. My question is, doesnt this result in jobs with large number of
tasks getting starved?...If not, then how?
--
*Sai Prasanna. AN*
*II M.Tech (CS), SSSIHL*
*Entire water in the ocean can never sink a ship, Unless it gets inside.All
the pressures of life can never hurt you, Unless you let them in.*
Re: Lazy Scheduling
Posted by Tathagata Das <ta...@gmail.com>.
You can enable fair sharing of resources between jobs in Spark. See
http://spark.incubator.apache.org/docs/latest/job-scheduling.html
On Sun, Jan 26, 2014 at 8:25 PM, Sai Prasanna <an...@gmail.com>wrote:
> Please someone throw some light into this.
>
> In lazy scheduling that spark had implemented, it is given that tasks are
> sorted when a slot becomes free at a node and the job with least tasks is
> scheduled first or made to wait maximum till time D if locality of data is
> not there. My question is, doesnt this result in jobs with large number of
> tasks getting starved?...If not, then how?
>
>
> On Fri, Jan 24, 2014 at 2:45 PM, Sai Prasanna <an...@gmail.com>wrote:
>
>> Hi, please clarify this,
>>
>> In lazy scheduling that spark had implemented, it is given that tasks are
>> sorted when a slot becomes free at a node and the job with least tasks is
>> scheduled first or made to wait maximum till time D if locality of data is
>> not there. My question is, doesnt this result in jobs with large number of
>> tasks getting starved?...If not, then how?
>>
>>
>>
>>
>> --
>> *Sai Prasanna. AN*
>> *II M.Tech (CS), SSSIHL*
>>
>>
>> * Entire water in the ocean can never sink a ship, Unless it gets inside.
>> All the pressures of life can never hurt you, Unless you let them in.*
>>
>
>
>
> --
> *Sai Prasanna. AN*
> *II M.Tech (CS), SSSIHL*
>
>
> *Entire water in the ocean can never sink a ship, Unless it gets inside.
> All the pressures of life can never hurt you, Unless you let them in.*
>
Re: Lazy Scheduling
Posted by Sai Prasanna <an...@gmail.com>.
Please someone throw some light into this.
In lazy scheduling that spark had implemented, it is given that tasks are
sorted when a slot becomes free at a node and the job with least tasks is
scheduled first or made to wait maximum till time D if locality of data is
not there. My question is, doesnt this result in jobs with large number of
tasks getting starved?...If not, then how?
On Fri, Jan 24, 2014 at 2:45 PM, Sai Prasanna <an...@gmail.com>wrote:
> Hi, please clarify this,
>
> In lazy scheduling that spark had implemented, it is given that tasks are
> sorted when a slot becomes free at a node and the job with least tasks is
> scheduled first or made to wait maximum till time D if locality of data is
> not there. My question is, doesnt this result in jobs with large number of
> tasks getting starved?...If not, then how?
>
>
>
>
> --
> *Sai Prasanna. AN*
> *II M.Tech (CS), SSSIHL*
>
>
> *Entire water in the ocean can never sink a ship, Unless it gets inside.
> All the pressures of life can never hurt you, Unless you let them in.*
>
--
*Sai Prasanna. AN*
*II M.Tech (CS), SSSIHL*
*Entire water in the ocean can never sink a ship, Unless it gets inside.All
the pressures of life can never hurt you, Unless you let them in.*