You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "W.P. McNeill" <bi...@gmail.com> on 2011/05/31 20:50:37 UTC
Why don't my jobs get preempted?
I'm launching long-running tasks on a cluster running the Fair Scheduler.
As I understand it, the Fair Scheduler is preemptive. What I expect to see
is that my long-running jobs sometimes get killed to make room for other
people's jobs. This never happens instead my long-running jobs hog mapper
and reducer slots and starve other people out.
Am I misunderstanding how the Fair Scheduler works?
Re: Why don't my jobs get preempted?
Posted by Matei Zaharia <ma...@eecs.berkeley.edu>.
Sorry, I meant 0.21+, not 0.20+ for the Apache releases.
Matei
On May 31, 2011, at 4:05 PM, Matei Zaharia wrote:
> Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop that have applied that patch, such as Cloudera's distribution. If you are running one of these, check out http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for information on how to enable preemption.
>
> Matei
>
> On May 31, 2011, at 12:20 PM, Edward Capriolo wrote:
>
>> On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <bi...@gmail.com> wrote:
>>
>>> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>>> As I understand it, the Fair Scheduler is preemptive. What I expect to see
>>> is that my long-running jobs sometimes get killed to make room for other
>>> people's jobs. This never happens instead my long-running jobs hog mapper
>>> and reducer slots and starve other people out.
>>>
>>> Am I misunderstanding how the Fair Scheduler works?
>>>
>>
>> Try adding
>>
>> <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
>> <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>
>>
>> To one of your pools and see if that pool pre-empts other pools
>
Re: Why don't my jobs get preempted?
Posted by Matei Zaharia <ma...@eecs.berkeley.edu>.
Preemption is only available in Hadoop 0.20+ or in distributions of Hadoop that have applied that patch, such as Cloudera's distribution. If you are running one of these, check out http://hadoop.apache.org/mapreduce/docs/r0.21.0/fair_scheduler.html for information on how to enable preemption.
Matei
On May 31, 2011, at 12:20 PM, Edward Capriolo wrote:
> On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <bi...@gmail.com> wrote:
>
>> I'm launching long-running tasks on a cluster running the Fair Scheduler.
>> As I understand it, the Fair Scheduler is preemptive. What I expect to see
>> is that my long-running jobs sometimes get killed to make room for other
>> people's jobs. This never happens instead my long-running jobs hog mapper
>> and reducer slots and starve other people out.
>>
>> Am I misunderstanding how the Fair Scheduler works?
>>
>
> Try adding
>
> <minSharePreemptionTimeout>120</minSharePreemptionTimeout>
> <fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>
>
> To one of your pools and see if that pool pre-empts other pools
Re: Why don't my jobs get preempted?
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, May 31, 2011 at 2:50 PM, W.P. McNeill <bi...@gmail.com> wrote:
> I'm launching long-running tasks on a cluster running the Fair Scheduler.
> As I understand it, the Fair Scheduler is preemptive. What I expect to see
> is that my long-running jobs sometimes get killed to make room for other
> people's jobs. This never happens instead my long-running jobs hog mapper
> and reducer slots and starve other people out.
>
> Am I misunderstanding how the Fair Scheduler works?
>
Try adding
<minSharePreemptionTimeout>120</minSharePreemptionTimeout>
<fairSharePreemptionTimeout>180</fairSharePreemptionTimeout>
To one of your pools and see if that pool pre-empts other pools