You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by aniket ray <an...@gmail.com> on 2010/09/13 10:47:03 UTC
Capacity scheduler and map/reduce jobs
Hi,
I've been trying to run some map/reduce jobs in parallel using the capacity
scheduler on hadoop 0.20.2.
I am running 2 kinds of jobs and each have been assigned a separate queue
with separate batches of data.
These jobs are independent and can be run in parallel.
I see that the reduces of Queue 1 don't start till maps of Job 2 are over
(even though maps of Job 1 are complete). I am not able to understand
this behavior and feel that this may be a configuration issue that I am
missing. Since they are independent tasks and capacity is free, shouldn't
reduce tasks of Queue kick in?
Thanks,
Aniket Ray
Re: Capacity scheduler and map/reduce jobs
Posted by aniket ray <an...@gmail.com>.
On Tue, Sep 14, 2010 at 1:45 AM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
> Moving mapreduce-user@, bcc common-user@. Please use the appropriate
> project lists for discussions.
>
>
> On Sep 13, 2010, at 1:47 AM, aniket ray wrote:
>
> I see that the reduces of Queue 1 don't start till maps of Job 2 are over
>> (even though maps of Job 1 are complete). I am not able to understand
>> this behavior and feel that this may be a configuration issue that I am
>> missing. Since they are independent tasks and capacity is free, shouldn't
>> reduce tasks of Queue kick in?
>>
>>
> This shouldn't happen in the CS. Maybe you are hitting 'slowstart' for the
> reduces of Job1?
>
> From src/mapred/mapred-default.xml:
> <property>
> <name>mapred.reduce.slowstart.completed.maps</name>
> <value>0.05</value>
> <description>Fraction of the number of maps in the job which should be
> complete before reduces are scheduled for the job.
> </description>
> </property>
>
>
> I've been trying to run some map/reduce jobs in parallel using the
>> capacity
>> scheduler on hadoop 0.20.2.
>>
>
> The CS in 0.20.2 is quite dated, you might want to use the Yahoo! GitHub (
> http://github.com/yahoo/hadoop-common) for the latest version of the CS.
>
> Thanks. I used this version of hadoop with its capacity scheduler and
things work great. I saw a major performance boost.
There is one hit I'm not able to figure out using this version though. Jobs
stay in the queue in a "waiting state" (According to console dump and the
web UI) for much longer, even if no other jobs are running in the queue.
I'm currently working to get the Yahoo codebase released as an Apache
> Release (maybe hadoop-0.20-security), once we get that done you should be
> able to use the latest CapacityScheduler via an Apache Release.
>
> Arun
>
Re: Capacity scheduler and map/reduce jobs
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Moving mapreduce-user@, bcc common-user@. Please use the appropriate
project lists for discussions.
On Sep 13, 2010, at 1:47 AM, aniket ray wrote:
> I see that the reduces of Queue 1 don't start till maps of Job 2 are
> over
> (even though maps of Job 1 are complete). I am not able to understand
> this behavior and feel that this may be a configuration issue that I
> am
> missing. Since they are independent tasks and capacity is free,
> shouldn't
> reduce tasks of Queue kick in?
>
This shouldn't happen in the CS. Maybe you are hitting 'slowstart' for
the reduces of Job1?
From src/mapred/mapred-default.xml:
<property>
<name>mapred.reduce.slowstart.completed.maps</name>
<value>0.05</value>
<description>Fraction of the number of maps in the job which should
be
complete before reduces are scheduled for the job.
</description>
</property>
> I've been trying to run some map/reduce jobs in parallel using the
> capacity
> scheduler on hadoop 0.20.2.
The CS in 0.20.2 is quite dated, you might want to use the Yahoo!
GitHub (http://github.com/yahoo/hadoop-common) for the latest version
of the CS.
I'm currently working to get the Yahoo codebase released as an Apache
Release (maybe hadoop-0.20-security), once we get that done you should
be able to use the latest CapacityScheduler via an Apache Release.
Arun
Re: Capacity scheduler and map/reduce jobs
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Moving mapreduce-user@, bcc common-user@. Please use the appropriate
project lists for discussions.
On Sep 13, 2010, at 1:47 AM, aniket ray wrote:
> I see that the reduces of Queue 1 don't start till maps of Job 2 are
> over
> (even though maps of Job 1 are complete). I am not able to understand
> this behavior and feel that this may be a configuration issue that I
> am
> missing. Since they are independent tasks and capacity is free,
> shouldn't
> reduce tasks of Queue kick in?
>
This shouldn't happen in the CS. Maybe you are hitting 'slowstart' for
the reduces of Job1?
From src/mapred/mapred-default.xml:
<property>
<name>mapred.reduce.slowstart.completed.maps</name>
<value>0.05</value>
<description>Fraction of the number of maps in the job which should
be
complete before reduces are scheduled for the job.
</description>
</property>
> I've been trying to run some map/reduce jobs in parallel using the
> capacity
> scheduler on hadoop 0.20.2.
The CS in 0.20.2 is quite dated, you might want to use the Yahoo!
GitHub (http://github.com/yahoo/hadoop-common) for the latest version
of the CS.
I'm currently working to get the Yahoo codebase released as an Apache
Release (maybe hadoop-0.20-security), once we get that done you should
be able to use the latest CapacityScheduler via an Apache Release.
Arun