You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Rasit OZDAS <ra...@gmail.com> on 2009/03/01 18:24:06 UTC
Reduce doesn't start until map finishes
Hi!
Whatever code I run on hadoop, reduce starts a few seconds after map
finishes.
And worse, when I run 10 jobs parallely (using threads and sending one after
another)
all maps finish sequentially, then after 8-10 seconds reduces start.
I use reducer also as combiner, my cluster has 6 machines, namenode and
jobtracker run also as slaves.
There were 44 maps and 6 reduces in the last example, I never tried a bigger
job.
What can the problem be? I've read somewhere that this is not the normal
behaviour.
Replication factor is 3.
Thank you in advance for any pointers.
Rasit
Re: Reduce doesn't start until map finishes
Posted by Owen O'Malley <ow...@gmail.com>.
What happened is that we added fast start (HADOOP-3136), which
launches more than one task per a heartbeat. Previously, if you maps
didn't take very long, they finished before the heartbeat and the task
tracker was assigned a new map task. A side effect was that no reduce
tasks were launched until the maps were complete, which prevents the
shuffle from overlapping with the maps.
-- Owen
Re: Reduce doesn't start until map finishes
Posted by Rasit OZDAS <ra...@gmail.com>.
Just to inform, we installed v.0.21.0-dev and there is no such issue now.
2009/3/6 Rasit OZDAS <ra...@gmail.com>
> So, is there currently no solution to my problem?
> Should I live with it? Or do we have to have a JIRA for this?
> What do you think?
>
>
> 2009/3/4 Nick Cen <ce...@gmail.com>
>
> Thanks, about the "Secondary Sort", can you provide some example. What does
>> the intermediate keys stands for?
>>
>> Assume I have two mapper, m1 and m2. The output of m1 is (k1,v1),(k2,v2)
>> and
>> the output of m2 is (k1,v3),(k2,v4). Assume k1 and k2 belongs to the same
>> partition and k1 < k2, so i think the order inside reducer maybe:
>> (k1,v1)
>> (k1,v3)
>> (k2,v2)
>> (k2,v4)
>>
>> can the Secondary Sort change this order?
>>
>>
>>
>> 2009/3/4 Chris Douglas <ch...@yahoo-inc.com>
>>
>> > The output of each map is sorted by partition and by key within that
>> > partition. The reduce merges sorted map output assigned to its partition
>> > into the reduce. The following may be helpful:
>> >
>> > http://hadoop.apache.org/core/docs/current/mapred_tutorial.html
>> >
>> > If your job requires total order, consider
>> > o.a.h.mapred.lib.TotalOrderPartitioner. -C
>> >
>> >
>> > On Mar 3, 2009, at 7:24 PM, Nick Cen wrote:
>> >
>> > can you provide more info about sortint? The sort is happend on the
>> whole
>> >> data set, or just on the specified partion?
>> >>
>> >> 2009/3/4 Mikhail Yakshin <gr...@gmail.com>
>> >>
>> >> On Wed, Mar 4, 2009 at 2:09 AM, Chris Douglas wrote:
>> >>>
>> >>>> This is normal behavior. The Reducer is guaranteed to receive all the
>> >>>> results for its partition in sorted order. No reduce can start until
>> all
>> >>>>
>> >>> the
>> >>>
>> >>>> maps are completed, since any running map could emit a result that
>> would
>> >>>> violate the order for the results it currently has. -C
>> >>>>
>> >>>
>> >>> _Reducers_ usually start almost immediately and start downloading data
>> >>> emitted by mappers as they go. This is their first phase. Their second
>> >>> phase can start only after completion of all mappers. In their second
>> >>> phase, they're sorting received data, and in their third phase they're
>> >>> doing real reduction.
>> >>>
>> >>> --
>> >>> WBR, Mikhail Yakshin
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> http://daily.appspot.com/food/
>> >>
>> >
>> >
>>
>>
>> --
>> http://daily.appspot.com/food/
>>
>
>
>
> --
> M. Raşit ÖZDAŞ
>
--
M. Raşit ÖZDAŞ
Re: Reduce doesn't start until map finishes
Posted by Rasit OZDAS <ra...@gmail.com>.
So, is there currently no solution to my problem?
Should I live with it? Or do we have to have a JIRA for this?
What do you think?
2009/3/4 Nick Cen <ce...@gmail.com>
> Thanks, about the "Secondary Sort", can you provide some example. What does
> the intermediate keys stands for?
>
> Assume I have two mapper, m1 and m2. The output of m1 is (k1,v1),(k2,v2)
> and
> the output of m2 is (k1,v3),(k2,v4). Assume k1 and k2 belongs to the same
> partition and k1 < k2, so i think the order inside reducer maybe:
> (k1,v1)
> (k1,v3)
> (k2,v2)
> (k2,v4)
>
> can the Secondary Sort change this order?
>
>
>
> 2009/3/4 Chris Douglas <ch...@yahoo-inc.com>
>
> > The output of each map is sorted by partition and by key within that
> > partition. The reduce merges sorted map output assigned to its partition
> > into the reduce. The following may be helpful:
> >
> > http://hadoop.apache.org/core/docs/current/mapred_tutorial.html
> >
> > If your job requires total order, consider
> > o.a.h.mapred.lib.TotalOrderPartitioner. -C
> >
> >
> > On Mar 3, 2009, at 7:24 PM, Nick Cen wrote:
> >
> > can you provide more info about sortint? The sort is happend on the
> whole
> >> data set, or just on the specified partion?
> >>
> >> 2009/3/4 Mikhail Yakshin <gr...@gmail.com>
> >>
> >> On Wed, Mar 4, 2009 at 2:09 AM, Chris Douglas wrote:
> >>>
> >>>> This is normal behavior. The Reducer is guaranteed to receive all the
> >>>> results for its partition in sorted order. No reduce can start until
> all
> >>>>
> >>> the
> >>>
> >>>> maps are completed, since any running map could emit a result that
> would
> >>>> violate the order for the results it currently has. -C
> >>>>
> >>>
> >>> _Reducers_ usually start almost immediately and start downloading data
> >>> emitted by mappers as they go. This is their first phase. Their second
> >>> phase can start only after completion of all mappers. In their second
> >>> phase, they're sorting received data, and in their third phase they're
> >>> doing real reduction.
> >>>
> >>> --
> >>> WBR, Mikhail Yakshin
> >>>
> >>>
> >>
> >>
> >> --
> >> http://daily.appspot.com/food/
> >>
> >
> >
>
>
> --
> http://daily.appspot.com/food/
>
--
M. Raşit ÖZDAŞ
Re: Reduce doesn't start until map finishes
Posted by Nick Cen <ce...@gmail.com>.
Thanks, about the "Secondary Sort", can you provide some example. What does
the intermediate keys stands for?
Assume I have two mapper, m1 and m2. The output of m1 is (k1,v1),(k2,v2) and
the output of m2 is (k1,v3),(k2,v4). Assume k1 and k2 belongs to the same
partition and k1 < k2, so i think the order inside reducer maybe:
(k1,v1)
(k1,v3)
(k2,v2)
(k2,v4)
can the Secondary Sort change this order?
2009/3/4 Chris Douglas <ch...@yahoo-inc.com>
> The output of each map is sorted by partition and by key within that
> partition. The reduce merges sorted map output assigned to its partition
> into the reduce. The following may be helpful:
>
> http://hadoop.apache.org/core/docs/current/mapred_tutorial.html
>
> If your job requires total order, consider
> o.a.h.mapred.lib.TotalOrderPartitioner. -C
>
>
> On Mar 3, 2009, at 7:24 PM, Nick Cen wrote:
>
> can you provide more info about sortint? The sort is happend on the whole
>> data set, or just on the specified partion?
>>
>> 2009/3/4 Mikhail Yakshin <gr...@gmail.com>
>>
>> On Wed, Mar 4, 2009 at 2:09 AM, Chris Douglas wrote:
>>>
>>>> This is normal behavior. The Reducer is guaranteed to receive all the
>>>> results for its partition in sorted order. No reduce can start until all
>>>>
>>> the
>>>
>>>> maps are completed, since any running map could emit a result that would
>>>> violate the order for the results it currently has. -C
>>>>
>>>
>>> _Reducers_ usually start almost immediately and start downloading data
>>> emitted by mappers as they go. This is their first phase. Their second
>>> phase can start only after completion of all mappers. In their second
>>> phase, they're sorting received data, and in their third phase they're
>>> doing real reduction.
>>>
>>> --
>>> WBR, Mikhail Yakshin
>>>
>>>
>>
>>
>> --
>> http://daily.appspot.com/food/
>>
>
>
--
http://daily.appspot.com/food/
Re: Reduce doesn't start until map finishes
Posted by Chris Douglas <ch...@yahoo-inc.com>.
The output of each map is sorted by partition and by key within that
partition. The reduce merges sorted map output assigned to its
partition into the reduce. The following may be helpful:
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html
If your job requires total order, consider
o.a.h.mapred.lib.TotalOrderPartitioner. -C
On Mar 3, 2009, at 7:24 PM, Nick Cen wrote:
> can you provide more info about sortint? The sort is happend on the
> whole
> data set, or just on the specified partion?
>
> 2009/3/4 Mikhail Yakshin <gr...@gmail.com>
>
>> On Wed, Mar 4, 2009 at 2:09 AM, Chris Douglas wrote:
>>> This is normal behavior. The Reducer is guaranteed to receive all
>>> the
>>> results for its partition in sorted order. No reduce can start
>>> until all
>> the
>>> maps are completed, since any running map could emit a result that
>>> would
>>> violate the order for the results it currently has. -C
>>
>> _Reducers_ usually start almost immediately and start downloading
>> data
>> emitted by mappers as they go. This is their first phase. Their
>> second
>> phase can start only after completion of all mappers. In their second
>> phase, they're sorting received data, and in their third phase
>> they're
>> doing real reduction.
>>
>> --
>> WBR, Mikhail Yakshin
>>
>
>
>
> --
> http://daily.appspot.com/food/
Re: Reduce doesn't start until map finishes
Posted by Nick Cen <ce...@gmail.com>.
can you provide more info about sortint? The sort is happend on the whole
data set, or just on the specified partion?
2009/3/4 Mikhail Yakshin <gr...@gmail.com>
> On Wed, Mar 4, 2009 at 2:09 AM, Chris Douglas wrote:
> > This is normal behavior. The Reducer is guaranteed to receive all the
> > results for its partition in sorted order. No reduce can start until all
> the
> > maps are completed, since any running map could emit a result that would
> > violate the order for the results it currently has. -C
>
> _Reducers_ usually start almost immediately and start downloading data
> emitted by mappers as they go. This is their first phase. Their second
> phase can start only after completion of all mappers. In their second
> phase, they're sorting received data, and in their third phase they're
> doing real reduction.
>
> --
> WBR, Mikhail Yakshin
>
--
http://daily.appspot.com/food/
Re: Reduce doesn't start until map finishes
Posted by Mikhail Yakshin <gr...@gmail.com>.
On Wed, Mar 4, 2009 at 2:09 AM, Chris Douglas wrote:
> This is normal behavior. The Reducer is guaranteed to receive all the
> results for its partition in sorted order. No reduce can start until all the
> maps are completed, since any running map could emit a result that would
> violate the order for the results it currently has. -C
_Reducers_ usually start almost immediately and start downloading data
emitted by mappers as they go. This is their first phase. Their second
phase can start only after completion of all mappers. In their second
phase, they're sorting received data, and in their third phase they're
doing real reduction.
--
WBR, Mikhail Yakshin
Re: Reduce doesn't start until map finishes
Posted by Chris Douglas <ch...@yahoo-inc.com>.
This is normal behavior. The Reducer is guaranteed to receive all the
results for its partition in sorted order. No reduce can start until
all the maps are completed, since any running map could emit a result
that would violate the order for the results it currently has. -C
On Mar 1, 2009, at 9:24 AM, Rasit OZDAS wrote:
> Hi!
>
> Whatever code I run on hadoop, reduce starts a few seconds after map
> finishes.
> And worse, when I run 10 jobs parallely (using threads and sending
> one after
> another)
> all maps finish sequentially, then after 8-10 seconds reduces start.
> I use reducer also as combiner, my cluster has 6 machines, namenode
> and
> jobtracker run also as slaves.
> There were 44 maps and 6 reduces in the last example, I never tried
> a bigger
> job.
>
> What can the problem be? I've read somewhere that this is not the
> normal
> behaviour.
> Replication factor is 3.
> Thank you in advance for any pointers.
>
> Rasit
Re: Reduce doesn't start until map finishes
Posted by Rasit OZDAS <ra...@gmail.com>.
Strange, that I've last night tried 10000 input files (maps), waiting time
after maps increases (probably linearly)
2009/3/2 Rasit OZDAS <ra...@gmail.com>
> I have 6 reducers, Nick, still no luck..
>
> 2009/3/2 Nick Cen <ce...@gmail.com>
>
> how many reducer do you have? You should make this value larger then 1 to
>> make mapper and reducer run concurrently. You can set this value from
>> JobConf.*setNumReduceTasks*().
>>
>>
>> 2009/3/2 Rasit OZDAS <ra...@gmail.com>
>>
>> > Hi!
>> >
>> > Whatever code I run on hadoop, reduce starts a few seconds after map
>> > finishes.
>> > And worse, when I run 10 jobs parallely (using threads and sending one
>> > after
>> > another)
>> > all maps finish sequentially, then after 8-10 seconds reduces start.
>> > I use reducer also as combiner, my cluster has 6 machines, namenode and
>> > jobtracker run also as slaves.
>> > There were 44 maps and 6 reduces in the last example, I never tried a
>> > bigger
>> > job.
>> >
>> > What can the problem be? I've read somewhere that this is not the normal
>> > behaviour.
>> > Replication factor is 3.
>> > Thank you in advance for any pointers.
>> >
>> > Rasit
>> >
>>
>>
>>
>> --
>> http://daily.appspot.com/food/
>>
>
>
>
> --
> M. Raşit ÖZDAŞ
>
--
M. Raşit ÖZDAŞ
Re: Reduce doesn't start until map finishes
Posted by Rasit OZDAS <ra...@gmail.com>.
I have 6 reducers, Nick, still no luck..
2009/3/2 Nick Cen <ce...@gmail.com>
> how many reducer do you have? You should make this value larger then 1 to
> make mapper and reducer run concurrently. You can set this value from
> JobConf.*setNumReduceTasks*().
>
>
> 2009/3/2 Rasit OZDAS <ra...@gmail.com>
>
> > Hi!
> >
> > Whatever code I run on hadoop, reduce starts a few seconds after map
> > finishes.
> > And worse, when I run 10 jobs parallely (using threads and sending one
> > after
> > another)
> > all maps finish sequentially, then after 8-10 seconds reduces start.
> > I use reducer also as combiner, my cluster has 6 machines, namenode and
> > jobtracker run also as slaves.
> > There were 44 maps and 6 reduces in the last example, I never tried a
> > bigger
> > job.
> >
> > What can the problem be? I've read somewhere that this is not the normal
> > behaviour.
> > Replication factor is 3.
> > Thank you in advance for any pointers.
> >
> > Rasit
> >
>
>
>
> --
> http://daily.appspot.com/food/
>
--
M. Raşit ÖZDAŞ
Re: Reduce doesn't start until map finishes
Posted by Nick Cen <ce...@gmail.com>.
how many reducer do you have? You should make this value larger then 1 to
make mapper and reducer run concurrently. You can set this value from
JobConf.*setNumReduceTasks*().
2009/3/2 Rasit OZDAS <ra...@gmail.com>
> Hi!
>
> Whatever code I run on hadoop, reduce starts a few seconds after map
> finishes.
> And worse, when I run 10 jobs parallely (using threads and sending one
> after
> another)
> all maps finish sequentially, then after 8-10 seconds reduces start.
> I use reducer also as combiner, my cluster has 6 machines, namenode and
> jobtracker run also as slaves.
> There were 44 maps and 6 reduces in the last example, I never tried a
> bigger
> job.
>
> What can the problem be? I've read somewhere that this is not the normal
> behaviour.
> Replication factor is 3.
> Thank you in advance for any pointers.
>
> Rasit
>
--
http://daily.appspot.com/food/