You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Brice Arnould <br...@vleu.net> on 2008/05/05 10:56:06 UTC

Parallelism of sorts

Hi !
I was asking myself if it could be a good idea to parallelize some of the
alogorithms of Hadoop, such as MergeSorter, for the case a single job of
run on a multicore system. If so, I would like to propose a patch that do
it, by adding a parameter like io.sort.parallelism.
If the number of sorting threads is less than this value, Hadoop will start
more than one thread.

Thanks
Brice



Re: Parallelism of sorts

Posted by Brice Arnould <br...@vleu.net>.

On Mon, 05 May 2008 11:29:00 -0700, Doug Cutting <cu...@apache.org>
wrote:
> Brice Arnould wrote:
>> I was asking myself if it could be a good idea to parallelize some of
> the
>> alogorithms of Hadoop, such as MergeSorter, for the case a single job of
>> run on a multicore system.
> 
> One can already exploit parallelism on a multicore system by using
> "pseudo-distributed" mode and increasing
> mapred.tasktracker.map.tasks.maximum and
> mapred.tasktracker.reduce.tasks.maximum.

> LocalRunner should also someday be enhanced to run multiple maps and
> reduces in separate threads, which would be more efficient, since
> intermediate data would not need to travel through the loopback network
> interface.  But I don't see an urgent case for making the sort code
> itself multi-threaded, since MapReduce itself performs parallel sorting.
Sorry, I really had  misunderstood the way it works. Thanks for your
explanations, I'm going to look at LocalJobRunner.

Brice



Re: Parallelism of sorts

Posted by Doug Cutting <cu...@apache.org>.
Brice Arnould wrote:
> I was asking myself if it could be a good idea to parallelize some of the
> alogorithms of Hadoop, such as MergeSorter, for the case a single job of
> run on a multicore system.

One can already exploit parallelism on a multicore system by using 
"pseudo-distributed" mode and increasing 
mapred.tasktracker.map.tasks.maximum and 
mapred.tasktracker.reduce.tasks.maximum.

LocalRunner should also someday be enhanced to run multiple maps and 
reduces in separate threads, which would be more efficient, since 
intermediate data would not need to travel through the loopback network 
interface.  But I don't see an urgent case for making the sort code 
itself multi-threaded, since MapReduce itself performs parallel sorting.

Doug