You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by jian yi <ey...@gmail.com> on 2010/02/06 15:24:45 UTC
MBR model diagram (Map-Balance-Reduce)
In MR (Map-Reduce) model, reducings are not balanced, because the scale of
partitiones are unbalanced. How to balance? We can control the size of
partition, rehash the bigger parition and combine to the specified size. If
a key has many values, it's necessary to execute mapreduce twice.The
following is the model digram:
[image: Map-Balance-Reduce.JPG]
Scheduler can regard a task as a timeslice similarly OS scheduler.
Re: MBR model diagram (Map-Balance-Reduce)
Posted by jian yi <ey...@gmail.com>.
If a split is bigger than a specified size, it will be splitted again. If a
split is smaller than a specified size, it will be combined with others, we
can name the combining procedure regroup. The combining is logic, it's not
necessay to combine these smaller splits to a disk file, which will not
affect the performance.The target is that every task spent same time
running.
2010/2/6 jian yi <ey...@gmail.com>
In MR (Map-Reduce) model, reducings are not balanced, because the scale of
> partitiones are unbalanced. How to balance? We can control the size of
> partition, rehash the bigger parition and combine to the specified size. If
> a key has many values, it's necessary to execute mapreduce twice.The
> following is the model digram:
Scheduler can regard a task as a timeslice similarly OS scheduler.
Re: MBR model diagram (Map-Balance-Reduce)
Posted by jian yi <ey...@gmail.com>.
If a split is bigger than a specified size, it will be splitted again. If a
split is smaller than a specified size, it will be combined with others, we
can name the combining procedure regroup. The combining is logic, it's not
necessay to combine these smaller splits to a disk file, which will not
affect the performance.The target is that every task spent same time
running.
[image: mbr_detailedx.JPG]
2010/2/6 jian yi <ey...@gmail.com>
> In MR (Map-Reduce) model, reducings are not balanced, because the scale of
> partitiones are unbalanced. How to balance? We can control the size of
> partition, rehash the bigger parition and combine to the specified size. If
> a key has many values, it's necessary to execute mapreduce twice.The
> following is the model digram:
>
> [image: Map-Balance-Reduce.JPG]
>
> Scheduler can regard a task as a timeslice similarly OS scheduler.
>
>