You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 宋轶 <yi...@outlook.com> on 2015/03/02 06:47:38 UTC
RE: proposal of cube building optimization
Hi Xu,
Thanks for the proposal. But I don't quite understand your new algorithm.
1. The step #2 #3 you mentioned are the built-in features of MapReduce computation framework (Partition, sort and spill). 2. I remember we've tried this one-stage method for cube building in our POC phase. The problem of it is that each mapper will generate too much intermediate data, and the network will be the bottleneck in Shuffle phase
ThanksGeorge Song
> From: jiangxu.china@qq.com
> To: dev@kylin.incubator.apache.org
> Subject: proposal of cube building optimization
> Date: Sat, 28 Feb 2015 11:24:59 +0800
>
> Hi Guys,
>
>
> Now the cube is built by multi-stage map-reduce job. It may introduce unnecessary latency for some cases (e.g. incremental building).
>
>
> We can introduce another cube building algorithm as below:
> 1. When the mapper process the raw record, it will generate all the valid combination record that will be put into memory.
> 2. When memory is almost full, mapper will write all the combination records to reducer.
> 3. After mapper write the records to reduce, it will cleanup the memory for further process.
>
>
> Basically, mapper logically split the data block by memory limitation.
>
>
> Thanks
> Jiang Xu
RE: proposal of cube building optimization
Posted by 宋轶 <yi...@outlook.com>.
agree
> Date: Tue, 3 Mar 2015 15:35:23 +0800
> Subject: Re: proposal of cube building optimization
> From: liyang@apache.org
> To: dev@kylin.incubator.apache.org
>
> This proposal is the same as https://issues.apache.org/jira/browse/KYLIN-607
> that I created earlier.
>
> @宋轶, the difference to our very first POC is that, here the mapper outputs
> the aggregated result of a small chunk of records, KVs of a micro segment,
> not the very raw records.
>
> In the ideal case, the solution could achieve 1 * [Total Cube Size]
> shuffling when there's a mandatory dimension and each mapper takes a
> different piece on the dimension. E.g. month is mandatory and each mapper
> is assign a different month data. Then every mapper's output won't
> duplicate. And the shuffle size is optimal.
>
> Of course, in the worst case, the shuffle size might be times of the
> current. So it really depends on the data set and aggregation config. What
> we are seeing now is more often than not, date/time will be a mandatory
> column, and if that's true, the new method will have an edge.
>
> Cheers
> Yang
>
>
>
> On Mon, Mar 2, 2015 at 2:13 PM, 蒋旭 <ji...@qq.com> wrote:
>
> > 1. One step building is more suitable for incremental building that has
> > small data size. Full building on large data set can still use multiple
> > stage building.
> >
> >
> > 2. Since mapper will manage memory by itself, it will cache the
> > intermediate result in memory as more as possible. Moreover, mapper will do
> > preaggregation in memory just like combiner. In this way, it should reduce
> > the shuffle data size.
> >
> >
> > 3. Since it's one step building, the data read size and job schedule
> > latency should be much less.
> >
> >
> > Thanks
> > Jiang Xu
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: Ted Dunning <te...@gmail.com>
> > 发送时间: 2015年03月02日 13:52
> > 收件人: dev <de...@kylin.incubator.apache.org>
> > 主题: Re: proposal of cube building optimization
> >
> >
> >
> > On Mon, Mar 2, 2015 at 6:47 AM, 宋轶 <yi...@outlook.com> wrote:
> >
> > > The problem of it is that each mapper will generate too much intermediate
> > > data, and the network will be the bottleneck in Shuffle phase
> >
> >
> > This would prevent multiple passes over the input data. Is there a
> > difference in the amount of shuffled data from the amount that would be
> > shuffled by multiple map-reduce steps?
> >
Re: proposal of cube building optimization
Posted by Li Yang <li...@apache.org>.
This proposal is the same as https://issues.apache.org/jira/browse/KYLIN-607
that I created earlier.
@宋轶, the difference to our very first POC is that, here the mapper outputs
the aggregated result of a small chunk of records, KVs of a micro segment,
not the very raw records.
In the ideal case, the solution could achieve 1 * [Total Cube Size]
shuffling when there's a mandatory dimension and each mapper takes a
different piece on the dimension. E.g. month is mandatory and each mapper
is assign a different month data. Then every mapper's output won't
duplicate. And the shuffle size is optimal.
Of course, in the worst case, the shuffle size might be times of the
current. So it really depends on the data set and aggregation config. What
we are seeing now is more often than not, date/time will be a mandatory
column, and if that's true, the new method will have an edge.
Cheers
Yang
On Mon, Mar 2, 2015 at 2:13 PM, 蒋旭 <ji...@qq.com> wrote:
> 1. One step building is more suitable for incremental building that has
> small data size. Full building on large data set can still use multiple
> stage building.
>
>
> 2. Since mapper will manage memory by itself, it will cache the
> intermediate result in memory as more as possible. Moreover, mapper will do
> preaggregation in memory just like combiner. In this way, it should reduce
> the shuffle data size.
>
>
> 3. Since it's one step building, the data read size and job schedule
> latency should be much less.
>
>
> Thanks
> Jiang Xu
>
>
> ------------------ 原始邮件 ------------------
> 发件人: Ted Dunning <te...@gmail.com>
> 发送时间: 2015年03月02日 13:52
> 收件人: dev <de...@kylin.incubator.apache.org>
> 主题: Re: proposal of cube building optimization
>
>
>
> On Mon, Mar 2, 2015 at 6:47 AM, 宋轶 <yi...@outlook.com> wrote:
>
> > The problem of it is that each mapper will generate too much intermediate
> > data, and the network will be the bottleneck in Shuffle phase
>
>
> This would prevent multiple passes over the input data. Is there a
> difference in the amount of shuffled data from the amount that would be
> shuffled by multiple map-reduce steps?
>
回复：proposal of cube building optimization
Posted by 蒋旭 <ji...@qq.com>.
1. One step building is more suitable for incremental building that has small data size. Full building on large data set can still use multiple stage building.
2. Since mapper will manage memory by itself, it will cache the intermediate result in memory as more as possible. Moreover, mapper will do preaggregation in memory just like combiner. In this way, it should reduce the shuffle data size.
3. Since it's one step building, the data read size and job schedule latency should be much less.
Thanks
Jiang Xu
------------------ 原始邮件 ------------------
发件人: Ted Dunning <te...@gmail.com>
发送时间: 2015年03月02日 13:52
收件人: dev <de...@kylin.incubator.apache.org>
主题: Re: proposal of cube building optimization
On Mon, Mar 2, 2015 at 6:47 AM, 宋轶 <yi...@outlook.com> wrote:
> The problem of it is that each mapper will generate too much intermediate
> data, and the network will be the bottleneck in Shuffle phase
This would prevent multiple passes over the input data. Is there a
difference in the amount of shuffled data from the amount that would be
shuffled by multiple map-reduce steps?
Re: proposal of cube building optimization
Posted by Ted Dunning <te...@gmail.com>.
On Mon, Mar 2, 2015 at 6:47 AM, 宋轶 <yi...@outlook.com> wrote:
> The problem of it is that each mapper will generate too much intermediate
> data, and the network will be the bottleneck in Shuffle phase
This would prevent multiple passes over the input data. Is there a
difference in the amount of shuffled data from the amount that would be
shuffled by multiple map-reduce steps?