You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Qing Yan <qi...@gmail.com> on 2009/02/26 14:57:16 UTC
Combine() optimization
Is there any way/plan for Hive to take advantage of M/R's combine()
phrase? There can be either rules embedded in in the query optimizer or
hints passed by user...
GROUP BY should benefit from this alot..
Any comment?
RE: Combine() optimization
Posted by Joydeep Sen Sarma <js...@facebook.com>.
Yeah - we definitely want to convert it to a MFU type flush algorithm.
If someone wants to take a crack at it before we can get to it - that would be awesome
________________________________
From: Namit Jain [mailto:njain@facebook.com]
Sent: Friday, February 27, 2009 1:59 PM
To: hive-user@hadoop.apache.org
Subject: RE: Combine() optimization
It dumps 10% of the hash table randomly today
From: Scott Carey [mailto:scott@richrelevance.com]
Sent: Friday, February 27, 2009 1:41 PM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Does it dump all contents and start over, or use a LRU or MFU algorithm? LinkedHashMap makes LRUs and similar constructs fairly easy to make.
My guess is that most data types have biased value distributions that will take advantage of map side partial aggregation fairly well.
On 2/26/09 6:02 PM, "Namit Jain" <nj...@facebook.com> wrote:
Yes, it flushes the data when the hash table is occupying too much memory
From: Qing Yan [mailto:qingyan@gmail.com]
Sent: Thursday, February 26, 2009 5:58 PM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Got it.
Does map side aggregation has any special requirement about the dataset? E.g. The number of unqiue group by keys could be too big to hold in memory. Will it still work?
On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com> wrote:
Hi Qing,
We did think about Combiner when we started Hive. However earlier discussions lead us to believe that hash-based aggregation inside the mapper will be as competitive as using combiner in most cases.
In order to enable map-side aggregation, we just need to do the following before running the hive query:
set hive.map.aggr=true;
Zheng
On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com> wrote:
Right now Hive does not exploit the combiner. But hash-based map-side
aggregation in hive (controlled by hints) provides a similar optimization.
Using the combiner in addition to map-side aggregation should improve the
performance even more if the combiner can further aggregate the partial
aggregates generated from the mapper.
On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
> Is there any way/plan for Hive to take advantage of M/R's combine()
> phrase? There can be either rules embedded in in the query optimizer or hints
> passed by user...
> GROUP BY should benefit from this alot..
>
> Any comment?
>
>
>
RE: Combine() optimization
Posted by Namit Jain <nj...@facebook.com>.
It dumps 10% of the hash table randomly today
From: Scott Carey [mailto:scott@richrelevance.com]
Sent: Friday, February 27, 2009 1:41 PM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Does it dump all contents and start over, or use a LRU or MFU algorithm? LinkedHashMap makes LRUs and similar constructs fairly easy to make.
My guess is that most data types have biased value distributions that will take advantage of map side partial aggregation fairly well.
On 2/26/09 6:02 PM, "Namit Jain" <nj...@facebook.com> wrote:
Yes, it flushes the data when the hash table is occupying too much memory
From: Qing Yan [mailto:qingyan@gmail.com]
Sent: Thursday, February 26, 2009 5:58 PM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Got it.
Does map side aggregation has any special requirement about the dataset? E.g. The number of unqiue group by keys could be too big to hold in memory. Will it still work?
On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com> wrote:
Hi Qing,
We did think about Combiner when we started Hive. However earlier discussions lead us to believe that hash-based aggregation inside the mapper will be as competitive as using combiner in most cases.
In order to enable map-side aggregation, we just need to do the following before running the hive query:
set hive.map.aggr=true;
Zheng
On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com> wrote:
Right now Hive does not exploit the combiner. But hash-based map-side
aggregation in hive (controlled by hints) provides a similar optimization.
Using the combiner in addition to map-side aggregation should improve the
performance even more if the combiner can further aggregate the partial
aggregates generated from the mapper.
On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
> Is there any way/plan for Hive to take advantage of M/R's combine()
> phrase? There can be either rules embedded in in the query optimizer or hints
> passed by user...
> GROUP BY should benefit from this alot..
>
> Any comment?
>
>
>
RE: Combine() optimization
Posted by Namit Jain <nj...@facebook.com>.
Look at the patch for
http://issues.apache.org/jira/browse/HIVE-223
It has not been committed yet.
Thanks,
-namit
________________________________________
From: Qing Yan [qingyan@gmail.com]
Sent: Friday, February 27, 2009 12:12 AM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Ouch, I was getting tons of exceptions after turning on map-side aggregation:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
at java.lang.StringCoding.encode(StringCoding.java:272)
at java.lang.String.getBytes(String.java:947)
at org.apache.hadoop.hive.serde2.thrift.TBinarySortableProtocol.writeString(TBinarySortableProtocol.java:299)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeString.serialize(DynamicSerDeTypeString.java:65)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
at org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:153)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:306)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:564)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.close(GroupByOperator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:96)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:552)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.close(GroupByOperator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:96)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
java.io.IOException: Task process exit with nonzero status of 1.
...
Just to confirm is this just a bug or is by design ?
On Fri, Feb 27, 2009 at 10:02 AM, Namit Jain <nj...@facebook.com>> wrote:
Yes, it flushes the data when the hash table is occupying too much memory
From: Qing Yan [mailto:qingyan@gmail.com<ma...@gmail.com>]
Sent: Thursday, February 26, 2009 5:58 PM
To: hive-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: Combine() optimization
Got it.
Does map side aggregation has any special requirement about the dataset? E.g. The number of unqiue group by keys could be too big to hold in memory. Will it still work?
On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com>> wrote:
Hi Qing,
We did think about Combiner when we started Hive. However earlier discussions lead us to believe that hash-based aggregation inside the mapper will be as competitive as using combiner in most cases.
In order to enable map-side aggregation, we just need to do the following before running the hive query:
set hive.map.aggr=true;
Zheng
On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com>> wrote:
Right now Hive does not exploit the combiner. But hash-based map-side
aggregation in hive (controlled by hints) provides a similar optimization.
Using the combiner in addition to map-side aggregation should improve the
performance even more if the combiner can further aggregate the partial
aggregates generated from the mapper.
On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com>> wrote:
> Is there any way/plan for Hive to take advantage of M/R's combine()
> phrase? There can be either rules embedded in in the query optimizer or hints
> passed by user...
> GROUP BY should benefit from this alot..
>
> Any comment?
>
>
>
--
Yours,
Zheng
Re: Combine() optimization
Posted by Qing Yan <qi...@gmail.com>.
Ouch, I was getting tons of exceptions after turning on map-side
aggregation:
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
at java.lang.StringCoding.encode(StringCoding.java:272)
at java.lang.String.getBytes(String.java:947)
at
org.apache.hadoop.hive.serde2.thrift.TBinarySortableProtocol.writeString(TBinarySortableProtocol.java:299)
at
org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeTypeString.serialize(DynamicSerDeTypeString.java:65)
at
org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeFieldList.serialize(DynamicSerDeFieldList.java:249)
at
org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDeStructBase.serialize(DynamicSerDeStructBase.java:81)
at
org.apache.hadoop.hive.serde2.dynamic_type.DynamicSerDe.serialize(DynamicSerDe.java:174)
at
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:153)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:306)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:564)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.close(GroupByOperator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:96)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:552)
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.close(GroupByOperator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:263)
at org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:96)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.Child.main(Child.java:155)
java.io.IOException: Task process exit with nonzero status of 1.
...
Just to confirm is this just a bug or is by design ?
On Fri, Feb 27, 2009 at 10:02 AM, Namit Jain <nj...@facebook.com> wrote:
> Yes, it flushes the data when the hash table is occupying too much memory
>
>
>
>
>
> *From:* Qing Yan [mailto:qingyan@gmail.com]
> *Sent:* Thursday, February 26, 2009 5:58 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: Combine() optimization
>
>
>
> Got it.
>
>
>
> Does map side aggregation has any special requirement about the dataset?
> E.g. The number of unqiue group by keys could be too big to hold
> in memory. Will it still work?
>
> On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com> wrote:
>
> Hi Qing,
>
> We did think about Combiner when we started Hive. However earlier
> discussions lead us to believe that hash-based aggregation inside the mapper
> will be as competitive as using combiner in most cases.
>
> In order to enable map-side aggregation, we just need to do the following
> before running the hive query:
> set hive.map.aggr=true;
>
> Zheng
>
>
>
> On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com> wrote:
>
> Right now Hive does not exploit the combiner. But hash-based map-side
> aggregation in hive (controlled by hints) provides a similar optimization.
> Using the combiner in addition to map-side aggregation should improve the
> performance even more if the combiner can further aggregate the partial
> aggregates generated from the mapper.
>
>
>
> On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
>
> > Is there any way/plan for Hive to take advantage of M/R's combine()
> > phrase? There can be either rules embedded in in the query optimizer or
> hints
> > passed by user...
> > GROUP BY should benefit from this alot..
> >
> > Any comment?
> >
> >
> >
>
>
>
> --
> Yours,
> Zheng
>
>
>
Re: Combine() optimization
Posted by Scott Carey <sc...@richrelevance.com>.
Does it dump all contents and start over, or use a LRU or MFU algorithm? LinkedHashMap makes LRUs and similar constructs fairly easy to make.
My guess is that most data types have biased value distributions that will take advantage of map side partial aggregation fairly well.
On 2/26/09 6:02 PM, "Namit Jain" <nj...@facebook.com> wrote:
Yes, it flushes the data when the hash table is occupying too much memory
From: Qing Yan [mailto:qingyan@gmail.com]
Sent: Thursday, February 26, 2009 5:58 PM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Got it.
Does map side aggregation has any special requirement about the dataset? E.g. The number of unqiue group by keys could be too big to hold in memory. Will it still work?
On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com> wrote:
Hi Qing,
We did think about Combiner when we started Hive. However earlier discussions lead us to believe that hash-based aggregation inside the mapper will be as competitive as using combiner in most cases.
In order to enable map-side aggregation, we just need to do the following before running the hive query:
set hive.map.aggr=true;
Zheng
On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com> wrote:
Right now Hive does not exploit the combiner. But hash-based map-side
aggregation in hive (controlled by hints) provides a similar optimization.
Using the combiner in addition to map-side aggregation should improve the
performance even more if the combiner can further aggregate the partial
aggregates generated from the mapper.
On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
> Is there any way/plan for Hive to take advantage of M/R's combine()
> phrase? There can be either rules embedded in in the query optimizer or hints
> passed by user...
> GROUP BY should benefit from this alot..
>
> Any comment?
>
>
>
RE: Combine() optimization
Posted by Namit Jain <nj...@facebook.com>.
Yes, it flushes the data when the hash table is occupying too much memory
From: Qing Yan [mailto:qingyan@gmail.com]
Sent: Thursday, February 26, 2009 5:58 PM
To: hive-user@hadoop.apache.org
Subject: Re: Combine() optimization
Got it.
Does map side aggregation has any special requirement about the dataset? E.g. The number of unqiue group by keys could be too big to hold in memory. Will it still work?
On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com>> wrote:
Hi Qing,
We did think about Combiner when we started Hive. However earlier discussions lead us to believe that hash-based aggregation inside the mapper will be as competitive as using combiner in most cases.
In order to enable map-side aggregation, we just need to do the following before running the hive query:
set hive.map.aggr=true;
Zheng
On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com>> wrote:
Right now Hive does not exploit the combiner. But hash-based map-side
aggregation in hive (controlled by hints) provides a similar optimization.
Using the combiner in addition to map-side aggregation should improve the
performance even more if the combiner can further aggregate the partial
aggregates generated from the mapper.
On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com>> wrote:
> Is there any way/plan for Hive to take advantage of M/R's combine()
> phrase? There can be either rules embedded in in the query optimizer or hints
> passed by user...
> GROUP BY should benefit from this alot..
>
> Any comment?
>
>
>
--
Yours,
Zheng
Re: Combine() optimization
Posted by Qing Yan <qi...@gmail.com>.
Got it.
Does map side aggregation has any special requirement about the dataset?
E.g. The number of unqiue group by keys could be too big to hold
in memory. Will it still work?
On Fri, Feb 27, 2009 at 5:50 AM, Zheng Shao <zs...@gmail.com> wrote:
> Hi Qing,
>
> We did think about Combiner when we started Hive. However earlier
> discussions lead us to believe that hash-based aggregation inside the mapper
> will be as competitive as using combiner in most cases.
>
> In order to enable map-side aggregation, we just need to do the following
> before running the hive query:
> set hive.map.aggr=true;
>
> Zheng
>
>
> On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com> wrote:
>
>> Right now Hive does not exploit the combiner. But hash-based map-side
>> aggregation in hive (controlled by hints) provides a similar optimization.
>> Using the combiner in addition to map-side aggregation should improve the
>> performance even more if the combiner can further aggregate the partial
>> aggregates generated from the mapper.
>>
>>
>> On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
>>
>> > Is there any way/plan for Hive to take advantage of M/R's combine()
>> > phrase? There can be either rules embedded in in the query optimizer or
>> hints
>> > passed by user...
>> > GROUP BY should benefit from this alot..
>> >
>> > Any comment?
>> >
>> >
>> >
>>
>>
>
>
> --
> Yours,
> Zheng
>
Re: Combine() optimization
Posted by Zheng Shao <zs...@gmail.com>.
Hi Qing,
We did think about Combiner when we started Hive. However earlier
discussions lead us to believe that hash-based aggregation inside the mapper
will be as competitive as using combiner in most cases.
In order to enable map-side aggregation, we just need to do the following
before running the hive query:
set hive.map.aggr=true;
Zheng
On Thu, Feb 26, 2009 at 6:03 AM, Raghu Murthy <ra...@facebook.com> wrote:
> Right now Hive does not exploit the combiner. But hash-based map-side
> aggregation in hive (controlled by hints) provides a similar optimization.
> Using the combiner in addition to map-side aggregation should improve the
> performance even more if the combiner can further aggregate the partial
> aggregates generated from the mapper.
>
>
> On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
>
> > Is there any way/plan for Hive to take advantage of M/R's combine()
> > phrase? There can be either rules embedded in in the query optimizer or
> hints
> > passed by user...
> > GROUP BY should benefit from this alot..
> >
> > Any comment?
> >
> >
> >
>
>
--
Yours,
Zheng
Re: Combine() optimization
Posted by Raghu Murthy <ra...@facebook.com>.
Right now Hive does not exploit the combiner. But hash-based map-side
aggregation in hive (controlled by hints) provides a similar optimization.
Using the combiner in addition to map-side aggregation should improve the
performance even more if the combiner can further aggregate the partial
aggregates generated from the mapper.
On 2/26/09 5:57 AM, "Qing Yan" <qi...@gmail.com> wrote:
> Is there any way/plan for Hive to take advantage of M/R's combine()
> phrase? There can be either rules embedded in in the query optimizer or hints
> passed by user...
> GROUP BY should benefit from this alot..
>
> Any comment?
>
>
>