[jira] [Commented] (HIVE-11502) Map side aggregation is extremely slow

["Yongzhi Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663199#comment-14663199 ] 

Yongzhi Chen commented on HIVE-11502:
-------------------------------------

It is a regression, I ran the query in 0.13.1 version with hive.map.aggr is true, it finished in 20 seconds.
In the master branch, the major time spend in following stack.
{noformat}
HashMap<K,V>.getEntry(Object) line: 465	
HashMap<K,V>.get(Object) line: 417	
PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(String) line: 373	
PrimitiveTypeInfo.getPrimitiveTypeEntry() line: 85	
PrimitiveTypeInfo.getPrimitiveCategory() line: 63	
WritableDoubleObjectInspector(AbstractPrimitiveObjectInspector).getPrimitiveCategory() line: 58	
ObjectInspectorUtils.compare(Object, ObjectInspector, Object, ObjectInspector, MapEqualComparer) line: 694	
ObjectInspectorUtils.compare(Object, ObjectInspector, Object, ObjectInspector) line: 668	
ListObjectsEqualComparer$FieldComparer.areEqual(Object, Object) line: 127	
ListObjectsEqualComparer.areEqual(Object[], Object[]) line: 172	
KeyWrapperFactory$ListKeyWrapper.equals(Object) line: 101	
HashMap<K,V>.getEntry(Object) line: 467	
HashMap<K,V>.get(Object) line: 417	
GroupByOperator.processHashAggr(Object, ObjectInspector, KeyWrapper) line: 777	
GroupByOperator.processKey(Object, ObjectInspector) line: 693	
GroupByOperator.process(Object, int) line: 761	
SelectOperator(Operator<T>).forward(Object, ObjectInspector) line: 837	
SelectOperator.process(Object, int) line: 88	
TableScanOperator(Operator<T>).forward(Object, ObjectInspector) line: 837	
TableScanOperator.process(Object, int) line: 97	
MapOperator$MapOpCtx.forward(Object) line: 162	
MapOperator.process(Writable) line: 508	
ExecMapper.map(Object, Object, OutputCollector, Reporter) line: 163	
{noformat}
It seems that heavily used PrimitiveObjectInspectorUtils.getTypeEntryFromTypeName(String) slows down the query. So I change the code to store the PrimitiveTypeEntry as instance variable in PrimitiveTypeInfo. This does improve  the performance a lot, now the query can finish in 1 hour. But it is still very slow.
I checked 0.13.1 code, it uses Hashmap too, but much much faster. 
I do not know why the HashMap search is so slow in master branch(and 1.1 or later version). 


> Map side aggregation is extremely slow
> --------------------------------------
>
>                 Key: HIVE-11502
>                 URL: https://issues.apache.org/jira/browse/HIVE-11502
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer, Physical Optimizer
>    Affects Versions: 1.2.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>
> For the query as following:
> {noformat}
> create table tbl2 as 
> select col1, max(col2) as col2 
> from tbl1 group by col1;
> {noformat}
> If the column for group by has many different values (for example 400000), the map side aggregation is very slow. I ran the query which took more than 3 hours , after 3 hours, I have to kill the query.
> The same query can finish in 7 seconds, if I turn off map side aggregation by:
> {noformat}
> set hive.map.aggr = false;
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)