You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by jc...@apache.org on 2018/03/22 16:57:51 UTC
[28/34] hive git commit: HIVE-18979: Enable
AggregateReduceFunctionsRule from Calcite (Jesus Camacho Rodriguez,
reviewed by Ashutosh Chauhan)
http://git-wip-us.apache.org/repos/asf/hive/blob/5cb8867b/ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets3_dec.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets3_dec.q.out b/ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets3_dec.q.out
index dabc987..74caa3f 100644
--- a/ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets3_dec.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets3_dec.q.out
@@ -76,18 +76,18 @@ STAGE PLANS:
projectedOutputColumnNums: [0, 1, 2]
Statistics: Num rows: 12 Data size: 5760 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(c_dec), count()
+ aggregations: sum(c_dec), count(c_dec), count()
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimal(col 2:decimal(10,2)) -> struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VectorUDAFCountStar(*) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 2:decimal(10,2)) -> decimal(20,2), VectorUDAFCount(col 2:decimal(10,2)) -> bigint, VectorUDAFCountStar(*) -> bigint
className: VectorGroupByOperator
groupByMode: HASH
keyExpressions: col 0:string, col 1:string, ConstantVectorExpression(val 0) -> 4:bigint
native: false
vectorProcessingMode: HASH
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: a (type: string), b (type: string), 0L (type: bigint)
mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 48 Data size: 23040 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: bigint)
@@ -98,9 +98,9 @@ STAGE PLANS:
keyColumnNums: [0, 1, 2]
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- valueColumnNums: [3, 4]
+ valueColumnNums: [3, 4, 5]
Statistics: Num rows: 48 Data size: 23040 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>), _col4 (type: bigint)
+ value expressions: _col3 (type: decimal(20,2)), _col4 (type: bigint), _col5 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -129,33 +129,34 @@ STAGE PLANS:
usesVectorUDFAdaptor: false
vectorized: true
rowBatchContext:
- dataColumnCount: 5
- dataColumns: KEY._col0:string, KEY._col1:string, KEY._col2:bigint, VALUE._col0:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VALUE._col1:bigint
+ dataColumnCount: 6
+ dataColumns: KEY._col0:string, KEY._col1:string, KEY._col2:bigint, VALUE._col0:decimal(20,2), VALUE._col1:bigint, VALUE._col2:bigint
partitionColumnCount: 0
scratchColumnTypeNames: []
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0), count(VALUE._col1)
+ aggregations: sum(VALUE._col0), count(VALUE._col1), count(VALUE._col2)
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimalFinal(col 3:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>) -> decimal(14,6), VectorUDAFCountMerge(col 4:bigint) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 3:decimal(20,2)) -> decimal(20,2), VectorUDAFCountMerge(col 4:bigint) -> bigint, VectorUDAFCountMerge(col 5:bigint) -> bigint
className: VectorGroupByOperator
groupByMode: MERGEPARTIAL
keyExpressions: col 0:string, col 1:string, col 2:bigint
native: false
vectorProcessingMode: MERGE_PARTIAL
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: bigint)
mode: mergepartial
- outputColumnNames: _col0, _col1, _col3, _col4
+ outputColumnNames: _col0, _col1, _col3, _col4, _col5
Statistics: Num rows: 24 Data size: 11520 Basic stats: COMPLETE Column stats: NONE
pruneGroupingSetId: true
Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col3 (type: decimal(14,6)), _col4 (type: bigint)
+ expressions: _col0 (type: string), _col1 (type: string), (_col3 / _col4) (type: decimal(38,20)), _col5 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [0, 1, 2, 3]
+ projectedOutputColumnNums: [0, 1, 6, 4]
+ selectExpressions: DecimalColDivideDecimalColumn(col 2:decimal(20,2), col 5:decimal(19,0))(children: CastLongToDecimal(col 3:bigint) -> 5:decimal(19,0)) -> 6:decimal(38,20)
Statistics: Num rows: 24 Data size: 11520 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
@@ -214,18 +215,18 @@ STAGE PLANS:
projectedOutputColumnNums: [0, 1, 2]
Statistics: Num rows: 12 Data size: 5760 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(c_dec), count()
+ aggregations: sum(c_dec), count(c_dec), count()
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimal(col 2:decimal(10,2)) -> struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VectorUDAFCountStar(*) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 2:decimal(10,2)) -> decimal(20,2), VectorUDAFCount(col 2:decimal(10,2)) -> bigint, VectorUDAFCountStar(*) -> bigint
className: VectorGroupByOperator
groupByMode: HASH
keyExpressions: col 0:string, col 1:string, ConstantVectorExpression(val 0) -> 4:bigint
native: false
vectorProcessingMode: HASH
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: a (type: string), b (type: string), 0L (type: bigint)
mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 48 Data size: 23040 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: bigint)
@@ -236,9 +237,9 @@ STAGE PLANS:
keyColumnNums: [0, 1, 2]
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- valueColumnNums: [3, 4]
+ valueColumnNums: [3, 4, 5]
Statistics: Num rows: 48 Data size: 23040 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>), _col4 (type: bigint)
+ value expressions: _col3 (type: decimal(20,2)), _col4 (type: bigint), _col5 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -267,33 +268,34 @@ STAGE PLANS:
usesVectorUDFAdaptor: false
vectorized: true
rowBatchContext:
- dataColumnCount: 5
- dataColumns: KEY._col0:string, KEY._col1:string, KEY._col2:bigint, VALUE._col0:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VALUE._col1:bigint
+ dataColumnCount: 6
+ dataColumns: KEY._col0:string, KEY._col1:string, KEY._col2:bigint, VALUE._col0:decimal(20,2), VALUE._col1:bigint, VALUE._col2:bigint
partitionColumnCount: 0
scratchColumnTypeNames: []
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0), count(VALUE._col1)
+ aggregations: sum(VALUE._col0), count(VALUE._col1), count(VALUE._col2)
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimalFinal(col 3:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>) -> decimal(14,6), VectorUDAFCountMerge(col 4:bigint) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 3:decimal(20,2)) -> decimal(20,2), VectorUDAFCountMerge(col 4:bigint) -> bigint, VectorUDAFCountMerge(col 5:bigint) -> bigint
className: VectorGroupByOperator
groupByMode: MERGEPARTIAL
keyExpressions: col 0:string, col 1:string, col 2:bigint
native: false
vectorProcessingMode: MERGE_PARTIAL
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: bigint)
mode: mergepartial
- outputColumnNames: _col0, _col1, _col3, _col4
+ outputColumnNames: _col0, _col1, _col3, _col4, _col5
Statistics: Num rows: 24 Data size: 11520 Basic stats: COMPLETE Column stats: NONE
pruneGroupingSetId: true
Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col3 (type: decimal(14,6)), _col4 (type: bigint)
+ expressions: _col0 (type: string), _col1 (type: string), (_col3 / _col4) (type: decimal(38,20)), _col5 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [0, 1, 2, 3]
+ projectedOutputColumnNums: [0, 1, 6, 4]
+ selectExpressions: DecimalColDivideDecimalColumn(col 2:decimal(20,2), col 5:decimal(19,0))(children: CastLongToDecimal(col 3:bigint) -> 5:decimal(19,0)) -> 6:decimal(38,20)
Statistics: Num rows: 24 Data size: 11520 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
@@ -321,22 +323,22 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@t1
#### A masked pattern was here ####
a b _c2 _c3
-1 1 3.000000 2
-1 2 2.000000 1
-1 NULL 2.666667 3
-2 2 5.333333 3
-2 3 5.000000 2
-2 NULL 5.200000 5
-3 2 8.000000 1
-3 NULL 8.000000 1
-5 1 2.000000 1
-5 NULL 2.000000 1
-8 1 1.000000 2
-8 NULL 1.000000 2
-NULL 1 2.000000 5
-NULL 2 5.200000 5
-NULL 3 5.000000 2
-NULL NULL 3.833333 12
+1 1 3.00000000000000000000 2
+1 2 2.00000000000000000000 1
+1 NULL 2.66666666666666666667 3
+2 2 5.33333333333333333333 3
+2 3 5.00000000000000000000 2
+2 NULL 5.20000000000000000000 5
+3 2 8.00000000000000000000 1
+3 NULL 8.00000000000000000000 1
+5 1 2.00000000000000000000 1
+5 NULL 2.00000000000000000000 1
+8 1 1.00000000000000000000 2
+8 NULL 1.00000000000000000000 2
+NULL 1 2.00000000000000000000 5
+NULL 2 5.20000000000000000000 5
+NULL 3 5.00000000000000000000 2
+NULL NULL 3.83333333333333333333 12
PREHOOK: query: EXPLAIN VECTORIZATION DETAIL
SELECT a, b, avg(c_dec), count(*) from T1 group by a, b with cube
PREHOOK: type: QUERY
@@ -378,18 +380,18 @@ STAGE PLANS:
projectedOutputColumnNums: [0, 1, 2]
Statistics: Num rows: 12 Data size: 5760 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(c_dec), count()
+ aggregations: sum(c_dec), count(c_dec), count()
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimal(col 2:decimal(10,2)) -> struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VectorUDAFCountStar(*) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 2:decimal(10,2)) -> decimal(20,2), VectorUDAFCount(col 2:decimal(10,2)) -> bigint, VectorUDAFCountStar(*) -> bigint
className: VectorGroupByOperator
groupByMode: HASH
keyExpressions: col 0:string, col 1:string
native: false
vectorProcessingMode: HASH
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: a (type: string), b (type: string)
mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 12 Data size: 5760 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
@@ -400,9 +402,9 @@ STAGE PLANS:
keyColumnNums: [0, 1]
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- valueColumnNums: [2, 3]
+ valueColumnNums: [2, 3, 4]
Statistics: Num rows: 12 Data size: 5760 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>), _col3 (type: bigint)
+ value expressions: _col2 (type: decimal(20,2)), _col3 (type: bigint), _col4 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -431,24 +433,24 @@ STAGE PLANS:
usesVectorUDFAdaptor: false
vectorized: true
rowBatchContext:
- dataColumnCount: 4
- dataColumns: KEY._col0:string, KEY._col1:string, VALUE._col0:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VALUE._col1:bigint
+ dataColumnCount: 5
+ dataColumns: KEY._col0:string, KEY._col1:string, VALUE._col0:decimal(20,2), VALUE._col1:bigint, VALUE._col2:bigint
partitionColumnCount: 0
scratchColumnTypeNames: [bigint]
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0), count(VALUE._col1)
+ aggregations: sum(VALUE._col0), count(VALUE._col1), count(VALUE._col2)
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimalPartial2(col 2:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>) -> struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VectorUDAFCountMerge(col 3:bigint) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 2:decimal(20,2)) -> decimal(20,2), VectorUDAFCountMerge(col 3:bigint) -> bigint, VectorUDAFCountMerge(col 4:bigint) -> bigint
className: VectorGroupByOperator
groupByMode: PARTIALS
- keyExpressions: col 0:string, col 1:string, ConstantVectorExpression(val 0) -> 4:bigint
+ keyExpressions: col 0:string, col 1:string, ConstantVectorExpression(val 0) -> 5:bigint
native: false
vectorProcessingMode: STREAMING
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: KEY._col0 (type: string), KEY._col1 (type: string), 0L (type: bigint)
mode: partials
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 48 Data size: 23040 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: bigint)
@@ -459,9 +461,9 @@ STAGE PLANS:
keyColumnNums: [0, 1, 2]
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- valueColumnNums: [3, 4]
+ valueColumnNums: [3, 4, 5]
Statistics: Num rows: 48 Data size: 23040 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>), _col4 (type: bigint)
+ value expressions: _col3 (type: decimal(20,2)), _col4 (type: bigint), _col5 (type: bigint)
Reducer 3
Execution mode: vectorized, llap
Reduce Vectorization:
@@ -473,33 +475,34 @@ STAGE PLANS:
usesVectorUDFAdaptor: false
vectorized: true
rowBatchContext:
- dataColumnCount: 5
- dataColumns: KEY._col0:string, KEY._col1:string, KEY._col2:bigint, VALUE._col0:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>, VALUE._col1:bigint
+ dataColumnCount: 6
+ dataColumns: KEY._col0:string, KEY._col1:string, KEY._col2:bigint, VALUE._col0:decimal(20,2), VALUE._col1:bigint, VALUE._col2:bigint
partitionColumnCount: 0
scratchColumnTypeNames: []
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0), count(VALUE._col1)
+ aggregations: sum(VALUE._col0), count(VALUE._col1), count(VALUE._col2)
Group By Vectorization:
- aggregators: VectorUDAFAvgDecimalFinal(col 3:struct<count:bigint,sum:decimal(20,2),input:decimal(10,2)>) -> decimal(14,6), VectorUDAFCountMerge(col 4:bigint) -> bigint
+ aggregators: VectorUDAFSumDecimal(col 3:decimal(20,2)) -> decimal(20,2), VectorUDAFCountMerge(col 4:bigint) -> bigint, VectorUDAFCountMerge(col 5:bigint) -> bigint
className: VectorGroupByOperator
groupByMode: FINAL
keyExpressions: col 0:string, col 1:string, col 2:bigint
native: false
vectorProcessingMode: STREAMING
- projectedOutputColumnNums: [0, 1]
+ projectedOutputColumnNums: [0, 1, 2]
keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: bigint)
mode: final
- outputColumnNames: _col0, _col1, _col3, _col4
+ outputColumnNames: _col0, _col1, _col3, _col4, _col5
Statistics: Num rows: 24 Data size: 11520 Basic stats: COMPLETE Column stats: NONE
pruneGroupingSetId: true
Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col3 (type: decimal(14,6)), _col4 (type: bigint)
+ expressions: _col0 (type: string), _col1 (type: string), (_col3 / _col4) (type: decimal(38,20)), _col5 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [0, 1, 2, 3]
+ projectedOutputColumnNums: [0, 1, 6, 4]
+ selectExpressions: DecimalColDivideDecimalColumn(col 2:decimal(20,2), col 5:decimal(19,0))(children: CastLongToDecimal(col 3:bigint) -> 5:decimal(19,0)) -> 6:decimal(38,20)
Statistics: Num rows: 24 Data size: 11520 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
@@ -527,19 +530,19 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@t1
#### A masked pattern was here ####
a b _c2 _c3
-1 1 3.000000 2
-1 2 2.000000 1
-1 NULL 2.666667 3
-2 2 5.333333 3
-2 3 5.000000 2
-2 NULL 5.200000 5
-3 2 8.000000 1
-3 NULL 8.000000 1
-5 1 2.000000 1
-5 NULL 2.000000 1
-8 1 1.000000 2
-8 NULL 1.000000 2
-NULL 1 2.000000 5
-NULL 2 5.200000 5
-NULL 3 5.000000 2
-NULL NULL 3.833333 12
+1 1 3.00000000000000000000 2
+1 2 2.00000000000000000000 1
+1 NULL 2.66666666666666666667 3
+2 2 5.33333333333333333333 3
+2 3 5.00000000000000000000 2
+2 NULL 5.20000000000000000000 5
+3 2 8.00000000000000000000 1
+3 NULL 8.00000000000000000000 1
+5 1 2.00000000000000000000 1
+5 NULL 2.00000000000000000000 1
+8 1 1.00000000000000000000 2
+8 NULL 1.00000000000000000000 2
+NULL 1 2.00000000000000000000 5
+NULL 2 5.20000000000000000000 5
+NULL 3 5.00000000000000000000 2
+NULL NULL 3.83333333333333333333 12
http://git-wip-us.apache.org/repos/asf/hive/blob/5cb8867b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
index c21a4fc..d90ebf0 100644
--- a/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
@@ -822,9 +822,9 @@ STAGE PLANS:
projectedOutputColumnNums: [0, 1, 2, 3]
Statistics: Num rows: 6 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: sum(_col1), sum(_col2), avg(_col2), sum(_col3), avg(_col3), sum(_col4), avg(_col4)
+ aggregations: sum(_col1), sum(_col2), count(_col2), sum(_col3), count(_col3), sum(_col4), count(_col4)
Group By Vectorization:
- aggregators: VectorUDAFSumLong(col 0:int) -> bigint, VectorUDAFSumLong(col 1:int) -> bigint, VectorUDAFAvgLong(col 1:int) -> struct<count:bigint,sum:double,input:int>, VectorUDAFSumDouble(col 2:double) -> double, VectorUDAFAvgDouble(col 2:double) -> struct<count:bigint,sum:double,input:double>, VectorUDAFSumDecimal(col 3:decimal(38,18)) -> decimal(38,18), VectorUDAFAvgDecimal(col 3:decimal(38,18)) -> struct<count:bigint,sum:decimal(38,18),input:decimal(38,18)>
+ aggregators: VectorUDAFSumLong(col 0:int) -> bigint, VectorUDAFSumLong(col 1:int) -> bigint, VectorUDAFCount(col 1:int) -> bigint, VectorUDAFSumDouble(col 2:double) -> double, VectorUDAFCount(col 2:double) -> bigint, VectorUDAFSumDecimal(col 3:decimal(38,18)) -> decimal(38,18), VectorUDAFCount(col 3:decimal(38,18)) -> bigint
className: VectorGroupByOperator
groupByMode: HASH
keyExpressions: ConstantVectorExpression(val 1) -> 4:int
@@ -834,7 +834,7 @@ STAGE PLANS:
keys: 1 (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
- Statistics: Num rows: 1 Data size: 584 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 164 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
@@ -843,8 +843,8 @@ STAGE PLANS:
className: VectorReduceSinkLongOperator
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- Statistics: Num rows: 1 Data size: 584 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint), _col3 (type: struct<count:bigint,sum:double,input:int>), _col4 (type: double), _col5 (type: struct<count:bigint,sum:double,input:double>), _col6 (type: decimal(38,18)), _col7 (type: struct<count:bigint,sum:decimal(38,18),input:decimal(38,18)>)
+ Statistics: Num rows: 1 Data size: 164 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint), _col3 (type: bigint), _col4 (type: double), _col5 (type: bigint), _col6 (type: decimal(38,18)), _col7 (type: bigint)
Reducer 3
Execution mode: vectorized, llap
Reduce Vectorization:
@@ -855,9 +855,9 @@ STAGE PLANS:
vectorized: true
Reduce Operator Tree:
Group By Operator
- aggregations: sum(VALUE._col0), sum(VALUE._col1), avg(VALUE._col2), sum(VALUE._col3), avg(VALUE._col4), sum(VALUE._col5), avg(VALUE._col6)
+ aggregations: sum(VALUE._col0), sum(VALUE._col1), count(VALUE._col2), sum(VALUE._col3), count(VALUE._col4), sum(VALUE._col5), count(VALUE._col6)
Group By Vectorization:
- aggregators: VectorUDAFSumLong(col 1:bigint) -> bigint, VectorUDAFSumLong(col 2:bigint) -> bigint, VectorUDAFAvgFinal(col 3:struct<count:bigint,sum:double,input:int>) -> double, VectorUDAFSumDouble(col 4:double) -> double, VectorUDAFAvgFinal(col 5:struct<count:bigint,sum:double,input:double>) -> double, VectorUDAFSumDecimal(col 6:decimal(38,18)) -> decimal(38,18), VectorUDAFAvgDecimalFinal(col 7:struct<count:bigint,sum:decimal(38,18),input:decimal(38,18)>) -> decimal(38,18)
+ aggregators: VectorUDAFSumLong(col 1:bigint) -> bigint, VectorUDAFSumLong(col 2:bigint) -> bigint, VectorUDAFCountMerge(col 3:bigint) -> bigint, VectorUDAFSumDouble(col 4:double) -> double, VectorUDAFCountMerge(col 5:bigint) -> bigint, VectorUDAFSumDecimal(col 6:decimal(38,18)) -> decimal(38,18), VectorUDAFCountMerge(col 7:bigint) -> bigint
className: VectorGroupByOperator
groupByMode: MERGEPARTIAL
keyExpressions: col 0:int
@@ -867,15 +867,15 @@ STAGE PLANS:
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
- Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 164 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: 1 (type: int), _col1 (type: bigint), _col2 (type: bigint), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col6 (type: decimal(38,18)), _col7 (type: decimal(38,18))
+ expressions: 1 (type: int), _col1 (type: bigint), _col2 (type: bigint), (UDFToDouble(_col2) / _col3) (type: double), _col4 (type: double), (_col4 / _col5) (type: double), _col6 (type: decimal(38,18)), (_col6 / _col7) (type: decimal(38,18))
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [8, 1, 2, 3, 4, 5, 6, 7]
- selectExpressions: ConstantVectorExpression(val 1) -> 8:int
+ projectedOutputColumnNums: [8, 1, 2, 10, 4, 9, 6, 12]
+ selectExpressions: ConstantVectorExpression(val 1) -> 8:int, DoubleColDivideLongColumn(col 9:double, col 3:bigint)(children: CastLongToDouble(col 2:bigint) -> 9:double) -> 10:double, DoubleColDivideLongColumn(col 4:double, col 5:bigint) -> 9:double, DecimalColDivideDecimalColumn(col 6:decimal(38,18), col 11:decimal(19,0))(children: CastLongToDecimal(col 7:bigint) -> 11:decimal(19,0)) -> 12:decimal(38,18)
Statistics: Num rows: 1 Data size: 268 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
@@ -1046,9 +1046,9 @@ STAGE PLANS:
projectedOutputColumnNums: [1, 0, 2, 3, 4]
Statistics: Num rows: 500 Data size: 66000 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: sum(_col2), avg(_col2), sum(_col3), avg(_col3), sum(_col4), avg(_col4)
+ aggregations: sum(_col2), count(_col2), sum(_col3), count(_col3), sum(_col4), count(_col4)
Group By Vectorization:
- aggregators: VectorUDAFSumLong(col 2:int) -> bigint, VectorUDAFAvgLongComplete(col 2:int) -> double, VectorUDAFSumDouble(col 3:double) -> double, VectorUDAFAvgDoubleComplete(col 3:double) -> double, VectorUDAFSumDecimal(col 4:decimal(38,18)) -> decimal(38,18), VectorUDAFAvgDecimalComplete(col 4:decimal(38,18)) -> decimal(38,18)
+ aggregators: VectorUDAFSumLong(col 2:int) -> bigint, VectorUDAFCount(col 2:int) -> bigint, VectorUDAFSumDouble(col 3:double) -> double, VectorUDAFCount(col 3:double) -> bigint, VectorUDAFSumDecimal(col 4:decimal(38,18)) -> decimal(38,18), VectorUDAFCount(col 4:decimal(38,18)) -> bigint
className: VectorGroupByOperator
groupByMode: COMPLETE
keyExpressions: col 0:int, col 1:int
@@ -1058,15 +1058,16 @@ STAGE PLANS:
keys: _col1 (type: int), _col0 (type: int)
mode: complete
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
- Statistics: Num rows: 500 Data size: 290000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 80000 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col0 (type: int), _col1 (type: int), _col2 (type: bigint), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col6 (type: decimal(38,18)), _col7 (type: decimal(38,18))
+ expressions: _col0 (type: int), _col1 (type: int), _col2 (type: bigint), (UDFToDouble(_col2) / _col3) (type: double), _col4 (type: double), (_col4 / _col5) (type: double), _col6 (type: decimal(38,18)), (_col6 / _col7) (type: decimal(38,18))
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7]
- Statistics: Num rows: 500 Data size: 290000 Basic stats: COMPLETE Column stats: COMPLETE
+ projectedOutputColumnNums: [0, 1, 2, 9, 4, 8, 6, 11]
+ selectExpressions: DoubleColDivideLongColumn(col 8:double, col 3:bigint)(children: CastLongToDouble(col 2:bigint) -> 8:double) -> 9:double, DoubleColDivideLongColumn(col 4:double, col 5:bigint) -> 8:double, DecimalColDivideDecimalColumn(col 6:decimal(38,18), col 10:decimal(19,0))(children: CastLongToDecimal(col 7:bigint) -> 10:decimal(19,0)) -> 11:decimal(38,18)
+ Statistics: Num rows: 500 Data size: 132000 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type: int)
sort order: ++
@@ -1074,7 +1075,7 @@ STAGE PLANS:
className: VectorReduceSinkObjectHashOperator
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- Statistics: Num rows: 500 Data size: 290000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 132000 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: bigint), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col6 (type: decimal(38,18)), _col7 (type: decimal(38,18))
Reducer 3
Execution mode: vectorized, llap
@@ -1092,13 +1093,13 @@ STAGE PLANS:
className: VectorSelectOperator
native: true
projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7]
- Statistics: Num rows: 500 Data size: 290000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 132000 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
File Sink Vectorization:
className: VectorFileSinkOperator
native: false
- Statistics: Num rows: 500 Data size: 290000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 132000 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
http://git-wip-us.apache.org/repos/asf/hive/blob/5cb8867b/ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out b/ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out
index 8d9ffb8..c0cb6c0 100644
--- a/ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_ptf_1.q.out
@@ -79,18 +79,18 @@ STAGE PLANS:
projectedOutputColumnNums: [0, 1, 2]
Statistics: Num rows: 100 Data size: 18816 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(gpa)
+ aggregations: sum(gpa), count(gpa)
Group By Vectorization:
- aggregators: VectorUDAFAvgDouble(col 2:double) -> struct<count:bigint,sum:double,input:double>
+ aggregators: VectorUDAFSumDouble(col 2:double) -> double, VectorUDAFCount(col 2:double) -> bigint
className: VectorGroupByOperator
groupByMode: HASH
keyExpressions: col 0:string, col 1:int
native: false
vectorProcessingMode: HASH
- projectedOutputColumnNums: [0]
+ projectedOutputColumnNums: [0, 1]
keys: name (type: string), age (type: int)
mode: hash
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 100 Data size: 18816 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: int)
@@ -102,9 +102,9 @@ STAGE PLANS:
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
partitionColumnNums: [0]
- valueColumnNums: [2]
+ valueColumnNums: [2, 3]
Statistics: Num rows: 100 Data size: 18816 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: struct<count:bigint,sum:double,input:double>)
+ value expressions: _col2 (type: double), _col3 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: no inputs
Map Vectorization:
@@ -132,20 +132,20 @@ STAGE PLANS:
vectorized: false
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0)
+ aggregations: sum(VALUE._col0), count(VALUE._col1)
keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 50 Data size: 9408 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col1 (type: int), _col0 (type: string), _col2 (type: double)
- outputColumnNames: _col0, _col1, _col2
+ expressions: _col1 (type: int), _col0 (type: string), _col2 (type: double), _col3 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 50 Data size: 9408 Basic stats: COMPLETE Column stats: NONE
PTF Operator
Function definitions:
Input definition
input alias: ptf_0
- output shape: _col0: int, _col1: string, _col2: double
+ output shape: _col0: int, _col1: string, _col2: double, _col3: bigint
type: WINDOWING
Windowing table definition
input alias: ptf_1
@@ -162,7 +162,7 @@ STAGE PLANS:
window frame: ROWS PRECEDING(MAX)~FOLLOWING(MAX)
Statistics: Num rows: 50 Data size: 9408 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: int), _col1 (type: string), _col2 (type: double), sum_window_0 (type: bigint)
+ expressions: _col0 (type: int), _col1 (type: string), (_col2 / _col3) (type: double), sum_window_0 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 50 Data size: 9408 Basic stats: COMPLETE Column stats: NONE
File Output Operator