You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by jc...@apache.org on 2018/03/22 16:57:50 UTC
[27/34] hive git commit: HIVE-18979: Enable
AggregateReduceFunctionsRule from Calcite (Jesus Camacho Rodriguez,
reviewed by Ashutosh Chauhan)
http://git-wip-us.apache.org/repos/asf/hive/blob/5cb8867b/ql/src/test/results/clientpositive/llap/vector_reuse_scratchcols.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_reuse_scratchcols.q.out b/ql/src/test/results/clientpositive/llap/vector_reuse_scratchcols.q.out
index 9eaf293..8fb0752 100644
--- a/ql/src/test/results/clientpositive/llap/vector_reuse_scratchcols.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_reuse_scratchcols.q.out
@@ -104,25 +104,26 @@ STAGE PLANS:
predicate: (((CAST( cbigint AS decimal(22,3)) <= -1.389) and (cstring2 <> 'a') and (CAST( cint AS decimal(13,3)) <> 79.553) and (cboolean2 <> cboolean1)) or ((UDFToFloat(csmallint) < cfloat) and (UDFToDouble(ctimestamp2) > -5.0D) and (cdouble <> UDFToDouble(cint))) or (cbigint = 762L) or (cstring1 = 'a')) (type: boolean)
Statistics: Num rows: 5465 Data size: 1157230 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cfloat (type: float), cdouble (type: double)
- outputColumnNames: ctinyint, csmallint, cint, cfloat, cdouble
+ expressions: cint (type: int), cdouble (type: double), csmallint (type: smallint), cfloat (type: float), ctinyint (type: tinyint), UDFToDouble(cint) (type: double), (UDFToDouble(cint) * UDFToDouble(cint)) (type: double), UDFToDouble(csmallint) (type: double), (UDFToDouble(csmallint) * UDFToDouble(csmallint)) (type: double)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [0, 1, 2, 4, 5]
+ projectedOutputColumnNums: [2, 5, 1, 4, 0, 13, 18, 16, 20]
+ selectExpressions: CastLongToDouble(col 2:int) -> 13:double, DoubleColMultiplyDoubleColumn(col 16:double, col 17:double)(children: CastLongToDouble(col 2:int) -> 16:double, CastLongToDouble(col 2:int) -> 17:double) -> 18:double, CastLongToDouble(col 1:smallint) -> 16:double, DoubleColMultiplyDoubleColumn(col 17:double, col 19:double)(children: CastLongToDouble(col 1:smallint) -> 17:double, CastLongToDouble(col 1:smallint) -> 19:double) -> 20:double
Statistics: Num rows: 5465 Data size: 1157230 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: avg(cint), sum(cdouble), stddev_pop(cint), stddev_samp(csmallint), var_samp(cint), avg(cfloat), stddev_samp(cint), min(ctinyint), count(csmallint)
+ aggregations: sum(_col0), count(_col0), sum(_col1), sum(_col6), sum(_col5), sum(_col8), sum(_col7), count(_col2), sum(_col3), count(_col3), min(_col4)
Group By Vectorization:
- aggregators: VectorUDAFAvgLong(col 2:int) -> struct<count:bigint,sum:double,input:int>, VectorUDAFSumDouble(col 5:double) -> double, VectorUDAFVarLong(col 2:int) -> struct<count:bigint,sum:double,variance:double> aggregation: stddev_pop, VectorUDAFVarLong(col 1:smallint) -> struct<count:bigint,sum:double,variance:double> aggregation: stddev_samp, VectorUDAFVarLong(col 2:int) -> struct<count:bigint,sum:double,variance:double> aggregation: var_samp, VectorUDAFAvgDouble(col 4:float) -> struct<count:bigint,sum:double,input:float>, VectorUDAFVarLong(col 2:int) -> struct<count:bigint,sum:double,variance:double> aggregation: stddev_samp, VectorUDAFMinLong(col 0:tinyint) -> tinyint, VectorUDAFCount(col 1:smallint) -> bigint
+ aggregators: VectorUDAFSumLong(col 2:int) -> bigint, VectorUDAFCount(col 2:int) -> bigint, VectorUDAFSumDouble(col 5:double) -> double, VectorUDAFSumDouble(col 18:double) -> double, VectorUDAFSumDouble(col 13:double) -> double, VectorUDAFSumDouble(col 20:double) -> double, VectorUDAFSumDouble(col 16:double) -> double, VectorUDAFCount(col 1:smallint) -> bigint, VectorUDAFSumDouble(col 4:float) -> double, VectorUDAFCount(col 4:float) -> bigint, VectorUDAFMinLong(col 0:tinyint) -> tinyint
className: VectorGroupByOperator
groupByMode: HASH
native: false
vectorProcessingMode: HASH
- projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8]
+ projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 1 Data size: 492 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
+ Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
sort order:
Reduce Sink Vectorization:
@@ -130,9 +131,9 @@ STAGE PLANS:
keyColumnNums: []
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- valueColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8]
- Statistics: Num rows: 1 Data size: 492 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: struct<count:bigint,sum:double,input:int>), _col1 (type: double), _col2 (type: struct<count:bigint,sum:double,variance:double>), _col3 (type: struct<count:bigint,sum:double,variance:double>), _col4 (type: struct<count:bigint,sum:double,variance:double>), _col5 (type: struct<count:bigint,sum:double,input:float>), _col6 (type: struct<count:bigint,sum:double,variance:double>), _col7 (type: tinyint), _col8 (type: bigint)
+ valueColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+ Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint), _col2 (type: double), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col6 (type: double), _col7 (type: bigint), _col8 (type: double), _col9 (type: bigint), _col10 (type: tinyint)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -149,26 +150,49 @@ STAGE PLANS:
includeColumns: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11]
dataColumns: ctinyint:tinyint, csmallint:smallint, cint:int, cbigint:bigint, cfloat:float, cdouble:double, cstring1:string, cstring2:string, ctimestamp1:timestamp, ctimestamp2:timestamp, cboolean1:boolean, cboolean2:boolean
partitionColumnCount: 0
- scratchColumnTypeNames: [double, decimal(22,3), decimal(13,3)]
+ scratchColumnTypeNames: [double, decimal(22,3), decimal(13,3), double, double, double, double, double]
Reducer 2
- Execution mode: llap
+ Execution mode: vectorized, llap
Reduce Vectorization:
enabled: true
enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
- notVectorizedReason: GROUPBY operator: Vector aggregation : "stddev_pop" for input type: "STRUCT" and output type: "DOUBLE" and mode: FINAL not supported for evaluator GenericUDAFStdEvaluator
- vectorized: false
+ reduceColumnNullOrder:
+ reduceColumnSortOrder:
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
+ rowBatchContext:
+ dataColumnCount: 11
+ dataColumns: VALUE._col0:bigint, VALUE._col1:bigint, VALUE._col2:double, VALUE._col3:double, VALUE._col4:double, VALUE._col5:double, VALUE._col6:double, VALUE._col7:bigint, VALUE._col8:double, VALUE._col9:bigint, VALUE._col10:tinyint
+ partitionColumnCount: 0
+ scratchColumnTypeNames: []
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0), sum(VALUE._col1), stddev_pop(VALUE._col2), stddev_samp(VALUE._col3), var_samp(VALUE._col4), avg(VALUE._col5), stddev_samp(VALUE._col6), min(VALUE._col7), count(VALUE._col8)
+ aggregations: sum(VALUE._col0), count(VALUE._col1), sum(VALUE._col2), sum(VALUE._col3), sum(VALUE._col4), sum(VALUE._col5), sum(VALUE._col6), count(VALUE._col7), sum(VALUE._col8), count(VALUE._col9), min(VALUE._col10)
+ Group By Vectorization:
+ aggregators: VectorUDAFSumLong(col 0:bigint) -> bigint, VectorUDAFCountMerge(col 1:bigint) -> bigint, VectorUDAFSumDouble(col 2:double) -> double, VectorUDAFSumDouble(col 3:double) -> double, VectorUDAFSumDouble(col 4:double) -> double, VectorUDAFSumDouble(col 5:double) -> double, VectorUDAFSumDouble(col 6:double) -> double, VectorUDAFCountMerge(col 7:bigint) -> bigint, VectorUDAFSumDouble(col 8:double) -> double, VectorUDAFCountMerge(col 9:bigint) -> bigint, VectorUDAFMinLong(col 10:tinyint) -> tinyint
+ className: VectorGroupByOperator
+ groupByMode: MERGEPARTIAL
+ native: false
+ vectorProcessingMode: GLOBAL
+ projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
mode: mergepartial
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 1 Data size: 68 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
+ Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col0 (type: double), (_col0 + -3728.0D) (type: double), (- (_col0 + -3728.0D)) (type: double), (- (- (_col0 + -3728.0D))) (type: double), ((- (- (_col0 + -3728.0D))) * (_col0 + -3728.0D)) (type: double), _col1 (type: double), (- _col0) (type: double), _col2 (type: double), (((- (- (_col0 + -3728.0D))) * (_col0 + -3728.0D)) * (- (- (_col0 + -3728.0D)))) (type: double), _col3 (type: double), (- _col2) (type: double), (_col2 - (- (- (_col0 + -3728.0D)))) (type: double), ((_col2 - (- (- (_col0 + -3728.0D)))) * _col2) (type: double), _col4 (type: double), _col5 (type: double), (10.175D - _col4) (type: double), (- (10.175D - _col4)) (type: double), ((- _col2) / -563.0D) (type: double), _col6 (type: double), (- ((- _col2) / -563.0D)) (type: double), (_col0 / _col1) (type: double), _col7 (type: tinyint), _col8 (type: bigint), (UDFToDouble(_col7) / ((- _col2) / -563.0D)) (type: double), (- (_col0 / _col1)) (type: double)
+ expressions: (_col0 / _col1) (type: double), ((_col0 / _col1) + -3728.0D) (type: double), (- ((_col0 / _col1) + -3728.0D)) (type: double), (- (- ((_col0 / _col1) + -3728.0D))) (type: double), ((- (- ((_col0 / _col1) + -3728.0D))) * ((_col0 / _col1) + -3728.0D)) (type: double), _col2 (type: double), (- (_col0 / _col1)) (type: double), power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5) (type: double), (((- (- ((_col0 / _col1) + -3728.0D))) * ((_col0 / _col1) + -3728.0D)) * (- (- ((_col0 / _col1) + -3728.0D)))) (type: double), power(((_col5 - ((_col6 * _col6) / _col7)) / CASE WHEN ((_col7 = 1L)) THEN (null) ELSE ((_col7 - 1)) END), 0.5) (type: double), (- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) (type: double), (power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5) - (- (- ((_col0 / _col1) + -3728.0D)))) (type: double), ((power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5) - (- (- ((_col0 / _col1) + -3728.0D)))) * power(((_col3 - ((_
col4 * _col4) / _col1)) / _col1), 0.5)) (type: double), ((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END) (type: double), (_col8 / _col9) (type: double), (10.175D - ((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END)) (type: double), (- (10.175D - ((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END))) (type: double), ((- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) / -563.0D) (type: double), power(((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END), 0.5) (type: double), (- ((- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) / -563.0D)) (type: double), ((_col0 / _col1) / _col2) (type: double), _col10 (type: tinyint), _col7 (type: bigint), (UDFToDouble(_col10) / ((- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) / -563.0D)) (type: double), (- ((_col0 / _col1)
/ _col2)) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumnNums: [11, 13, 12, 15, 14, 2, 17, 16, 19, 18, 24, 25, 27, 26, 20, 30, 34, 31, 37, 41, 42, 10, 7, 44, 38]
+ selectExpressions: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 11:double, DoubleColAddDoubleScalar(col 12:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 12:double) -> 13:double, DoubleColUnaryMinus(col 14:double)(children: DoubleColAddDoubleScalar(col 12:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 12:double) -> 14:double) -> 12:double, DoubleColUnaryMinus(col 14:double)(children: DoubleColUnaryMinus(col 15:double)(children: DoubleColAddDoubleScalar(col 14:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 14:double) -> 15:double) -> 14:double) -> 15:double, DoubleColMultiplyDoubleColumn(col 16:double, col 17:double)(children: DoubleColUnaryMinus(col 14:double)(children: DoubleColUnaryMinus(col 16:double)(children: DoubleColAddDoubleScalar(col 14:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 14:dou
ble) -> 16:double) -> 14:double) -> 16:double, DoubleColAddDoubleScalar(col 14:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 14:double) -> 17:double) -> 14:double, DoubleColUnaryMinus(col 16:double)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 16:double) -> 17:double, FuncPowerDoubleToDouble(col 18:double)(children: DoubleColDivideLongColumn(col 16:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 18:double)(children: DoubleColDivideLongColumn(col 16:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 16:double) -> 18:double) -> 16:double) -> 18:double) -> 16:double, DoubleColMultiplyDoubleColumn(col 18:double, col 20:double)(children: DoubleColMultiplyDoubleColumn(col 19:double, col 20:double)(children: DoubleColUnaryMinus(col 18:double)(children: DoubleColUnaryMinus(col 19:double)(children: DoubleColAddDoubleScalar(col 18:double, val -3728.0)(children:
LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 18:double) -> 19:double) -> 18:double) -> 19:double, DoubleColAddDoubleScalar(col 18:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 18:double) -> 20:double) -> 18:double, DoubleColUnaryMinus(col 19:double)(children: DoubleColUnaryMinus(col 20:double)(children: DoubleColAddDoubleScalar(col 19:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 19:double) -> 20:double) -> 19:double) -> 20:double) -> 19:double, FuncPowerDoubleToDouble(col 20:double)(children: DoubleColDivideLongColumn(col 18:double, col 23:bigint)(children: DoubleColSubtractDoubleColumn(col 5:double, col 20:double)(children: DoubleColDivideLongColumn(col 18:double, col 7:bigint)(children: DoubleColMultiplyDoubleColumn(col 6:double, col 6:double) -> 18:double) -> 20:double) -> 18:double, IfExprNullCondExpr(col 21:boolean, null, col 22:bigint)(children: LongColEqualLongScalar(col 7:bigint, val
1) -> 21:boolean, LongColSubtractLongScalar(col 7:bigint, val 1) -> 22:bigint) -> 23:bigint) -> 20:double) -> 18:double, DoubleColUnaryMinus(col 20:double)(children: FuncPowerDoubleToDouble(col 24:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 24:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 20:double) -> 24:double) -> 20:double) -> 24:double) -> 20:double) -> 24:double, DoubleColSubtractDoubleColumn(col 20:double, col 26:double)(children: FuncPowerDoubleToDouble(col 25:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 25:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 20:double) -> 25:double) -> 20:double) -> 25:double) -> 20:dou
ble, DoubleColUnaryMinus(col 25:double)(children: DoubleColUnaryMinus(col 26:double)(children: DoubleColAddDoubleScalar(col 25:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 25:double) -> 26:double) -> 25:double) -> 26:double) -> 25:double, DoubleColMultiplyDoubleColumn(col 26:double, col 20:double)(children: DoubleColSubtractDoubleColumn(col 20:double, col 27:double)(children: FuncPowerDoubleToDouble(col 26:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 26:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 20:double) -> 26:double) -> 20:double) -> 26:double) -> 20:double, DoubleColUnaryMinus(col 26:double)(children: DoubleColUnaryMinus(col 27:double)(children: DoubleColAddDoubleScalar(col 26:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigi
nt) -> 26:double) -> 27:double) -> 26:double) -> 27:double) -> 26:double, FuncPowerDoubleToDouble(col 27:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 27:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 20:double) -> 27:double) -> 20:double) -> 27:double) -> 20:double) -> 27:double, DoubleColDivideLongColumn(col 20:double, col 29:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 26:double)(children: DoubleColDivideLongColumn(col 20:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 20:double) -> 26:double) -> 20:double, IfExprNullCondExpr(col 23:boolean, null, col 28:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 23:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 28:bigint) -> 29:bigint) -> 26:double, DoubleColDivideLong
Column(col 8:double, col 9:bigint) -> 20:double, DoubleScalarSubtractDoubleColumn(val 10.175, col 31:double)(children: DoubleColDivideLongColumn(col 30:double, col 33:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 31:double)(children: DoubleColDivideLongColumn(col 30:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 30:double) -> 31:double) -> 30:double, IfExprNullCondExpr(col 29:boolean, null, col 32:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 29:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 32:bigint) -> 33:bigint) -> 31:double) -> 30:double, DoubleColUnaryMinus(col 31:double)(children: DoubleScalarSubtractDoubleColumn(val 10.175, col 34:double)(children: DoubleColDivideLongColumn(col 31:double, col 36:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 34:double)(children: DoubleColDivideLongColumn(col 31:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4
:double, col 4:double) -> 31:double) -> 34:double) -> 31:double, IfExprNullCondExpr(col 33:boolean, null, col 35:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 33:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 35:bigint) -> 36:bigint) -> 34:double) -> 31:double) -> 34:double, DoubleColDivideDoubleScalar(col 37:double, val -563.0)(children: DoubleColUnaryMinus(col 31:double)(children: FuncPowerDoubleToDouble(col 37:double)(children: DoubleColDivideLongColumn(col 31:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 37:double)(children: DoubleColDivideLongColumn(col 31:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 31:double) -> 37:double) -> 31:double) -> 37:double) -> 31:double) -> 37:double) -> 31:double, FuncPowerDoubleToDouble(col 38:double)(children: DoubleColDivideLongColumn(col 37:double, col 40:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 38:double)(ch
ildren: DoubleColDivideLongColumn(col 37:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 37:double) -> 38:double) -> 37:double, IfExprNullCondExpr(col 36:boolean, null, col 39:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 36:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 39:bigint) -> 40:bigint) -> 38:double) -> 37:double, DoubleColUnaryMinus(col 38:double)(children: DoubleColDivideDoubleScalar(col 41:double, val -563.0)(children: DoubleColUnaryMinus(col 38:double)(children: FuncPowerDoubleToDouble(col 41:double)(children: DoubleColDivideLongColumn(col 38:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 41:double)(children: DoubleColDivideLongColumn(col 38:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 38:double) -> 41:double) -> 38:double) -> 41:double) -> 38:double) -> 41:double) -> 38:double) -> 41:double, DoubleColDivideDoubleCo
lumn(col 38:double, col 2:double)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 38:double) -> 42:double, DoubleColDivideDoubleColumn(col 38:double, col 43:double)(children: CastLongToDouble(col 10:tinyint) -> 38:double, DoubleColDivideDoubleScalar(col 44:double, val -563.0)(children: DoubleColUnaryMinus(col 43:double)(children: FuncPowerDoubleToDouble(col 44:double)(children: DoubleColDivideLongColumn(col 43:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 44:double)(children: DoubleColDivideLongColumn(col 43:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 43:double) -> 44:double) -> 43:double) -> 44:double) -> 43:double) -> 44:double) -> 43:double) -> 44:double, DoubleColUnaryMinus(col 43:double)(children: DoubleColDivideDoubleColumn(col 38:double, col 2:double)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 38:double) -> 43:double) -> 38:double
Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
@@ -287,25 +311,26 @@ STAGE PLANS:
predicate: (((CAST( cbigint AS decimal(22,3)) <= -1.389) and (cstring2 <> 'a') and (CAST( cint AS decimal(13,3)) <> 79.553) and (cboolean2 <> cboolean1)) or ((UDFToFloat(csmallint) < cfloat) and (UDFToDouble(ctimestamp2) > -5.0D) and (cdouble <> UDFToDouble(cint))) or (cbigint = 762L) or (cstring1 = 'a')) (type: boolean)
Statistics: Num rows: 5465 Data size: 1157230 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cfloat (type: float), cdouble (type: double)
- outputColumnNames: ctinyint, csmallint, cint, cfloat, cdouble
+ expressions: cint (type: int), cdouble (type: double), csmallint (type: smallint), cfloat (type: float), ctinyint (type: tinyint), UDFToDouble(cint) (type: double), (UDFToDouble(cint) * UDFToDouble(cint)) (type: double), UDFToDouble(csmallint) (type: double), (UDFToDouble(csmallint) * UDFToDouble(csmallint)) (type: double)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Select Vectorization:
className: VectorSelectOperator
native: true
- projectedOutputColumnNums: [0, 1, 2, 4, 5]
+ projectedOutputColumnNums: [2, 5, 1, 4, 0, 18, 21, 22, 25]
+ selectExpressions: CastLongToDouble(col 2:int) -> 18:double, DoubleColMultiplyDoubleColumn(col 19:double, col 20:double)(children: CastLongToDouble(col 2:int) -> 19:double, CastLongToDouble(col 2:int) -> 20:double) -> 21:double, CastLongToDouble(col 1:smallint) -> 22:double, DoubleColMultiplyDoubleColumn(col 23:double, col 24:double)(children: CastLongToDouble(col 1:smallint) -> 23:double, CastLongToDouble(col 1:smallint) -> 24:double) -> 25:double
Statistics: Num rows: 5465 Data size: 1157230 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: avg(cint), sum(cdouble), stddev_pop(cint), stddev_samp(csmallint), var_samp(cint), avg(cfloat), stddev_samp(cint), min(ctinyint), count(csmallint)
+ aggregations: sum(_col0), count(_col0), sum(_col1), sum(_col6), sum(_col5), sum(_col8), sum(_col7), count(_col2), sum(_col3), count(_col3), min(_col4)
Group By Vectorization:
- aggregators: VectorUDAFAvgLong(col 2:int) -> struct<count:bigint,sum:double,input:int>, VectorUDAFSumDouble(col 5:double) -> double, VectorUDAFVarLong(col 2:int) -> struct<count:bigint,sum:double,variance:double> aggregation: stddev_pop, VectorUDAFVarLong(col 1:smallint) -> struct<count:bigint,sum:double,variance:double> aggregation: stddev_samp, VectorUDAFVarLong(col 2:int) -> struct<count:bigint,sum:double,variance:double> aggregation: var_samp, VectorUDAFAvgDouble(col 4:float) -> struct<count:bigint,sum:double,input:float>, VectorUDAFVarLong(col 2:int) -> struct<count:bigint,sum:double,variance:double> aggregation: stddev_samp, VectorUDAFMinLong(col 0:tinyint) -> tinyint, VectorUDAFCount(col 1:smallint) -> bigint
+ aggregators: VectorUDAFSumLong(col 2:int) -> bigint, VectorUDAFCount(col 2:int) -> bigint, VectorUDAFSumDouble(col 5:double) -> double, VectorUDAFSumDouble(col 21:double) -> double, VectorUDAFSumDouble(col 18:double) -> double, VectorUDAFSumDouble(col 25:double) -> double, VectorUDAFSumDouble(col 22:double) -> double, VectorUDAFCount(col 1:smallint) -> bigint, VectorUDAFSumDouble(col 4:float) -> double, VectorUDAFCount(col 4:float) -> bigint, VectorUDAFMinLong(col 0:tinyint) -> tinyint
className: VectorGroupByOperator
groupByMode: HASH
native: false
vectorProcessingMode: HASH
- projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8]
+ projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 1 Data size: 492 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
+ Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
sort order:
Reduce Sink Vectorization:
@@ -313,9 +338,9 @@ STAGE PLANS:
keyColumnNums: []
native: true
nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, No PTF TopN IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- valueColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8]
- Statistics: Num rows: 1 Data size: 492 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: struct<count:bigint,sum:double,input:int>), _col1 (type: double), _col2 (type: struct<count:bigint,sum:double,variance:double>), _col3 (type: struct<count:bigint,sum:double,variance:double>), _col4 (type: struct<count:bigint,sum:double,variance:double>), _col5 (type: struct<count:bigint,sum:double,input:float>), _col6 (type: struct<count:bigint,sum:double,variance:double>), _col7 (type: tinyint), _col8 (type: bigint)
+ valueColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
+ Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint), _col2 (type: double), _col3 (type: double), _col4 (type: double), _col5 (type: double), _col6 (type: double), _col7 (type: bigint), _col8 (type: double), _col9 (type: bigint), _col10 (type: tinyint)
Execution mode: vectorized, llap
LLAP IO: all inputs
Map Vectorization:
@@ -332,26 +357,49 @@ STAGE PLANS:
includeColumns: [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 11]
dataColumns: ctinyint:tinyint, csmallint:smallint, cint:int, cbigint:bigint, cfloat:float, cdouble:double, cstring1:string, cstring2:string, ctimestamp1:timestamp, ctimestamp2:timestamp, cboolean1:boolean, cboolean2:boolean
partitionColumnCount: 0
- scratchColumnTypeNames: [double, double, double, decimal(22,3), decimal(13,3)]
+ scratchColumnTypeNames: [double, double, double, decimal(22,3), decimal(13,3), double, double, double, double, double, double, double, double]
Reducer 2
- Execution mode: llap
+ Execution mode: vectorized, llap
Reduce Vectorization:
enabled: true
enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
- notVectorizedReason: GROUPBY operator: Vector aggregation : "stddev_pop" for input type: "STRUCT" and output type: "DOUBLE" and mode: FINAL not supported for evaluator GenericUDAFStdEvaluator
- vectorized: false
+ reduceColumnNullOrder:
+ reduceColumnSortOrder:
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
+ rowBatchContext:
+ dataColumnCount: 11
+ dataColumns: VALUE._col0:bigint, VALUE._col1:bigint, VALUE._col2:double, VALUE._col3:double, VALUE._col4:double, VALUE._col5:double, VALUE._col6:double, VALUE._col7:bigint, VALUE._col8:double, VALUE._col9:bigint, VALUE._col10:tinyint
+ partitionColumnCount: 0
+ scratchColumnTypeNames: []
Reduce Operator Tree:
Group By Operator
- aggregations: avg(VALUE._col0), sum(VALUE._col1), stddev_pop(VALUE._col2), stddev_samp(VALUE._col3), var_samp(VALUE._col4), avg(VALUE._col5), stddev_samp(VALUE._col6), min(VALUE._col7), count(VALUE._col8)
+ aggregations: sum(VALUE._col0), count(VALUE._col1), sum(VALUE._col2), sum(VALUE._col3), sum(VALUE._col4), sum(VALUE._col5), sum(VALUE._col6), count(VALUE._col7), sum(VALUE._col8), count(VALUE._col9), min(VALUE._col10)
+ Group By Vectorization:
+ aggregators: VectorUDAFSumLong(col 0:bigint) -> bigint, VectorUDAFCountMerge(col 1:bigint) -> bigint, VectorUDAFSumDouble(col 2:double) -> double, VectorUDAFSumDouble(col 3:double) -> double, VectorUDAFSumDouble(col 4:double) -> double, VectorUDAFSumDouble(col 5:double) -> double, VectorUDAFSumDouble(col 6:double) -> double, VectorUDAFCountMerge(col 7:bigint) -> bigint, VectorUDAFSumDouble(col 8:double) -> double, VectorUDAFCountMerge(col 9:bigint) -> bigint, VectorUDAFMinLong(col 10:tinyint) -> tinyint
+ className: VectorGroupByOperator
+ groupByMode: MERGEPARTIAL
+ native: false
+ vectorProcessingMode: GLOBAL
+ projectedOutputColumnNums: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
mode: mergepartial
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 1 Data size: 68 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
+ Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col0 (type: double), (_col0 + -3728.0D) (type: double), (- (_col0 + -3728.0D)) (type: double), (- (- (_col0 + -3728.0D))) (type: double), ((- (- (_col0 + -3728.0D))) * (_col0 + -3728.0D)) (type: double), _col1 (type: double), (- _col0) (type: double), _col2 (type: double), (((- (- (_col0 + -3728.0D))) * (_col0 + -3728.0D)) * (- (- (_col0 + -3728.0D)))) (type: double), _col3 (type: double), (- _col2) (type: double), (_col2 - (- (- (_col0 + -3728.0D)))) (type: double), ((_col2 - (- (- (_col0 + -3728.0D)))) * _col2) (type: double), _col4 (type: double), _col5 (type: double), (10.175D - _col4) (type: double), (- (10.175D - _col4)) (type: double), ((- _col2) / -563.0D) (type: double), _col6 (type: double), (- ((- _col2) / -563.0D)) (type: double), (_col0 / _col1) (type: double), _col7 (type: tinyint), _col8 (type: bigint), (UDFToDouble(_col7) / ((- _col2) / -563.0D)) (type: double), (- (_col0 / _col1)) (type: double)
+ expressions: (_col0 / _col1) (type: double), ((_col0 / _col1) + -3728.0D) (type: double), (- ((_col0 / _col1) + -3728.0D)) (type: double), (- (- ((_col0 / _col1) + -3728.0D))) (type: double), ((- (- ((_col0 / _col1) + -3728.0D))) * ((_col0 / _col1) + -3728.0D)) (type: double), _col2 (type: double), (- (_col0 / _col1)) (type: double), power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5) (type: double), (((- (- ((_col0 / _col1) + -3728.0D))) * ((_col0 / _col1) + -3728.0D)) * (- (- ((_col0 / _col1) + -3728.0D)))) (type: double), power(((_col5 - ((_col6 * _col6) / _col7)) / CASE WHEN ((_col7 = 1L)) THEN (null) ELSE ((_col7 - 1)) END), 0.5) (type: double), (- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) (type: double), (power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5) - (- (- ((_col0 / _col1) + -3728.0D)))) (type: double), ((power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5) - (- (- ((_col0 / _col1) + -3728.0D)))) * power(((_col3 - ((_
col4 * _col4) / _col1)) / _col1), 0.5)) (type: double), ((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END) (type: double), (_col8 / _col9) (type: double), (10.175D - ((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END)) (type: double), (- (10.175D - ((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END))) (type: double), ((- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) / -563.0D) (type: double), power(((_col3 - ((_col4 * _col4) / _col1)) / CASE WHEN ((_col1 = 1L)) THEN (null) ELSE ((_col1 - 1)) END), 0.5) (type: double), (- ((- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) / -563.0D)) (type: double), ((_col0 / _col1) / _col2) (type: double), _col10 (type: tinyint), _col7 (type: bigint), (UDFToDouble(_col10) / ((- power(((_col3 - ((_col4 * _col4) / _col1)) / _col1), 0.5)) / -563.0D)) (type: double), (- ((_col0 / _col1)
/ _col2)) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumnNums: [11, 13, 16, 20, 27, 2, 29, 34, 46, 54, 60, 70, 86, 93, 94, 102, 111, 118, 126, 134, 136, 10, 7, 145, 148]
+ selectExpressions: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 11:double, DoubleColAddDoubleScalar(col 12:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 12:double) -> 13:double, DoubleColUnaryMinus(col 15:double)(children: DoubleColAddDoubleScalar(col 14:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 14:double) -> 15:double) -> 16:double, DoubleColUnaryMinus(col 19:double)(children: DoubleColUnaryMinus(col 18:double)(children: DoubleColAddDoubleScalar(col 17:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 17:double) -> 18:double) -> 19:double) -> 20:double, DoubleColMultiplyDoubleColumn(col 24:double, col 26:double)(children: DoubleColUnaryMinus(col 23:double)(children: DoubleColUnaryMinus(col 22:double)(children: DoubleColAddDoubleScalar(col 21:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 21:dou
ble) -> 22:double) -> 23:double) -> 24:double, DoubleColAddDoubleScalar(col 25:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 25:double) -> 26:double) -> 27:double, DoubleColUnaryMinus(col 28:double)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 28:double) -> 29:double, FuncPowerDoubleToDouble(col 33:double)(children: DoubleColDivideLongColumn(col 32:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 31:double)(children: DoubleColDivideLongColumn(col 30:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 30:double) -> 31:double) -> 32:double) -> 33:double) -> 34:double, DoubleColMultiplyDoubleColumn(col 41:double, col 45:double)(children: DoubleColMultiplyDoubleColumn(col 38:double, col 40:double)(children: DoubleColUnaryMinus(col 37:double)(children: DoubleColUnaryMinus(col 36:double)(children: DoubleColAddDoubleScalar(col 35:double, val -3728.0)(children:
LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 35:double) -> 36:double) -> 37:double) -> 38:double, DoubleColAddDoubleScalar(col 39:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 39:double) -> 40:double) -> 41:double, DoubleColUnaryMinus(col 44:double)(children: DoubleColUnaryMinus(col 43:double)(children: DoubleColAddDoubleScalar(col 42:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 42:double) -> 43:double) -> 44:double) -> 45:double) -> 46:double, FuncPowerDoubleToDouble(col 53:double)(children: DoubleColDivideLongColumn(col 49:double, col 52:bigint)(children: DoubleColSubtractDoubleColumn(col 5:double, col 48:double)(children: DoubleColDivideLongColumn(col 47:double, col 7:bigint)(children: DoubleColMultiplyDoubleColumn(col 6:double, col 6:double) -> 47:double) -> 48:double) -> 49:double, IfExprNullCondExpr(col 50:boolean, null, col 51:bigint)(children: LongColEqualLongScalar(col 7:bigint, val
1) -> 50:boolean, LongColSubtractLongScalar(col 7:bigint, val 1) -> 51:bigint) -> 52:bigint) -> 53:double) -> 54:double, DoubleColUnaryMinus(col 59:double)(children: FuncPowerDoubleToDouble(col 58:double)(children: DoubleColDivideLongColumn(col 57:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 56:double)(children: DoubleColDivideLongColumn(col 55:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 55:double) -> 56:double) -> 57:double) -> 58:double) -> 59:double) -> 60:double, DoubleColSubtractDoubleColumn(col 65:double, col 69:double)(children: FuncPowerDoubleToDouble(col 64:double)(children: DoubleColDivideLongColumn(col 63:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 62:double)(children: DoubleColDivideLongColumn(col 61:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 61:double) -> 62:double) -> 63:double) -> 64:double) -> 65:dou
ble, DoubleColUnaryMinus(col 68:double)(children: DoubleColUnaryMinus(col 67:double)(children: DoubleColAddDoubleScalar(col 66:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 66:double) -> 67:double) -> 68:double) -> 69:double) -> 70:double, DoubleColMultiplyDoubleColumn(col 80:double, col 85:double)(children: DoubleColSubtractDoubleColumn(col 75:double, col 79:double)(children: FuncPowerDoubleToDouble(col 74:double)(children: DoubleColDivideLongColumn(col 73:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 72:double)(children: DoubleColDivideLongColumn(col 71:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 71:double) -> 72:double) -> 73:double) -> 74:double) -> 75:double, DoubleColUnaryMinus(col 78:double)(children: DoubleColUnaryMinus(col 77:double)(children: DoubleColAddDoubleScalar(col 76:double, val -3728.0)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigi
nt) -> 76:double) -> 77:double) -> 78:double) -> 79:double) -> 80:double, FuncPowerDoubleToDouble(col 84:double)(children: DoubleColDivideLongColumn(col 83:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 82:double)(children: DoubleColDivideLongColumn(col 81:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 81:double) -> 82:double) -> 83:double) -> 84:double) -> 85:double) -> 86:double, DoubleColDivideLongColumn(col 89:double, col 92:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 88:double)(children: DoubleColDivideLongColumn(col 87:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 87:double) -> 88:double) -> 89:double, IfExprNullCondExpr(col 90:boolean, null, col 91:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 90:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 91:bigint) -> 92:bigint) -> 93:double, DoubleColDivideLong
Column(col 8:double, col 9:bigint) -> 94:double, DoubleScalarSubtractDoubleColumn(val 10.175, col 101:double)(children: DoubleColDivideLongColumn(col 97:double, col 100:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 96:double)(children: DoubleColDivideLongColumn(col 95:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 95:double) -> 96:double) -> 97:double, IfExprNullCondExpr(col 98:boolean, null, col 99:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 98:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 99:bigint) -> 100:bigint) -> 101:double) -> 102:double, DoubleColUnaryMinus(col 110:double)(children: DoubleScalarSubtractDoubleColumn(val 10.175, col 109:double)(children: DoubleColDivideLongColumn(col 105:double, col 108:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 104:double)(children: DoubleColDivideLongColumn(col 103:double, col 1:bigint)(children: DoubleColMultiplyDoubleC
olumn(col 4:double, col 4:double) -> 103:double) -> 104:double) -> 105:double, IfExprNullCondExpr(col 106:boolean, null, col 107:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 106:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 107:bigint) -> 108:bigint) -> 109:double) -> 110:double) -> 111:double, DoubleColDivideDoubleScalar(col 117:double, val -563.0)(children: DoubleColUnaryMinus(col 116:double)(children: FuncPowerDoubleToDouble(col 115:double)(children: DoubleColDivideLongColumn(col 114:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 113:double)(children: DoubleColDivideLongColumn(col 112:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 112:double) -> 113:double) -> 114:double) -> 115:double) -> 116:double) -> 117:double) -> 118:double, FuncPowerDoubleToDouble(col 125:double)(children: DoubleColDivideLongColumn(col 121:double, col 124:bigint)(children: DoubleColSubtractDouble
Column(col 3:double, col 120:double)(children: DoubleColDivideLongColumn(col 119:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 119:double) -> 120:double) -> 121:double, IfExprNullCondExpr(col 122:boolean, null, col 123:bigint)(children: LongColEqualLongScalar(col 1:bigint, val 1) -> 122:boolean, LongColSubtractLongScalar(col 1:bigint, val 1) -> 123:bigint) -> 124:bigint) -> 125:double) -> 126:double, DoubleColUnaryMinus(col 133:double)(children: DoubleColDivideDoubleScalar(col 132:double, val -563.0)(children: DoubleColUnaryMinus(col 131:double)(children: FuncPowerDoubleToDouble(col 130:double)(children: DoubleColDivideLongColumn(col 129:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 128:double)(children: DoubleColDivideLongColumn(col 127:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 127:double) -> 128:double) -> 129:double) -> 130:double) -> 131:double) ->
132:double) -> 133:double) -> 134:double, DoubleColDivideDoubleColumn(col 135:double, col 2:double)(children: LongColDivideLongColumn(col 0:bigint, col 1:bigint) -> 135:double) -> 136:double, DoubleColDivideDoubleColumn(col 137:double, col 144:double)(children: CastLongToDouble(col 10:tinyint) -> 137:double, DoubleColDivideDoubleScalar(col 143:double, val -563.0)(children: DoubleColUnaryMinus(col 142:double)(children: FuncPowerDoubleToDouble(col 141:double)(children: DoubleColDivideLongColumn(col 140:double, col 1:bigint)(children: DoubleColSubtractDoubleColumn(col 3:double, col 139:double)(children: DoubleColDivideLongColumn(col 138:double, col 1:bigint)(children: DoubleColMultiplyDoubleColumn(col 4:double, col 4:double) -> 138:double) -> 139:double) -> 140:double) -> 141:double) -> 142:double) -> 143:double) -> 144:double) -> 145:double, DoubleColUnaryMinus(col 147:double)(children: DoubleColDivideDoubleColumn(col 146:double, col 2:double)(children: LongColDivideLongColumn(col 0:b
igint, col 1:bigint) -> 146:double) -> 147:double) -> 148:double
Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
http://git-wip-us.apache.org/repos/asf/hive/blob/5cb8867b/ql/src/test/results/clientpositive/llap/vector_windowing.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_windowing.q.out b/ql/src/test/results/clientpositive/llap/vector_windowing.q.out
index a821265..8e73eb0 100644
--- a/ql/src/test/results/clientpositive/llap/vector_windowing.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_windowing.q.out
@@ -2883,9 +2883,9 @@ STAGE PLANS:
window function: GenericUDAFAverageEvaluatorDouble
window frame: RANGE PRECEDING(MAX)~CURRENT
window function definition
- alias: stddev_window_7
+ alias: stddev_pop_window_7
arguments: _col5
- name: stddev
+ name: stddev_pop
window function: GenericUDAFStdEvaluator
window frame: RANGE PRECEDING(MAX)~CURRENT
window function definition
@@ -2902,15 +2902,15 @@ STAGE PLANS:
window frame: RANGE PRECEDING(MAX)~CURRENT
Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: rank_window_0 (type: int), dense_rank_window_1 (type: int), cume_dist_window_2 (type: double), percent_rank_window_3 (type: double), ntile_window_4 (type: int), count_window_5 (type: bigint), avg_window_6 (type: double), stddev_window_7 (type: double), first_value_window_8 (type: int), last_value_window_9 (type: int), _col1 (type: string), _col2 (type: string), _col5 (type: int)
- outputColumnNames: rank_window_0, dense_rank_window_1, cume_dist_window_2, percent_rank_window_3, ntile_window_4, count_window_5, avg_window_6, stddev_window_7, first_value_window_8, last_value_window_9, _col1, _col2, _col5
+ expressions: rank_window_0 (type: int), dense_rank_window_1 (type: int), cume_dist_window_2 (type: double), percent_rank_window_3 (type: double), ntile_window_4 (type: int), count_window_5 (type: bigint), avg_window_6 (type: double), stddev_pop_window_7 (type: double), first_value_window_8 (type: int), last_value_window_9 (type: int), _col1 (type: string), _col2 (type: string), _col5 (type: int)
+ outputColumnNames: rank_window_0, dense_rank_window_1, cume_dist_window_2, percent_rank_window_3, ntile_window_4, count_window_5, avg_window_6, stddev_pop_window_7, first_value_window_8, last_value_window_9, _col1, _col2, _col5
Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col2 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col2 (type: string)
Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: rank_window_0 (type: int), dense_rank_window_1 (type: int), cume_dist_window_2 (type: double), percent_rank_window_3 (type: double), ntile_window_4 (type: int), count_window_5 (type: bigint), avg_window_6 (type: double), stddev_window_7 (type: double), first_value_window_8 (type: int), last_value_window_9 (type: int), _col5 (type: int)
+ value expressions: rank_window_0 (type: int), dense_rank_window_1 (type: int), cume_dist_window_2 (type: double), percent_rank_window_3 (type: double), ntile_window_4 (type: int), count_window_5 (type: bigint), avg_window_6 (type: double), stddev_pop_window_7 (type: double), first_value_window_8 (type: int), last_value_window_9 (type: int), _col5 (type: int)
Reducer 3
Execution mode: llap
Reduce Vectorization:
@@ -4086,7 +4086,7 @@ STAGE PLANS:
Reduce Vectorization:
enabled: true
enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
- notVectorizedReason: PTF Output Columns expression for PTF operator: Data type array<int> of column collect_set_window_2 not supported
+ notVectorizedReason: PTF Output Columns expression for PTF operator: Data type array<int> of column collect_set_window_1 not supported
vectorized: false
Reduce Operator Tree:
Select Operator
@@ -4107,44 +4107,38 @@ STAGE PLANS:
raw input shape:
window functions:
window function definition
- alias: stddev_window_0
- arguments: _col7
- name: stddev
- window function: GenericUDAFStdEvaluator
- window frame: ROWS PRECEDING(2)~FOLLOWING(2)
- window function definition
- alias: stddev_pop_window_1
+ alias: stddev_pop_window_0
arguments: _col7
name: stddev_pop
window function: GenericUDAFStdEvaluator
window frame: ROWS PRECEDING(2)~FOLLOWING(2)
window function definition
- alias: collect_set_window_2
+ alias: collect_set_window_1
arguments: _col5
name: collect_set
window function: GenericUDAFMkCollectionEvaluator
window frame: ROWS PRECEDING(2)~FOLLOWING(2)
window function definition
- alias: variance_window_3
+ alias: var_pop_window_2
arguments: _col7
- name: variance
+ name: var_pop
window function: GenericUDAFVarianceEvaluator
window frame: ROWS PRECEDING(2)~FOLLOWING(2)
window function definition
- alias: corr_window_4
+ alias: corr_window_3
arguments: _col5, _col7
name: corr
window function: GenericUDAFCorrelationEvaluator
window frame: ROWS PRECEDING(2)~FOLLOWING(2)
window function definition
- alias: covar_pop_window_5
+ alias: covar_pop_window_4
arguments: _col5, _col7
name: covar_pop
window function: GenericUDAFCovarianceEvaluator
window frame: ROWS PRECEDING(2)~FOLLOWING(2)
Statistics: Num rows: 26 Data size: 12974 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), stddev_window_0 (type: double), stddev_pop_window_1 (type: double), collect_set_window_2 (type: array<int>), variance_window_3 (type: double), round(corr_window_4, 5) (type: double), covar_pop_window_5 (type: double)
+ expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int), stddev_pop_window_0 (type: double), stddev_pop_window_0 (type: double), collect_set_window_1 (type: array<int>), var_pop_window_2 (type: double), round(corr_window_3, 5) (type: double), covar_pop_window_4 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 26 Data size: 9958 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator