You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by mm...@apache.org on 2016/10/13 10:50:35 UTC
[16/51] [partial] hive git commit: HIVE-11394: Enhance EXPLAIN
display for vectorization (Matt McCline, reviewed by Gopal Vijayaraghavan)
http://git-wip-us.apache.org/repos/asf/hive/blob/f923db0b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
index 9eeb0d6..26fa9d9 100644
--- a/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
@@ -256,12 +256,16 @@ POSTHOOK: Input: default@flights_tiny_orc
2010-10-29 12
2010-10-30 11
2010-10-31 8
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select * from flights_tiny_orc sort by fl_num, fl_date limit 25
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select * from flights_tiny_orc sort by fl_num, fl_date limit 25
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -280,46 +284,102 @@ STAGE PLANS:
TableScan
alias: flights_tiny_orc
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Select Operator
expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col5 (type: int), _col2 (type: date)
sort order: ++
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkOperator
+ native: false
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col0 (type: string), _col1 (type: string), _col3 (type: timestamp), _col4 (type: float)
Execution mode: vectorized, llap
LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reducer 2
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY.reducesinkkey1 (type: date), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2, 3, 1, 4, 5, 0]
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
Limit
Number of rows: 25
+ Limit Vectorization:
+ className: VectorLimitOperator
+ native: true
Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col5 (type: int), _col2 (type: date)
sort order: ++
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkOperator
+ native: false
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false
Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col0 (type: string), _col1 (type: string), _col3 (type: timestamp), _col4 (type: float)
Reducer 3
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), KEY.reducesinkkey1 (type: date), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2, 3, 1, 4, 5, 0]
Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE
Limit
Number of rows: 25
+ Limit Vectorization:
+ className: VectorLimitOperator
+ native: true
Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 25 Data size: 7200 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
@@ -365,12 +425,16 @@ Chicago New York 2010-10-24 2010-10-24 07:00:00 113.0 897
Chicago New York 2010-10-25 2010-10-25 07:00:00 -1.0 897
Chicago New York 2010-10-26 2010-10-26 07:00:00 0.0 897
Chicago New York 2010-10-27 2010-10-27 07:00:00 -11.0 897
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select fl_date, count(*) from flights_tiny_orc group by fl_date
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select fl_date, count(*) from flights_tiny_orc group by fl_date
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -388,12 +452,26 @@ STAGE PLANS:
TableScan
alias: flights_tiny_orc
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Select Operator
expressions: fl_date (type: date)
outputColumnNames: fl_date
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2]
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
+ Group By Vectorization:
+ aggregators: VectorUDAFCountStar(*) -> bigint
+ className: VectorGroupByOperator
+ vectorOutput: true
+ keyExpressions: col 2
+ native: false
+ projectedOutputColumns: [0]
keys: fl_date (type: date)
mode: hash
outputColumnNames: _col0, _col1
@@ -402,21 +480,50 @@ STAGE PLANS:
key expressions: _col0 (type: date)
sort order: +
Map-reduce partition columns: _col0 (type: date)
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkMultiKeyOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reducer 2
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
+ Group By Vectorization:
+ aggregators: VectorUDAFCountMerge(col 1) -> bigint
+ className: VectorGroupByOperator
+ vectorOutput: true
+ keyExpressions: col 0
+ native: false
+ projectedOutputColumns: [0]
keys: KEY._col0 (type: date)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 68 Data size: 19584 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 68 Data size: 19584 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
@@ -590,17 +697,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-29
POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-30
POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-31
#### A masked pattern was here ####
-Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20
-Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20
-Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20
-Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20
-Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20
-Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20
-Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20
-Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20
-Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20
-Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20
-Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20
+Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26
+Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26
+Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26
+Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26
+Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26
+Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26
+Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26
+Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26
+Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26
Baltimore New York 2010-10-21 07:00:00 17.0 1064 2010-10-21
Baltimore New York 2010-10-21 07:00:00 105.0 1142 2010-10-21
Baltimore New York 2010-10-21 07:00:00 28.0 1599 2010-10-21
@@ -613,41 +722,6 @@ Cleveland New York 2010-10-21 07:00:00 3.0 2630 2010-10-21
Cleveland New York 2010-10-21 07:00:00 29.0 2646 2010-10-21
Cleveland New York 2010-10-21 07:00:00 72.0 3014 2010-10-21
Washington New York 2010-10-21 07:00:00 22.0 7291 2010-10-21
-Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22
-Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22
-Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22
-Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22
-Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22
-Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22
-Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22
-Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22
-Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22
-Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22
-Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22
-Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23
-Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23
-Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23
-Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23
-Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23
-Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23
-Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23
-Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23
-Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23
-Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23
-Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23
-Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23
-Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24
-Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24
-Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24
-Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24
-Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24
-Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24
-Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24
-Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24
Baltimore New York 2010-10-25 07:00:00 -25.0 1064 2010-10-25
Baltimore New York 2010-10-25 07:00:00 92.0 1142 2010-10-25
Baltimore New York 2010-10-25 07:00:00 106.0 1599 2010-10-25
@@ -660,19 +734,42 @@ Cleveland New York 2010-10-25 07:00:00 -4.0 2630 2010-10-25
Cleveland New York 2010-10-25 07:00:00 81.0 2646 2010-10-25
Cleveland New York 2010-10-25 07:00:00 42.0 3014 2010-10-25
Washington New York 2010-10-25 07:00:00 9.0 7291 2010-10-25
-Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26
-Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26
-Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26
-Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26
-Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26
-Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26
-Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26
-Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26
-Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26
+Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24
+Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24
+Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24
+Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24
+Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24
+Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24
+Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24
+Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24
+Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23
+Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23
+Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23
+Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23
+Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23
+Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23
+Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23
+Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23
+Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23
+Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23
+Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23
+Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23
+Baltimore New York 2010-10-29 07:00:00 -24.0 1064 2010-10-29
+Baltimore New York 2010-10-29 07:00:00 21.0 1142 2010-10-29
+Baltimore New York 2010-10-29 07:00:00 -2.0 1599 2010-10-29
+Chicago New York 2010-10-29 07:00:00 -12.0 361 2010-10-29
+Chicago New York 2010-10-29 07:00:00 -11.0 897 2010-10-29
+Chicago New York 2010-10-29 07:00:00 15.0 1531 2010-10-29
+Chicago New York 2010-10-29 07:00:00 -18.0 1610 2010-10-29
+Chicago New York 2010-10-29 07:00:00 -4.0 3198 2010-10-29
+Cleveland New York 2010-10-29 07:00:00 -4.0 2630 2010-10-29
+Cleveland New York 2010-10-29 07:00:00 -19.0 2646 2010-10-29
+Cleveland New York 2010-10-29 07:00:00 -12.0 3014 2010-10-29
+Washington New York 2010-10-29 07:00:00 1.0 7291 2010-10-29
Baltimore New York 2010-10-27 07:00:00 -18.0 1064 2010-10-27
Baltimore New York 2010-10-27 07:00:00 49.0 1142 2010-10-27
Baltimore New York 2010-10-27 07:00:00 92.0 1599 2010-10-27
@@ -696,18 +793,6 @@ Cleveland New York 2010-10-28 07:00:00 3.0 2630 2010-10-28
Cleveland New York 2010-10-28 07:00:00 -6.0 2646 2010-10-28
Cleveland New York 2010-10-28 07:00:00 1.0 3014 2010-10-28
Washington New York 2010-10-28 07:00:00 45.0 7291 2010-10-28
-Baltimore New York 2010-10-29 07:00:00 -24.0 1064 2010-10-29
-Baltimore New York 2010-10-29 07:00:00 21.0 1142 2010-10-29
-Baltimore New York 2010-10-29 07:00:00 -2.0 1599 2010-10-29
-Chicago New York 2010-10-29 07:00:00 -12.0 361 2010-10-29
-Chicago New York 2010-10-29 07:00:00 -11.0 897 2010-10-29
-Chicago New York 2010-10-29 07:00:00 15.0 1531 2010-10-29
-Chicago New York 2010-10-29 07:00:00 -18.0 1610 2010-10-29
-Chicago New York 2010-10-29 07:00:00 -4.0 3198 2010-10-29
-Cleveland New York 2010-10-29 07:00:00 -4.0 2630 2010-10-29
-Cleveland New York 2010-10-29 07:00:00 -19.0 2646 2010-10-29
-Cleveland New York 2010-10-29 07:00:00 -12.0 3014 2010-10-29
-Washington New York 2010-10-29 07:00:00 1.0 7291 2010-10-29
Baltimore New York 2010-10-30 07:00:00 14.0 272 2010-10-30
Baltimore New York 2010-10-30 07:00:00 -1.0 1805 2010-10-30
Baltimore New York 2010-10-30 07:00:00 5.0 3171 2010-10-30
@@ -719,6 +804,28 @@ Cleveland New York 2010-10-30 07:00:00 -23.0 2018 2010-10-30
Cleveland New York 2010-10-30 07:00:00 -12.0 2932 2010-10-30
Washington New York 2010-10-30 07:00:00 -27.0 5904 2010-10-30
Washington New York 2010-10-30 07:00:00 -16.0 5917 2010-10-30
+Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20
+Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20
+Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20
+Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20
+Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20
+Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20
+Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20
+Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20
+Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20
+Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20
+Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20
+Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22
+Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22
+Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22
+Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22
+Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22
+Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22
+Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22
+Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22
+Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22
+Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22
+Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22
Baltimore New York 2010-10-31 07:00:00 -1.0 1599 2010-10-31
Baltimore New York 2010-10-31 07:00:00 -14.0 2571 2010-10-31
Chicago New York 2010-10-31 07:00:00 -25.0 361 2010-10-31
@@ -828,26 +935,67 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-31
2010-10-29 12
2010-10-30 11
2010-10-31 8
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_date
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_date
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
- Stage-0 is a root stage
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
STAGE PLANS:
+ Stage: Stage-1
+ Tez
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: flights_tiny_orc_partitioned_date
+ Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
+ Select Operator
+ expressions: origin_city_name (type: string), dest_city_name (type: string), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int), fl_date (type: date)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
+ Statistics: Num rows: 137 Data size: 7672 Basic stats: COMPLETE Column stats: PARTIAL
+ File Output Operator
+ compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
+ Statistics: Num rows: 137 Data size: 7672 Basic stats: COMPLETE Column stats: PARTIAL
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
+
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
- TableScan
- alias: flights_tiny_orc_partitioned_date
- Select Operator
- expressions: origin_city_name (type: string), dest_city_name (type: string), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int), fl_date (type: date)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- ListSink
+ ListSink
PREHOOK: query: select * from flights_tiny_orc_partitioned_date
PREHOOK: type: QUERY
@@ -881,17 +1029,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-29
POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-30
POSTHOOK: Input: default@flights_tiny_orc_partitioned_date@fl_date=2010-10-31
#### A masked pattern was here ####
-Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20
-Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20
-Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20
-Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20
-Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20
-Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20
-Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20
-Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20
-Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20
-Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20
-Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20
+Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26
+Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26
+Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26
+Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26
+Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26
+Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26
+Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26
+Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26
+Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26
+Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26
Baltimore New York 2010-10-21 07:00:00 17.0 1064 2010-10-21
Baltimore New York 2010-10-21 07:00:00 105.0 1142 2010-10-21
Baltimore New York 2010-10-21 07:00:00 28.0 1599 2010-10-21
@@ -904,41 +1054,6 @@ Cleveland New York 2010-10-21 07:00:00 3.0 2630 2010-10-21
Cleveland New York 2010-10-21 07:00:00 29.0 2646 2010-10-21
Cleveland New York 2010-10-21 07:00:00 72.0 3014 2010-10-21
Washington New York 2010-10-21 07:00:00 22.0 7291 2010-10-21
-Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22
-Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22
-Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22
-Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22
-Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22
-Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22
-Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22
-Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22
-Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22
-Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22
-Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22
-Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23
-Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23
-Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23
-Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23
-Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23
-Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23
-Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23
-Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23
-Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23
-Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23
-Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23
-Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23
-Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24
-Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24
-Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24
-Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24
-Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24
-Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24
-Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24
-Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24
-Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24
Baltimore New York 2010-10-25 07:00:00 -25.0 1064 2010-10-25
Baltimore New York 2010-10-25 07:00:00 92.0 1142 2010-10-25
Baltimore New York 2010-10-25 07:00:00 106.0 1599 2010-10-25
@@ -951,42 +1066,30 @@ Cleveland New York 2010-10-25 07:00:00 -4.0 2630 2010-10-25
Cleveland New York 2010-10-25 07:00:00 81.0 2646 2010-10-25
Cleveland New York 2010-10-25 07:00:00 42.0 3014 2010-10-25
Washington New York 2010-10-25 07:00:00 9.0 7291 2010-10-25
-Baltimore New York 2010-10-26 07:00:00 -22.0 1064 2010-10-26
-Baltimore New York 2010-10-26 07:00:00 123.0 1142 2010-10-26
-Baltimore New York 2010-10-26 07:00:00 90.0 1599 2010-10-26
-Chicago New York 2010-10-26 07:00:00 12.0 361 2010-10-26
-Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26
-Chicago New York 2010-10-26 07:00:00 29.0 1531 2010-10-26
-Chicago New York 2010-10-26 07:00:00 -17.0 1610 2010-10-26
-Chicago New York 2010-10-26 07:00:00 6.0 3198 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 4.0 2630 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 -27.0 2646 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 -11.0 2662 2010-10-26
-Cleveland New York 2010-10-26 07:00:00 13.0 3014 2010-10-26
-Washington New York 2010-10-26 07:00:00 4.0 7291 2010-10-26
-Baltimore New York 2010-10-27 07:00:00 -18.0 1064 2010-10-27
-Baltimore New York 2010-10-27 07:00:00 49.0 1142 2010-10-27
-Baltimore New York 2010-10-27 07:00:00 92.0 1599 2010-10-27
-Chicago New York 2010-10-27 07:00:00 148.0 361 2010-10-27
-Chicago New York 2010-10-27 07:00:00 -11.0 897 2010-10-27
-Chicago New York 2010-10-27 07:00:00 70.0 1531 2010-10-27
-Chicago New York 2010-10-27 07:00:00 8.0 1610 2010-10-27
-Chicago New York 2010-10-27 07:00:00 21.0 3198 2010-10-27
-Cleveland New York 2010-10-27 07:00:00 16.0 2630 2010-10-27
-Cleveland New York 2010-10-27 07:00:00 27.0 3014 2010-10-27
-Washington New York 2010-10-27 07:00:00 26.0 7291 2010-10-27
-Baltimore New York 2010-10-28 07:00:00 -4.0 1064 2010-10-28
-Baltimore New York 2010-10-28 07:00:00 -14.0 1142 2010-10-28
-Baltimore New York 2010-10-28 07:00:00 -14.0 1599 2010-10-28
-Chicago New York 2010-10-28 07:00:00 2.0 361 2010-10-28
-Chicago New York 2010-10-28 07:00:00 2.0 897 2010-10-28
-Chicago New York 2010-10-28 07:00:00 -11.0 1531 2010-10-28
-Chicago New York 2010-10-28 07:00:00 3.0 1610 2010-10-28
-Chicago New York 2010-10-28 07:00:00 -18.0 3198 2010-10-28
-Cleveland New York 2010-10-28 07:00:00 3.0 2630 2010-10-28
-Cleveland New York 2010-10-28 07:00:00 -6.0 2646 2010-10-28
-Cleveland New York 2010-10-28 07:00:00 1.0 3014 2010-10-28
-Washington New York 2010-10-28 07:00:00 45.0 7291 2010-10-28
+Baltimore New York 2010-10-24 07:00:00 12.0 1599 2010-10-24
+Baltimore New York 2010-10-24 07:00:00 20.0 2571 2010-10-24
+Chicago New York 2010-10-24 07:00:00 10.0 361 2010-10-24
+Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24
+Chicago New York 2010-10-24 07:00:00 -5.0 1531 2010-10-24
+Chicago New York 2010-10-24 07:00:00 -17.0 1610 2010-10-24
+Chicago New York 2010-10-24 07:00:00 -3.0 3198 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 5.0 2254 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 -11.0 2630 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 -20.0 2646 2010-10-24
+Cleveland New York 2010-10-24 07:00:00 -9.0 3014 2010-10-24
+Washington New York 2010-10-24 07:00:00 -26.0 7282 2010-10-24
+Baltimore New York 2010-10-23 07:00:00 18.0 272 2010-10-23
+Baltimore New York 2010-10-23 07:00:00 -10.0 1805 2010-10-23
+Baltimore New York 2010-10-23 07:00:00 6.0 3171 2010-10-23
+Chicago New York 2010-10-23 07:00:00 3.0 384 2010-10-23
+Chicago New York 2010-10-23 07:00:00 32.0 426 2010-10-23
+Chicago New York 2010-10-23 07:00:00 1.0 650 2010-10-23
+Chicago New York 2010-10-23 07:00:00 11.0 3085 2010-10-23
+Cleveland New York 2010-10-23 07:00:00 -21.0 2932 2010-10-23
+Washington New York 2010-10-23 07:00:00 -25.0 5832 2010-10-23
+Washington New York 2010-10-23 07:00:00 -21.0 5904 2010-10-23
+Washington New York 2010-10-23 07:00:00 -18.0 5917 2010-10-23
+Washington New York 2010-10-23 07:00:00 -16.0 7274 2010-10-23
Baltimore New York 2010-10-29 07:00:00 -24.0 1064 2010-10-29
Baltimore New York 2010-10-29 07:00:00 21.0 1142 2010-10-29
Baltimore New York 2010-10-29 07:00:00 -2.0 1599 2010-10-29
@@ -999,6 +1102,29 @@ Cleveland New York 2010-10-29 07:00:00 -4.0 2630 2010-10-29
Cleveland New York 2010-10-29 07:00:00 -19.0 2646 2010-10-29
Cleveland New York 2010-10-29 07:00:00 -12.0 3014 2010-10-29
Washington New York 2010-10-29 07:00:00 1.0 7291 2010-10-29
+Baltimore New York 2010-10-27 07:00:00 -18.0 1064 2010-10-27
+Baltimore New York 2010-10-27 07:00:00 49.0 1142 2010-10-27
+Baltimore New York 2010-10-27 07:00:00 92.0 1599 2010-10-27
+Chicago New York 2010-10-27 07:00:00 148.0 361 2010-10-27
+Chicago New York 2010-10-27 07:00:00 -11.0 897 2010-10-27
+Chicago New York 2010-10-27 07:00:00 70.0 1531 2010-10-27
+Chicago New York 2010-10-27 07:00:00 8.0 1610 2010-10-27
+Chicago New York 2010-10-27 07:00:00 21.0 3198 2010-10-27
+Cleveland New York 2010-10-27 07:00:00 16.0 2630 2010-10-27
+Cleveland New York 2010-10-27 07:00:00 27.0 3014 2010-10-27
+Washington New York 2010-10-27 07:00:00 26.0 7291 2010-10-27
+Baltimore New York 2010-10-28 07:00:00 -4.0 1064 2010-10-28
+Baltimore New York 2010-10-28 07:00:00 -14.0 1142 2010-10-28
+Baltimore New York 2010-10-28 07:00:00 -14.0 1599 2010-10-28
+Chicago New York 2010-10-28 07:00:00 2.0 361 2010-10-28
+Chicago New York 2010-10-28 07:00:00 2.0 897 2010-10-28
+Chicago New York 2010-10-28 07:00:00 -11.0 1531 2010-10-28
+Chicago New York 2010-10-28 07:00:00 3.0 1610 2010-10-28
+Chicago New York 2010-10-28 07:00:00 -18.0 3198 2010-10-28
+Cleveland New York 2010-10-28 07:00:00 3.0 2630 2010-10-28
+Cleveland New York 2010-10-28 07:00:00 -6.0 2646 2010-10-28
+Cleveland New York 2010-10-28 07:00:00 1.0 3014 2010-10-28
+Washington New York 2010-10-28 07:00:00 45.0 7291 2010-10-28
Baltimore New York 2010-10-30 07:00:00 14.0 272 2010-10-30
Baltimore New York 2010-10-30 07:00:00 -1.0 1805 2010-10-30
Baltimore New York 2010-10-30 07:00:00 5.0 3171 2010-10-30
@@ -1010,6 +1136,28 @@ Cleveland New York 2010-10-30 07:00:00 -23.0 2018 2010-10-30
Cleveland New York 2010-10-30 07:00:00 -12.0 2932 2010-10-30
Washington New York 2010-10-30 07:00:00 -27.0 5904 2010-10-30
Washington New York 2010-10-30 07:00:00 -16.0 5917 2010-10-30
+Baltimore New York 2010-10-20 07:00:00 -30.0 1064 2010-10-20
+Baltimore New York 2010-10-20 07:00:00 23.0 1142 2010-10-20
+Baltimore New York 2010-10-20 07:00:00 6.0 1599 2010-10-20
+Chicago New York 2010-10-20 07:00:00 42.0 361 2010-10-20
+Chicago New York 2010-10-20 07:00:00 24.0 897 2010-10-20
+Chicago New York 2010-10-20 07:00:00 15.0 1531 2010-10-20
+Chicago New York 2010-10-20 07:00:00 -6.0 1610 2010-10-20
+Chicago New York 2010-10-20 07:00:00 -2.0 3198 2010-10-20
+Cleveland New York 2010-10-20 07:00:00 -8.0 2630 2010-10-20
+Cleveland New York 2010-10-20 07:00:00 -15.0 3014 2010-10-20
+Washington New York 2010-10-20 07:00:00 -2.0 7291 2010-10-20
+Baltimore New York 2010-10-22 07:00:00 -12.0 1064 2010-10-22
+Baltimore New York 2010-10-22 07:00:00 54.0 1142 2010-10-22
+Baltimore New York 2010-10-22 07:00:00 18.0 1599 2010-10-22
+Chicago New York 2010-10-22 07:00:00 2.0 361 2010-10-22
+Chicago New York 2010-10-22 07:00:00 24.0 897 2010-10-22
+Chicago New York 2010-10-22 07:00:00 16.0 1531 2010-10-22
+Chicago New York 2010-10-22 07:00:00 -6.0 1610 2010-10-22
+Chicago New York 2010-10-22 07:00:00 -11.0 3198 2010-10-22
+Cleveland New York 2010-10-22 07:00:00 1.0 2630 2010-10-22
+Cleveland New York 2010-10-22 07:00:00 -25.0 2646 2010-10-22
+Cleveland New York 2010-10-22 07:00:00 -3.0 3014 2010-10-22
Baltimore New York 2010-10-31 07:00:00 -1.0 1599 2010-10-31
Baltimore New York 2010-10-31 07:00:00 -14.0 2571 2010-10-31
Chicago New York 2010-10-31 07:00:00 -25.0 361 2010-10-31
@@ -1018,12 +1166,16 @@ Chicago New York 2010-10-31 07:00:00 -4.0 1531 2010-10-31
Chicago New York 2010-10-31 07:00:00 -22.0 1610 2010-10-31
Chicago New York 2010-10-31 07:00:00 -15.0 3198 2010-10-31
Washington New York 2010-10-31 07:00:00 -18.0 7282 2010-10-31
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_date sort by fl_num, fl_date limit 25
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_date sort by fl_num, fl_date limit 25
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1042,46 +1194,102 @@ STAGE PLANS:
TableScan
alias: flights_tiny_orc_partitioned_date
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Select Operator
expressions: origin_city_name (type: string), dest_city_name (type: string), fl_time (type: timestamp), arr_delay (type: float), fl_num (type: int), fl_date (type: date)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col4 (type: int), _col5 (type: date)
sort order: ++
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkOperator
+ native: false
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
TopN Hash Memory Usage: 0.1
value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: timestamp), _col3 (type: float)
Execution mode: vectorized, llap
LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reducer 2
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: date)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2, 3, 4, 5, 0, 1]
Statistics: Num rows: 137 Data size: 7672 Basic stats: COMPLETE Column stats: PARTIAL
Limit
Number of rows: 25
+ Limit Vectorization:
+ className: VectorLimitOperator
+ native: true
Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col4 (type: int), _col5 (type: date)
sort order: ++
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkOperator
+ native: false
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false
Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL
TopN Hash Memory Usage: 0.1
value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: timestamp), _col3 (type: float)
Reducer 3
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: timestamp), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: date)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2, 3, 4, 5, 0, 1]
Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL
Limit
Number of rows: 25
+ Limit Vectorization:
+ className: VectorLimitOperator
+ native: true
Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 25 Data size: 1400 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
@@ -1151,12 +1359,16 @@ Chicago New York 2010-10-24 07:00:00 113.0 897 2010-10-24
Chicago New York 2010-10-25 07:00:00 -1.0 897 2010-10-25
Chicago New York 2010-10-26 07:00:00 0.0 897 2010-10-26
Chicago New York 2010-10-27 07:00:00 -11.0 897 2010-10-27
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select fl_date, count(*) from flights_tiny_orc_partitioned_date group by fl_date
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select fl_date, count(*) from flights_tiny_orc_partitioned_date group by fl_date
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1174,12 +1386,26 @@ STAGE PLANS:
TableScan
alias: flights_tiny_orc_partitioned_date
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Select Operator
expressions: fl_date (type: date)
outputColumnNames: fl_date
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [5]
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
+ Group By Vectorization:
+ aggregators: VectorUDAFCountStar(*) -> bigint
+ className: VectorGroupByOperator
+ vectorOutput: true
+ keyExpressions: col 5
+ native: false
+ projectedOutputColumns: [0]
keys: fl_date (type: date)
mode: hash
outputColumnNames: _col0, _col1
@@ -1188,21 +1414,50 @@ STAGE PLANS:
key expressions: _col0 (type: date)
sort order: +
Map-reduce partition columns: _col0 (type: date)
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkMultiKeyOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reducer 2
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
+ Group By Vectorization:
+ aggregators: VectorUDAFCountMerge(col 1) -> bigint
+ className: VectorGroupByOperator
+ vectorOutput: true
+ keyExpressions: col 0
+ native: false
+ projectedOutputColumns: [0]
keys: KEY._col0 (type: date)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 12 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 12 Data size: 768 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
@@ -1400,17 +1655,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-
POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-30 07%3A00%3A00
POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-31 07%3A00%3A00
#### A masked pattern was here ####
-Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00
-Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00
-Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00
-Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00
-Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00
-Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00
-Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00
-Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00
-Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00
-Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00
-Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00
+Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00
+Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00
+Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00
+Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00
+Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00
+Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00
+Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00
+Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00
+Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00
Baltimore New York 2010-10-21 17.0 1064 2010-10-21 07:00:00
Baltimore New York 2010-10-21 105.0 1142 2010-10-21 07:00:00
Baltimore New York 2010-10-21 28.0 1599 2010-10-21 07:00:00
@@ -1423,41 +1680,6 @@ Cleveland New York 2010-10-21 3.0 2630 2010-10-21 07:00:00
Cleveland New York 2010-10-21 29.0 2646 2010-10-21 07:00:00
Cleveland New York 2010-10-21 72.0 3014 2010-10-21 07:00:00
Washington New York 2010-10-21 22.0 7291 2010-10-21 07:00:00
-Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00
-Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00
-Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00
-Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00
-Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00
-Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00
-Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00
-Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00
-Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00
-Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00
-Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00
-Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00
-Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00
-Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00
-Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00
-Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00
-Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00
-Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00
-Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00
-Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00
-Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00
-Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00
-Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00
-Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00
-Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00
-Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00
-Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00
-Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00
-Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00
-Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00
-Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00
Baltimore New York 2010-10-25 -25.0 1064 2010-10-25 07:00:00
Baltimore New York 2010-10-25 92.0 1142 2010-10-25 07:00:00
Baltimore New York 2010-10-25 106.0 1599 2010-10-25 07:00:00
@@ -1470,19 +1692,42 @@ Cleveland New York 2010-10-25 -4.0 2630 2010-10-25 07:00:00
Cleveland New York 2010-10-25 81.0 2646 2010-10-25 07:00:00
Cleveland New York 2010-10-25 42.0 3014 2010-10-25 07:00:00
Washington New York 2010-10-25 9.0 7291 2010-10-25 07:00:00
-Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00
-Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00
-Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00
-Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00
-Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00
-Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00
-Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00
-Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00
-Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00
+Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00
+Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00
+Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00
+Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00
+Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00
+Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00
+Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00
+Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00
+Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00
+Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00
+Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00
+Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00
+Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00
+Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00
+Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00
+Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00
+Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00
+Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00
+Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00
+Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00
+Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00
+Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00
+Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00
+Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00
+Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00
+Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00
+Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00
+Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00
Baltimore New York 2010-10-27 -18.0 1064 2010-10-27 07:00:00
Baltimore New York 2010-10-27 49.0 1142 2010-10-27 07:00:00
Baltimore New York 2010-10-27 92.0 1599 2010-10-27 07:00:00
@@ -1506,18 +1751,6 @@ Cleveland New York 2010-10-28 3.0 2630 2010-10-28 07:00:00
Cleveland New York 2010-10-28 -6.0 2646 2010-10-28 07:00:00
Cleveland New York 2010-10-28 1.0 3014 2010-10-28 07:00:00
Washington New York 2010-10-28 45.0 7291 2010-10-28 07:00:00
-Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00
-Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00
-Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00
-Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00
-Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00
-Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00
-Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00
-Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00
Baltimore New York 2010-10-30 14.0 272 2010-10-30 07:00:00
Baltimore New York 2010-10-30 -1.0 1805 2010-10-30 07:00:00
Baltimore New York 2010-10-30 5.0 3171 2010-10-30 07:00:00
@@ -1529,6 +1762,28 @@ Cleveland New York 2010-10-30 -23.0 2018 2010-10-30 07:00:00
Cleveland New York 2010-10-30 -12.0 2932 2010-10-30 07:00:00
Washington New York 2010-10-30 -27.0 5904 2010-10-30 07:00:00
Washington New York 2010-10-30 -16.0 5917 2010-10-30 07:00:00
+Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00
+Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00
+Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00
+Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00
+Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00
+Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00
+Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00
+Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00
+Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00
+Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00
+Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00
+Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00
+Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00
+Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00
+Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00
+Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00
+Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00
+Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00
+Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00
+Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00
+Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00
+Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00
Baltimore New York 2010-10-31 -1.0 1599 2010-10-31 07:00:00
Baltimore New York 2010-10-31 -14.0 2571 2010-10-31 07:00:00
Chicago New York 2010-10-31 -25.0 361 2010-10-31 07:00:00
@@ -1638,26 +1893,67 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-
2010-10-29 07:00:00 12
2010-10-30 07:00:00 11
2010-10-31 07:00:00 8
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_timestamp
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_timestamp
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
- Stage-0 is a root stage
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
STAGE PLANS:
+ Stage: Stage-1
+ Tez
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: flights_tiny_orc_partitioned_timestamp
+ Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
+ Select Operator
+ expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), arr_delay (type: float), fl_num (type: int), fl_time (type: timestamp)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
+ Statistics: Num rows: 137 Data size: 5480 Basic stats: COMPLETE Column stats: PARTIAL
+ File Output Operator
+ compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
+ Statistics: Num rows: 137 Data size: 5480 Basic stats: COMPLETE Column stats: PARTIAL
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized, llap
+ LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
+
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
- TableScan
- alias: flights_tiny_orc_partitioned_timestamp
- Select Operator
- expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), arr_delay (type: float), fl_num (type: int), fl_time (type: timestamp)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
- ListSink
+ ListSink
PREHOOK: query: select * from flights_tiny_orc_partitioned_timestamp
PREHOOK: type: QUERY
@@ -1691,17 +1987,19 @@ POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-
POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-30 07%3A00%3A00
POSTHOOK: Input: default@flights_tiny_orc_partitioned_timestamp@fl_time=2010-10-31 07%3A00%3A00
#### A masked pattern was here ####
-Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00
-Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00
-Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00
-Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00
-Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00
-Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00
-Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00
-Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00
-Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00
-Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00
-Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00
+Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00
+Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00
+Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00
+Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00
+Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00
+Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00
+Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00
+Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00
+Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00
+Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00
Baltimore New York 2010-10-21 17.0 1064 2010-10-21 07:00:00
Baltimore New York 2010-10-21 105.0 1142 2010-10-21 07:00:00
Baltimore New York 2010-10-21 28.0 1599 2010-10-21 07:00:00
@@ -1714,41 +2012,6 @@ Cleveland New York 2010-10-21 3.0 2630 2010-10-21 07:00:00
Cleveland New York 2010-10-21 29.0 2646 2010-10-21 07:00:00
Cleveland New York 2010-10-21 72.0 3014 2010-10-21 07:00:00
Washington New York 2010-10-21 22.0 7291 2010-10-21 07:00:00
-Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00
-Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00
-Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00
-Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00
-Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00
-Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00
-Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00
-Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00
-Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00
-Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00
-Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00
-Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00
-Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00
-Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00
-Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00
-Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00
-Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00
-Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00
-Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00
-Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00
-Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00
-Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00
-Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00
-Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00
-Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00
-Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00
-Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00
-Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00
-Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00
-Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00
-Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00
-Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00
Baltimore New York 2010-10-25 -25.0 1064 2010-10-25 07:00:00
Baltimore New York 2010-10-25 92.0 1142 2010-10-25 07:00:00
Baltimore New York 2010-10-25 106.0 1599 2010-10-25 07:00:00
@@ -1761,19 +2024,42 @@ Cleveland New York 2010-10-25 -4.0 2630 2010-10-25 07:00:00
Cleveland New York 2010-10-25 81.0 2646 2010-10-25 07:00:00
Cleveland New York 2010-10-25 42.0 3014 2010-10-25 07:00:00
Washington New York 2010-10-25 9.0 7291 2010-10-25 07:00:00
-Baltimore New York 2010-10-26 -22.0 1064 2010-10-26 07:00:00
-Baltimore New York 2010-10-26 123.0 1142 2010-10-26 07:00:00
-Baltimore New York 2010-10-26 90.0 1599 2010-10-26 07:00:00
-Chicago New York 2010-10-26 12.0 361 2010-10-26 07:00:00
-Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00
-Chicago New York 2010-10-26 29.0 1531 2010-10-26 07:00:00
-Chicago New York 2010-10-26 -17.0 1610 2010-10-26 07:00:00
-Chicago New York 2010-10-26 6.0 3198 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 4.0 2630 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 -27.0 2646 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 -11.0 2662 2010-10-26 07:00:00
-Cleveland New York 2010-10-26 13.0 3014 2010-10-26 07:00:00
-Washington New York 2010-10-26 4.0 7291 2010-10-26 07:00:00
+Baltimore New York 2010-10-24 12.0 1599 2010-10-24 07:00:00
+Baltimore New York 2010-10-24 20.0 2571 2010-10-24 07:00:00
+Chicago New York 2010-10-24 10.0 361 2010-10-24 07:00:00
+Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00
+Chicago New York 2010-10-24 -5.0 1531 2010-10-24 07:00:00
+Chicago New York 2010-10-24 -17.0 1610 2010-10-24 07:00:00
+Chicago New York 2010-10-24 -3.0 3198 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 5.0 2254 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 -11.0 2630 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 -20.0 2646 2010-10-24 07:00:00
+Cleveland New York 2010-10-24 -9.0 3014 2010-10-24 07:00:00
+Washington New York 2010-10-24 -26.0 7282 2010-10-24 07:00:00
+Baltimore New York 2010-10-23 18.0 272 2010-10-23 07:00:00
+Baltimore New York 2010-10-23 -10.0 1805 2010-10-23 07:00:00
+Baltimore New York 2010-10-23 6.0 3171 2010-10-23 07:00:00
+Chicago New York 2010-10-23 3.0 384 2010-10-23 07:00:00
+Chicago New York 2010-10-23 32.0 426 2010-10-23 07:00:00
+Chicago New York 2010-10-23 1.0 650 2010-10-23 07:00:00
+Chicago New York 2010-10-23 11.0 3085 2010-10-23 07:00:00
+Cleveland New York 2010-10-23 -21.0 2932 2010-10-23 07:00:00
+Washington New York 2010-10-23 -25.0 5832 2010-10-23 07:00:00
+Washington New York 2010-10-23 -21.0 5904 2010-10-23 07:00:00
+Washington New York 2010-10-23 -18.0 5917 2010-10-23 07:00:00
+Washington New York 2010-10-23 -16.0 7274 2010-10-23 07:00:00
+Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00
+Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00
+Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00
+Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00
+Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00
+Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00
+Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00
+Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00
+Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00
Baltimore New York 2010-10-27 -18.0 1064 2010-10-27 07:00:00
Baltimore New York 2010-10-27 49.0 1142 2010-10-27 07:00:00
Baltimore New York 2010-10-27 92.0 1599 2010-10-27 07:00:00
@@ -1797,18 +2083,6 @@ Cleveland New York 2010-10-28 3.0 2630 2010-10-28 07:00:00
Cleveland New York 2010-10-28 -6.0 2646 2010-10-28 07:00:00
Cleveland New York 2010-10-28 1.0 3014 2010-10-28 07:00:00
Washington New York 2010-10-28 45.0 7291 2010-10-28 07:00:00
-Baltimore New York 2010-10-29 -24.0 1064 2010-10-29 07:00:00
-Baltimore New York 2010-10-29 21.0 1142 2010-10-29 07:00:00
-Baltimore New York 2010-10-29 -2.0 1599 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -12.0 361 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -11.0 897 2010-10-29 07:00:00
-Chicago New York 2010-10-29 15.0 1531 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -18.0 1610 2010-10-29 07:00:00
-Chicago New York 2010-10-29 -4.0 3198 2010-10-29 07:00:00
-Cleveland New York 2010-10-29 -4.0 2630 2010-10-29 07:00:00
-Cleveland New York 2010-10-29 -19.0 2646 2010-10-29 07:00:00
-Cleveland New York 2010-10-29 -12.0 3014 2010-10-29 07:00:00
-Washington New York 2010-10-29 1.0 7291 2010-10-29 07:00:00
Baltimore New York 2010-10-30 14.0 272 2010-10-30 07:00:00
Baltimore New York 2010-10-30 -1.0 1805 2010-10-30 07:00:00
Baltimore New York 2010-10-30 5.0 3171 2010-10-30 07:00:00
@@ -1820,6 +2094,28 @@ Cleveland New York 2010-10-30 -23.0 2018 2010-10-30 07:00:00
Cleveland New York 2010-10-30 -12.0 2932 2010-10-30 07:00:00
Washington New York 2010-10-30 -27.0 5904 2010-10-30 07:00:00
Washington New York 2010-10-30 -16.0 5917 2010-10-30 07:00:00
+Baltimore New York 2010-10-20 -30.0 1064 2010-10-20 07:00:00
+Baltimore New York 2010-10-20 23.0 1142 2010-10-20 07:00:00
+Baltimore New York 2010-10-20 6.0 1599 2010-10-20 07:00:00
+Chicago New York 2010-10-20 42.0 361 2010-10-20 07:00:00
+Chicago New York 2010-10-20 24.0 897 2010-10-20 07:00:00
+Chicago New York 2010-10-20 15.0 1531 2010-10-20 07:00:00
+Chicago New York 2010-10-20 -6.0 1610 2010-10-20 07:00:00
+Chicago New York 2010-10-20 -2.0 3198 2010-10-20 07:00:00
+Cleveland New York 2010-10-20 -8.0 2630 2010-10-20 07:00:00
+Cleveland New York 2010-10-20 -15.0 3014 2010-10-20 07:00:00
+Washington New York 2010-10-20 -2.0 7291 2010-10-20 07:00:00
+Baltimore New York 2010-10-22 -12.0 1064 2010-10-22 07:00:00
+Baltimore New York 2010-10-22 54.0 1142 2010-10-22 07:00:00
+Baltimore New York 2010-10-22 18.0 1599 2010-10-22 07:00:00
+Chicago New York 2010-10-22 2.0 361 2010-10-22 07:00:00
+Chicago New York 2010-10-22 24.0 897 2010-10-22 07:00:00
+Chicago New York 2010-10-22 16.0 1531 2010-10-22 07:00:00
+Chicago New York 2010-10-22 -6.0 1610 2010-10-22 07:00:00
+Chicago New York 2010-10-22 -11.0 3198 2010-10-22 07:00:00
+Cleveland New York 2010-10-22 1.0 2630 2010-10-22 07:00:00
+Cleveland New York 2010-10-22 -25.0 2646 2010-10-22 07:00:00
+Cleveland New York 2010-10-22 -3.0 3014 2010-10-22 07:00:00
Baltimore New York 2010-10-31 -1.0 1599 2010-10-31 07:00:00
Baltimore New York 2010-10-31 -14.0 2571 2010-10-31 07:00:00
Chicago New York 2010-10-31 -25.0 361 2010-10-31 07:00:00
@@ -1828,12 +2124,16 @@ Chicago New York 2010-10-31 -4.0 1531 2010-10-31 07:00:00
Chicago New York 2010-10-31 -22.0 1610 2010-10-31 07:00:00
Chicago New York 2010-10-31 -15.0 3198 2010-10-31 07:00:00
Washington New York 2010-10-31 -18.0 7282 2010-10-31 07:00:00
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_timestamp sort by fl_num, fl_time limit 25
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select * from flights_tiny_orc_partitioned_timestamp sort by fl_num, fl_time limit 25
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1852,46 +2152,102 @@ STAGE PLANS:
TableScan
alias: flights_tiny_orc_partitioned_timestamp
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Select Operator
expressions: origin_city_name (type: string), dest_city_name (type: string), fl_date (type: date), arr_delay (type: float), fl_num (type: int), fl_time (type: timestamp)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col4 (type: int), _col5 (type: timestamp)
sort order: ++
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkOperator
+ native: false
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: PARTIAL
TopN Hash Memory Usage: 0.1
value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: date), _col3 (type: float)
Execution mode: vectorized, llap
LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reducer 2
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: date), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: timestamp)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2, 3, 4, 5, 0, 1]
Statistics: Num rows: 137 Data size: 5480 Basic stats: COMPLETE Column stats: PARTIAL
Limit
Number of rows: 25
+ Limit Vectorization:
+ className: VectorLimitOperator
+ native: true
Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL
Reduce Output Operator
key expressions: _col4 (type: int), _col5 (type: timestamp)
sort order: ++
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkOperator
+ native: false
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
+ nativeConditionsNotMet: No TopN IS false, Uniform Hash IS false
Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL
TopN Hash Memory Usage: 0.1
value expressions: _col0 (type: string), _col1 (type: string), _col2 (type: date), _col3 (type: float)
Reducer 3
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Select Operator
expressions: VALUE._col0 (type: string), VALUE._col1 (type: string), VALUE._col2 (type: date), VALUE._col3 (type: float), KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: timestamp)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [2, 3, 4, 5, 0, 1]
Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL
Limit
Number of rows: 25
+ Limit Vectorization:
+ className: VectorLimitOperator
+ native: true
Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL
File Output Operator
compressed: false
+ File Sink Vectorization:
+ className: VectorFileSinkOperator
+ native: false
Statistics: Num rows: 25 Data size: 1000 Basic stats: COMPLETE Column stats: PARTIAL
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
@@ -1961,12 +2317,16 @@ Chicago New York 2010-10-24 113.0 897 2010-10-24 07:00:00
Chicago New York 2010-10-25 -1.0 897 2010-10-25 07:00:00
Chicago New York 2010-10-26 0.0 897 2010-10-26 07:00:00
Chicago New York 2010-10-27 -11.0 897 2010-10-27 07:00:00
-PREHOOK: query: explain
+PREHOOK: query: explain vectorization expression
select fl_time, count(*) from flights_tiny_orc_partitioned_timestamp group by fl_time
PREHOOK: type: QUERY
-POSTHOOK: query: explain
+POSTHOOK: query: explain vectorization expression
select fl_time, count(*) from flights_tiny_orc_partitioned_timestamp group by fl_time
POSTHOOK: type: QUERY
+PLAN VECTORIZATION:
+ enabled: true
+ enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
+
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
@@ -1984,12 +2344,26 @@ STAGE PLANS:
TableScan
alias: flights_tiny_orc_partitioned_timestamp
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan Vectorization:
+ native: true
+ projectedOutputColumns: [0, 1, 2, 3, 4, 5]
Select Operator
expressions: fl_time (type: timestamp)
outputColumnNames: fl_time
+ Select Vectorization:
+ className: VectorSelectOperator
+ native: true
+ projectedOutputColumns: [5]
Statistics: Num rows: 137 Data size: 39448 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
+ Group By Vectorization:
+ aggregators: VectorUDAFCountStar(*) -> bigint
+ className: VectorGroupByOperator
+ vectorOutput: true
+ keyExpressions: col 5
+ native: false
+ projectedOutputColumns: [0]
keys: fl_time (type: timestamp)
mode: hash
outputColumnNames: _col0, _col1
@@ -1998,21 +2372,50 @@ STAGE PLANS:
key expressions: _col0 (type: timestamp)
sort order: +
Map-reduce partition columns: _col0 (type: timestamp)
+ Reduce Sink Vectorization:
+ className: VectorReduceSinkMultiKeyOperator
+ native: true
+ nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
Statistics: Num rows: 12 Data size: 576 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: bigint)
Execution mode: vectorized, llap
LLAP IO: all inputs
+ Map Vectorization:
+ enabled: true
+ enabledConditionsMet: hive.vectorized.use.vectorized.input.format IS true
+ groupByVectorOutput: true
+ inputFileFormats: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reducer 2
Execution mode: vectorized, llap
+ Reduce Vectorization:
+ enabled: true
+ enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
+ groupByVectorOutput: true
+ allNative: false
+ usesVectorUDFAdaptor: false
+ vectorized: true
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
+ Group By Vectorization:
+ aggregators: VectorUDAFCountMerge(col 1) -> bigint
+ className: VectorGroupByOperator
+ vectorOutput: true
+ keyExpressions: col 0
+ native: false
+ projectedOutputColumns: [0]
keys: KEY._col0 (type: timestamp)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 12 Data size: 576 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
+
<TRUNCATED>