You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by se...@apache.org on 2015/10/13 02:12:26 UTC
[01/51] [abbrv] hive git commit: HIVE-11886 : LLAP: merge master into
branch (Sergey Shelukhin)
Repository: hive
Updated Branches:
refs/heads/master 9b4826e76 -> 23ede5dd1
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_groupby_3.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/vector_groupby_3.q.out
index ada310c,6aa39f1..ab2d856
--- a/ql/src/test/results/clientpositive/tez/vector_groupby_3.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_groupby_3.q.out
@@@ -153,13 -152,18 +153,17 @@@ STAGE PLANS
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: tinyint), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_grouping_sets.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_mapjoin_reduce.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/vector_mapjoin_reduce.q.out
index 01f8f65,d33b0ed..d2937a5
--- a/ql/src/test/results/clientpositive/tez/vector_mapjoin_reduce.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_mapjoin_reduce.q.out
@@@ -109,22 -109,17 +109,18 @@@ STAGE PLANS
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reducer 4
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@@ -252,22 -248,17 +248,18 @@@ STAGE PLANS
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reducer 4
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_orderby_5.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_outer_join2.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_partition_diff_num_cols.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_partitioned_date_time.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_reduce_groupby_decimal.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/vector_reduce_groupby_decimal.q.out
index d14e1b7,14aa777..1cd5959
--- a/ql/src/test/results/clientpositive/tez/vector_reduce_groupby_decimal.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_reduce_groupby_decimal.q.out
@@@ -44,25 -44,20 +44,21 @@@ STAGE PLANS
Filter Operator
predicate: (cdecimal1 is not null and cdecimal2 is not null) (type: boolean)
Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: cint (type: int), cdouble (type: double), cdecimal1 (type: decimal(20,10)), cdecimal2 (type: decimal(23,14))
- outputColumnNames: _col0, _col1, _col2, _col3
+ Group By Operator
+ aggregations: min(cdecimal1)
+ keys: cint (type: int), cdouble (type: double), cdecimal1 (type: decimal(20,10)), cdecimal2 (type: decimal(23,14))
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(_col2)
- keys: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
+ sort order: ++++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
- sort order: ++++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
- Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col4 (type: decimal(20,10))
+ value expressions: _col4 (type: decimal(20,10))
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorization_0.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorization_13.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorization_15.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorization_limit.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorization_short_regress.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorized_distinct_gby.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorized_nested_mapjoin.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vectorized_timestamp_funcs.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vector_left_outer_join.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vector_outer_join1.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vector_outer_join2.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/vector_outer_join2.q.out
index 3058423,236fa18..ce855ef
--- a/ql/src/test/results/clientpositive/vector_outer_join2.q.out
+++ b/ql/src/test/results/clientpositive/vector_outer_join2.q.out
@@@ -251,22 -251,18 +251,18 @@@ STAGE PLANS
1 _col0 (type: bigint)
outputColumnNames: _col1
Statistics: Num rows: 24 Data size: 5060 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: bigint)
- outputColumnNames: _col0
- Statistics: Num rows: 24 Data size: 5060 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), sum(_col0)
- mode: hash
- outputColumnNames: _col0, _col1
+ Group By Operator
+ aggregations: count(), sum(_col1)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ Execution mode: vectorized
Local Work:
Map Reduce Local Work
- Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), sum(VALUE._col1)
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vector_outer_join3.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vector_outer_join4.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vector_outer_join5.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out
index 85b8cc0,6a9532e..75acec9
--- a/ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/vectorized_nested_mapjoin.q.out
@@@ -86,22 -86,18 +86,18 @@@ STAGE PLANS
1 _col0 (type: smallint)
outputColumnNames: _col1
Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: double)
+ Group By Operator
+ aggregations: sum(_col1)
+ mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col0)
- mode: hash
- outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: double)
+ value expressions: _col0 (type: double)
+ Execution mode: vectorized
Local Work:
Map Reduce Local Work
- Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0)
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java
----------------------------------------------------------------------
diff --cc serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java
index c9807bf,cbad3b2..77e6de0
--- a/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java
+++ b/serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java
@@@ -19,8 -19,11 +19,10 @@@
package org.apache.hadoop.hive.serde2;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
+ import org.apache.commons.logging.Log;
+ import org.apache.commons.logging.LogFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.util.StringUtils;
[34/51] [abbrv] hive git commit: HIVE-12042 : LLAP: update some out
files (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12042 : LLAP: update some out files (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/86e1c89e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/86e1c89e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/86e1c89e
Branch: refs/heads/master
Commit: 86e1c89ea9dbb1162a3d143c7f2da553dcbba769
Parents: 68e5ad9
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 5 18:54:02 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 5 18:54:02 2015 -0700
----------------------------------------------------------------------
.../llap/dynamic_partition_pruning.q.out | 1 -
.../clientpositive/llap/explainuser_1.q.out | 70 ++++++++++----------
.../vectorized_dynamic_partition_pruning.q.out | 1 -
.../tez/vector_groupby_reduce.q.out | 14 ++--
4 files changed, 42 insertions(+), 44 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/86e1c89e/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
index cdb11b6..705e31d 100644
--- a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
@@ -4299,7 +4299,6 @@ STAGE PLANS:
input vertices:
1 Reducer 4
Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE
- HybridGraceHashJoin: true
Group By Operator
aggregations: count()
mode: hash
http://git-wip-us.apache.org/repos/asf/hive/blob/86e1c89e/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index d41e02f..7161e01 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -6031,36 +6031,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key AND a.value <=> b.value
PREHOOK: type: QUERY
@@ -6078,34 +6078,34 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int), value (type: int)","1":"key (type: int), value (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int), value (type: int)
| Map-reduce partition columns:key (type: int), value (type: int)
| sort order:++
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int), value (type: int)
Map-reduce partition columns:key (type: int), value (type: int)
sort order:++
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a RIGHT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6123,36 +6123,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Right Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6170,36 +6170,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a LEFT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6217,36 +6217,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Left Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 14 Data size: 122 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 13 Data size: 111 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: drop table sales
PREHOOK: type: DROPTABLE
http://git-wip-us.apache.org/repos/asf/hive/blob/86e1c89e/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out b/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
index 182d6f7..44f4568 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
@@ -4299,7 +4299,6 @@ STAGE PLANS:
input vertices:
1 Reducer 4
Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE
- HybridGraceHashJoin: true
Group By Operator
aggregations: count()
mode: hash
http://git-wip-us.apache.org/repos/asf/hive/blob/86e1c89e/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out b/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
index a0771b3..9942629 100644
--- a/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
@@ -378,6 +378,7 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -398,8 +399,8 @@ STAGE PLANS:
key expressions: _col0 (type: int)
sort order: +
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
- Execution mode: vectorized
Reducer 3
+ Execution mode: vectorized
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: int)
@@ -412,7 +413,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
Stage: Stage-0
Fetch Operator
@@ -588,6 +588,7 @@ STAGE PLANS:
value expressions: _col2 (type: int)
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
@@ -607,8 +608,8 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Execution mode: vectorized
Reducer 3
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0), sum(VALUE._col1)
@@ -621,8 +622,8 @@ STAGE PLANS:
sort order: +
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Execution mode: vectorized
Reducer 4
+ Execution mode: vectorized
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 (type: bigint), VALUE._col1 (type: bigint)
@@ -635,7 +636,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
Stage: Stage-0
Fetch Operator
@@ -810,6 +810,7 @@ STAGE PLANS:
value expressions: _col2 (type: int)
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
@@ -832,8 +833,8 @@ STAGE PLANS:
sort order: ++
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: bigint)
- Execution mode: vectorized
Reducer 3
+ Execution mode: vectorized
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: int), VALUE._col0 (type: bigint)
@@ -846,7 +847,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
Stage: Stage-0
Fetch Operator
[20/51] [abbrv] hive git commit: HIVE-11963: Llap: Disable web app
for mini llap tests (Prasanth Jayachandran reviewed by Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-11963: Llap: Disable web app for mini llap tests (Prasanth Jayachandran reviewed by Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/3b64bd6c
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/3b64bd6c
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/3b64bd6c
Branch: refs/heads/master
Commit: 3b64bd6c1a4838937aff451ab829529fe0be9521
Parents: f84eefa
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Fri Sep 25 13:31:09 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Fri Sep 25 13:31:09 2015 -0500
----------------------------------------------------------------------
.../org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3b64bd6c/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
----------------------------------------------------------------------
diff --git a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
index 9139de6..6f75001 100644
--- a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
+++ b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapDaemon.java
@@ -177,8 +177,12 @@ public class LlapDaemon extends CompositeService implements ContainerRunner, Lla
this.registry = new LlapRegistryService(true);
addIfService(registry);
- this.webServices = new LlapWebServices();
- addIfService(webServices);
+ if (HiveConf.getBoolVar(daemonConf, HiveConf.ConfVars.HIVE_IN_TEST)) {
+ this.webServices = null;
+ } else {
+ this.webServices = new LlapWebServices();
+ addIfService(webServices);
+ }
// Bring up the server only after all other components have started.
addIfService(server);
// AMReporter after the server so that it gets the correct address. It knows how to deal with
[33/51] [abbrv] hive git commit: HIVE-12034 : HIVE-4243 broke things
for llap branch (Sergey Shelukhin) ADDENDUM
Posted by se...@apache.org.
HIVE-12034 : HIVE-4243 broke things for llap branch (Sergey Shelukhin) ADDENDUM
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/68e5ad9e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/68e5ad9e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/68e5ad9e
Branch: refs/heads/master
Commit: 68e5ad9e68ca1dc1ab89708396565ab8369e7302
Parents: edd3079
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 5 13:39:46 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 5 13:39:46 2015 -0700
----------------------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/68e5ad9e/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
index 68d3255..b4dd4ab 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
@@ -132,7 +132,9 @@ public final class OrcFile {
}
values = new WriterVersion[max + 1];
for (WriterVersion v : WriterVersion.values()) {
- values[v.id] = v;
+ if (v.id < values.length) {
+ values[v.id] = v;
+ }
}
}
[27/51] [abbrv] hive git commit: HIVE-12001 : LLAP: update some out
files (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12001 : LLAP: update some out files (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/7273a4c4
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/7273a4c4
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/7273a4c4
Branch: refs/heads/master
Commit: 7273a4c4405f38ed7de95a4bf915b0309cf75dc0
Parents: 7a3e745
Author: Sergey Shelukhin <se...@apache.org>
Authored: Wed Sep 30 12:07:04 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Wed Sep 30 12:07:04 2015 -0700
----------------------------------------------------------------------
.../llap/dynpart_sort_opt_vectorization.q.out | 12 +-
.../llap/dynpart_sort_optimization.q.out | 12 +-
.../clientpositive/llap/explainuser_1.q.out | 76 +-
.../llap/vector_auto_smb_mapjoin_14.q.out | 1918 ++++++++++++++++++
.../llap/vector_groupby_reduce.q.out | 1473 +++++++++++++-
.../llap/vectorization_limit.q.out | 8 +-
6 files changed, 3421 insertions(+), 78 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/7273a4c4/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
index 07a87c8..2f70b40 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
@@ -1821,13 +1821,15 @@ Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
-PREHOOK: query: select * from over1k_part2_orc
+PREHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2_orc
PREHOOK: type: QUERY
PREHOOK: Input: default@over1k_part2_orc
PREHOOK: Input: default@over1k_part2_orc@ds=foo/t=27
PREHOOK: Input: default@over1k_part2_orc@ds=foo/t=__HIVE_DEFAULT_PARTITION__
#### A masked pattern was here ####
-POSTHOOK: query: select * from over1k_part2_orc
+POSTHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2_orc
POSTHOOK: type: QUERY
POSTHOOK: Input: default@over1k_part2_orc
POSTHOOK: Input: default@over1k_part2_orc@ds=foo/t=27
@@ -1966,13 +1968,15 @@ Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
-PREHOOK: query: select * from over1k_part2_orc
+PREHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2_orc
PREHOOK: type: QUERY
PREHOOK: Input: default@over1k_part2_orc
PREHOOK: Input: default@over1k_part2_orc@ds=foo/t=27
PREHOOK: Input: default@over1k_part2_orc@ds=foo/t=__HIVE_DEFAULT_PARTITION__
#### A masked pattern was here ####
-POSTHOOK: query: select * from over1k_part2_orc
+POSTHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2_orc
POSTHOOK: type: QUERY
POSTHOOK: Input: default@over1k_part2_orc
POSTHOOK: Input: default@over1k_part2_orc@ds=foo/t=27
http://git-wip-us.apache.org/repos/asf/hive/blob/7273a4c4/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
index 2534254..fa15157 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
@@ -1752,13 +1752,15 @@ Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
-PREHOOK: query: select * from over1k_part2
+PREHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2
PREHOOK: type: QUERY
PREHOOK: Input: default@over1k_part2
PREHOOK: Input: default@over1k_part2@ds=foo/t=27
PREHOOK: Input: default@over1k_part2@ds=foo/t=__HIVE_DEFAULT_PARTITION__
#### A masked pattern was here ####
-POSTHOOK: query: select * from over1k_part2
+POSTHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@over1k_part2
POSTHOOK: Input: default@over1k_part2@ds=foo/t=27
@@ -1897,13 +1899,15 @@ Bucket Columns: []
Sort Columns: []
Storage Desc Params:
serialization.format 1
-PREHOOK: query: select * from over1k_part2
+PREHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2
PREHOOK: type: QUERY
PREHOOK: Input: default@over1k_part2
PREHOOK: Input: default@over1k_part2@ds=foo/t=27
PREHOOK: Input: default@over1k_part2@ds=foo/t=__HIVE_DEFAULT_PARTITION__
#### A masked pattern was here ####
-POSTHOOK: query: select * from over1k_part2
+POSTHOOK: query: -- SORT_BEFORE_DIFF
+select * from over1k_part2
POSTHOOK: type: QUERY
POSTHOOK: Input: default@over1k_part2
POSTHOOK: Input: default@over1k_part2@ds=foo/t=27
http://git-wip-us.apache.org/repos/asf/hive/blob/7273a4c4/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 7d9d99e..d41e02f 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -1986,7 +1986,7 @@ Stage-0
Statistics:Num rows: 4 Data size: 728 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col1 (type: int), _col2 (type: float), _col3 (type: string), _col4 (type: int)
Filter Operator [FIL_27]
- predicate:((((_col1 + _col4) = 2) and _col0 is not null) and ((_col4 + 1) = 2)) (type: boolean)
+ predicate:((((_col1 + _col4) = 2) and ((_col4 + 1) = 2)) and _col0 is not null) (type: boolean)
Statistics:Num rows: 4 Data size: 728 Basic stats: COMPLETE Column stats: COMPLETE
Merge Join Operator [MERGEJOIN_31]
| condition map:[{"":"Outer Join 0 to 1"}]
@@ -2797,7 +2797,7 @@ Stage-0
| outputColumnNames:["_col0","_col1","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
| Filter Operator [FIL_61]
- | predicate:(((UDFToDouble(_col0) + UDFToDouble(_col3)) >= 0.0) and ((UDFToDouble(_col0) >= 1.0) or (_col3 >= 1))) (type: boolean)
+ | predicate:(((UDFToDouble(_col0) >= 1.0) or (_col3 >= 1)) and ((UDFToDouble(_col0) + UDFToDouble(_col3)) >= 0.0)) (type: boolean)
| Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
| Select Operator [SEL_65]
| outputColumnNames:["_col0","_col1","_col3"]
@@ -4240,7 +4240,7 @@ Stage-0
Select Operator [SEL_13]
Statistics:Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator [FIL_45]
- predicate:((_col0 is null or _col1 is null) and ((_col2 - _col1) > 600.0)) (type: boolean)
+ predicate:(((_col2 - _col1) > 600.0) and (_col0 is null or _col1 is null)) (type: boolean)
Statistics:Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator [GBY_11]
| aggregations:["min(VALUE._col0)","max(VALUE._col1)"]
@@ -6031,36 +6031,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key AND a.value <=> b.value
PREHOOK: type: QUERY
@@ -6078,34 +6078,34 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int), value (type: int)","1":"key (type: int), value (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int), value (type: int)
| Map-reduce partition columns:key (type: int), value (type: int)
| sort order:++
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int), value (type: int)
Map-reduce partition columns:key (type: int), value (type: int)
sort order:++
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a RIGHT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6123,36 +6123,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Right Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6170,36 +6170,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a LEFT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6217,36 +6217,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Left Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: drop table sales
PREHOOK: type: DROPTABLE
[11/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/having.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/having.q.out b/ql/src/test/results/clientpositive/llap/having.q.out
index 28a515d..026fdbf 100644
--- a/ql/src/test/results/clientpositive/llap/having.q.out
+++ b/ql/src/test/results/clientpositive/llap/having.q.out
@@ -112,22 +112,18 @@ STAGE PLANS:
Filter Operator
predicate: (UDFToDouble(key) <> 302.0) (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ aggregations: max(value)
+ keys: key (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: max(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ value expressions: _col1 (type: string)
Execution mode: llap
Reducer 2
Execution mode: uber
@@ -490,11 +486,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1)
- keys: _col0 (type: string)
+ aggregations: max(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -765,22 +761,18 @@ STAGE PLANS:
Filter Operator
predicate: (UDFToDouble(key) > 300.0) (type: boolean)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ aggregations: max(value)
+ keys: key (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: max(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ value expressions: _col1 (type: string)
Execution mode: llap
Reducer 2
Execution mode: uber
@@ -967,11 +959,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1)
- keys: _col0 (type: string)
+ aggregations: max(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -1237,11 +1229,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/limit_pushdown.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/limit_pushdown.q.out b/ql/src/test/results/clientpositive/llap/limit_pushdown.q.out
index 9119780..e39ac18 100644
--- a/ql/src/test/results/clientpositive/llap/limit_pushdown.q.out
+++ b/ql/src/test/results/clientpositive/llap/limit_pushdown.q.out
@@ -392,10 +392,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cdouble (type: double)
- outputColumnNames: _col0
+ outputColumnNames: cdouble
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: double)
+ keys: cdouble (type: double)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
@@ -483,10 +483,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint), cdouble (type: double)
- outputColumnNames: _col0, _col1
+ outputColumnNames: ctinyint, cdouble
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: tinyint), _col1 (type: double)
+ keys: ctinyint (type: tinyint), cdouble (type: double)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
@@ -579,10 +579,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint), cdouble (type: double)
- outputColumnNames: _col0, _col1
+ outputColumnNames: ctinyint, cdouble
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: tinyint), _col1 (type: double)
+ keys: ctinyint (type: tinyint), cdouble (type: double)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
@@ -677,11 +677,11 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint), cstring1 (type: string), cstring2 (type: string)
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: ctinyint, cstring1, cstring2
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(DISTINCT _col1), count(DISTINCT _col2)
- keys: _col0 (type: tinyint), _col1 (type: string), _col2 (type: string)
+ aggregations: count(DISTINCT cstring1), count(DISTINCT cstring2)
+ keys: ctinyint (type: tinyint), cstring1 (type: string), cstring2 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
@@ -799,11 +799,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string), key (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: value, key
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col1)
- keys: _col0 (type: string)
+ aggregations: sum(key)
+ keys: value (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -1067,16 +1067,16 @@ STAGE PLANS:
alias: src
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: value (type: string), key (type: string)
- outputColumnNames: _col0, _col1
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
+ key expressions: value (type: string)
sort order: +
- Map-reduce partition columns: _col0 (type: string)
+ Map-reduce partition columns: value (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.3
- value expressions: _col1 (type: string)
+ value expressions: key (type: string)
Execution mode: llap
Reducer 2
Execution mode: uber
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/mapjoin_mapjoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/mapjoin_mapjoin.q.out b/ql/src/test/results/clientpositive/llap/mapjoin_mapjoin.q.out
index 17be306..236f0ed 100644
--- a/ql/src/test/results/clientpositive/llap/mapjoin_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/llap/mapjoin_mapjoin.q.out
@@ -623,22 +623,18 @@ STAGE PLANS:
1 Map 4
Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
- Select Operator
- expressions: _col2 (type: string)
- outputColumnNames: _col0
+ Group By Operator
+ aggregations: count()
+ keys: _col2 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint)
+ value expressions: _col1 (type: bigint)
Execution mode: llap
Map 3
Map Operator Tree:
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out b/ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
index bacbff3..ed3179d 100644
--- a/ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
+++ b/ql/src/test/results/clientpositive/llap/metadata_only_queries.q.out
@@ -452,10 +452,10 @@ STAGE PLANS:
Statistics: Num rows: 9489 Data size: 978785 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ts (type: timestamp)
- outputColumnNames: _col0
+ outputColumnNames: ts
Statistics: Num rows: 9489 Data size: 978785 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(ts)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/metadataonly1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/metadataonly1.q.out b/ql/src/test/results/clientpositive/llap/metadataonly1.q.out
index d569a23..40f122a 100644
--- a/ql/src/test/results/clientpositive/llap/metadataonly1.q.out
+++ b/ql/src/test/results/clientpositive/llap/metadataonly1.q.out
@@ -48,10 +48,10 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -157,10 +157,10 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -304,11 +304,11 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: count(DISTINCT _col0)
- keys: _col0 (type: string)
+ aggregations: count(DISTINCT ds)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
@@ -452,10 +452,10 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -743,10 +743,10 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -1028,11 +1028,11 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string), hr (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: ds, hr
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: count(DISTINCT _col1)
- keys: _col0 (type: string), _col1 (type: string)
+ aggregations: count(DISTINCT hr)
+ keys: ds (type: string), hr (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
@@ -1273,11 +1273,11 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string), hr (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: ds, hr
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(hr)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
@@ -1514,10 +1514,10 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -1761,11 +1761,11 @@ STAGE PLANS:
GatherStats: false
Select Operator
expressions: ds (type: string), hr (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: ds, hr
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: count(DISTINCT _col1)
- keys: _col0 (type: string), _col1 (type: string)
+ aggregations: count(DISTINCT hr)
+ keys: ds (type: string), hr (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/mrr.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/mrr.q.out b/ql/src/test/results/clientpositive/llap/mrr.q.out
index f507e4b..d0200d2 100644
--- a/ql/src/test/results/clientpositive/llap/mrr.q.out
+++ b/ql/src/test/results/clientpositive/llap/mrr.q.out
@@ -27,11 +27,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -1314,22 +1314,18 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ aggregations: count(value)
+ keys: key (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint)
+ value expressions: _col1 (type: bigint)
Execution mode: llap
Map 6
Map Operator Tree:
@@ -1339,22 +1335,18 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ aggregations: count(value)
+ keys: key (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint)
+ value expressions: _col1 (type: bigint)
Execution mode: llap
Map 9
Map Operator Tree:
@@ -1364,22 +1356,18 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ aggregations: count(value)
+ keys: key (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint)
+ value expressions: _col1 (type: bigint)
Execution mode: llap
Reducer 10
Execution mode: llap
@@ -1713,22 +1701,18 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ aggregations: count(value)
+ keys: key (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint)
+ value expressions: _col1 (type: bigint)
Execution mode: llap
Map 3
Map Operator Tree:
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out b/ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out
index fa3cf92..663d999 100644
--- a/ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out
+++ b/ql/src/test/results/clientpositive/llap/selectDistinctStar.q.out
@@ -24,10 +24,10 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
+ keys: key (type: string), value (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -720,20 +720,16 @@ STAGE PLANS:
Filter Operator
predicate: (key < '3') (type: boolean)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
Execution mode: uber
@@ -2481,10 +2477,10 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
+ keys: key (type: string), value (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -3173,20 +3169,16 @@ STAGE PLANS:
Filter Operator
predicate: (key < '3') (type: boolean)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
Execution mode: uber
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/show_create_database.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/show_create_database.q.out b/ql/src/test/results/clientpositive/llap/show_create_database.q.out
new file mode 100644
index 0000000..4755d2d
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/show_create_database.q.out
@@ -0,0 +1,19 @@
+PREHOOK: query: CREATE DATABASE some_database comment 'for show create db test' WITH DBPROPERTIES ('somekey'='somevalue')
+PREHOOK: type: CREATEDATABASE
+PREHOOK: Output: database:some_database
+POSTHOOK: query: CREATE DATABASE some_database comment 'for show create db test' WITH DBPROPERTIES ('somekey'='somevalue')
+POSTHOOK: type: CREATEDATABASE
+POSTHOOK: Output: database:some_database
+PREHOOK: query: SHOW CREATE DATABASE some_database
+PREHOOK: type: SHOW_CREATEDATABASE
+PREHOOK: Input: database:some_database
+POSTHOOK: query: SHOW CREATE DATABASE some_database
+POSTHOOK: type: SHOW_CREATEDATABASE
+POSTHOOK: Input: database:some_database
+CREATE DATABASE `some_database`
+COMMENT
+ 'for show create db test'
+LOCATION
+#### A masked pattern was here ####
+WITH DBPROPERTIES (
+ 'somekey'='somevalue')
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/stats_only_null.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/stats_only_null.q.out b/ql/src/test/results/clientpositive/llap/stats_only_null.q.out
index 1336b79..bf3a6b0 100644
--- a/ql/src/test/results/clientpositive/llap/stats_only_null.q.out
+++ b/ql/src/test/results/clientpositive/llap/stats_only_null.q.out
@@ -90,10 +90,10 @@ STAGE PLANS:
Statistics: Num rows: 10 Data size: 120 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: a (type: double), b (type: int), c (type: string), d (type: smallint)
- outputColumnNames: _col0, _col1, _col2, _col3
+ outputColumnNames: a, b, c, d
Statistics: Num rows: 10 Data size: 120 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(), count(_col0), count(_col1), count(_col2), count(_col3)
+ aggregations: count(), count(a), count(b), count(c), count(d)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1 Data size: 40 Basic stats: COMPLETE Column stats: NONE
@@ -148,10 +148,10 @@ STAGE PLANS:
Statistics: Num rows: 10 Data size: 120 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: a (type: double), b (type: int), c (type: string), d (type: smallint)
- outputColumnNames: _col0, _col1, _col2, _col3
+ outputColumnNames: a, b, c, d
Statistics: Num rows: 10 Data size: 120 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(), count(_col0), count(_col1), count(_col2), count(_col3)
+ aggregations: count(), count(a), count(b), count(c), count(d)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1 Data size: 40 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/subquery_in.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_in.q.out b/ql/src/test/results/clientpositive/llap/subquery_in.q.out
index daa26e9..034557f 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_in.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_in.q.out
@@ -663,20 +663,16 @@ STAGE PLANS:
Filter Operator
predicate: ((key > '9') and value is not null) (type: boolean)
Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
Execution mode: llap
@@ -851,20 +847,16 @@ STAGE PLANS:
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
Execution mode: llap
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/tez_dml.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/tez_dml.q.out b/ql/src/test/results/clientpositive/llap/tez_dml.q.out
index 2254d10..a87ccf3 100644
--- a/ql/src/test/results/clientpositive/llap/tez_dml.q.out
+++ b/ql/src/test/results/clientpositive/llap/tez_dml.q.out
@@ -30,11 +30,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string)
- outputColumnNames: _col0
+ outputColumnNames: value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: value (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/union5.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/union5.q.out b/ql/src/test/results/clientpositive/llap/union5.q.out
index 5a5c290..bc3821b 100644
--- a/ql/src/test/results/clientpositive/llap/union5.q.out
+++ b/ql/src/test/results/clientpositive/llap/union5.q.out
@@ -31,34 +31,38 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: s1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: key (type: string)
+ outputColumnNames: key
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(1)
+ aggregations: count(key)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: llap
Map 5
Map Operator Tree:
TableScan
alias: s1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: key (type: string)
+ outputColumnNames: key
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(1)
+ aggregations: count(key)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: llap
Reducer 2
@@ -68,22 +72,22 @@ STAGE PLANS:
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 'tst1' (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 88 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
keys: _col0 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 4
Execution mode: uber
@@ -93,10 +97,10 @@ STAGE PLANS:
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -108,22 +112,22 @@ STAGE PLANS:
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 'tst2' (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 88 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
keys: _col0 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Union 3
Vertex: Union 3
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/union7.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/union7.q.out b/ql/src/test/results/clientpositive/llap/union7.q.out
index 6aa6676..a301cad 100644
--- a/ql/src/test/results/clientpositive/llap/union7.q.out
+++ b/ql/src/test/results/clientpositive/llap/union7.q.out
@@ -31,17 +31,19 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: s1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: key (type: string)
+ outputColumnNames: key
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(1)
+ aggregations: count(key)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Execution mode: llap
Map 5
@@ -58,12 +60,12 @@ STAGE PLANS:
keys: _col0 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: PARTIAL
+ Statistics: Num rows: 26 Data size: 199 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: PARTIAL
+ Statistics: Num rows: 26 Data size: 199 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Execution mode: llap
Reducer 2
@@ -73,22 +75,22 @@ STAGE PLANS:
aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 'tst1' (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 88 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
keys: _col0 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: PARTIAL
+ Statistics: Num rows: 26 Data size: 199 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: PARTIAL
+ Statistics: Num rows: 26 Data size: 199 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
Reducer 4
Execution mode: uber
@@ -98,10 +100,10 @@ STAGE PLANS:
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: PARTIAL
+ Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: PARTIAL
+ Statistics: Num rows: 13 Data size: 99 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out b/ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out
index c18b460..7e5fe2d 100644
--- a/ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/unionDistinct_1.q.out
@@ -14290,11 +14290,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string)
- outputColumnNames: _col0
+ outputColumnNames: key
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: string)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -14436,11 +14436,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string)
- outputColumnNames: _col0
+ outputColumnNames: key
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: string)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -14631,6 +14631,38 @@ POSTHOOK: Lineage: src10_3.key SIMPLE [(src)src.FieldSchema(name:key, type:strin
POSTHOOK: Lineage: src10_3.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
POSTHOOK: Lineage: src10_4.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ]
POSTHOOK: Lineage: src10_4.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
+PREHOOK: query: analyze table src10_1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_1
+PREHOOK: Output: default@src10_1
+POSTHOOK: query: analyze table src10_1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_1
+POSTHOOK: Output: default@src10_1
+PREHOOK: query: analyze table src10_2 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_2
+PREHOOK: Output: default@src10_2
+POSTHOOK: query: analyze table src10_2 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_2
+POSTHOOK: Output: default@src10_2
+PREHOOK: query: analyze table src10_3 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_3
+PREHOOK: Output: default@src10_3
+POSTHOOK: query: analyze table src10_3 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_3
+POSTHOOK: Output: default@src10_3
+PREHOOK: query: analyze table src10_4 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_4
+PREHOOK: Output: default@src10_4
+POSTHOOK: query: analyze table src10_4 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_4
+POSTHOOK: Output: default@src10_4
PREHOOK: query: -- When we convert the Join of sub1 and sub0 into a MapJoin,
-- we can use a single MR job to evaluate this entire query.
explain
@@ -14669,14 +14701,14 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src10_1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Map Join Operator
condition map:
Inner Join 0 to 1
@@ -14686,76 +14718,76 @@ STAGE PLANS:
outputColumnNames: _col0, _col1
input vertices:
1 Map 5
- Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 57 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 5
Map Operator Tree:
TableScan
alias: src10_2
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 6
Map Operator Tree:
TableScan
alias: src10_3
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 9
Map Operator Tree:
TableScan
alias: src10_4
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 3
Execution mode: llap
@@ -14764,11 +14796,11 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Reducer 4
Execution mode: uber
@@ -14776,10 +14808,10 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -14791,17 +14823,17 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Union 2
Vertex: Union 2
Union 7
@@ -14887,78 +14919,78 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: src10_1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Execution mode: llap
Map 10
Map Operator Tree:
TableScan
alias: src10_4
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 6
Map Operator Tree:
TableScan
alias: src10_2
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: key is not null (type: boolean)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 52 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 7
Map Operator Tree:
TableScan
alias: src10_3
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 208 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
Execution mode: llap
@@ -14970,17 +15002,17 @@ STAGE PLANS:
0 _col0 (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 5 Data size: 57 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Reducer 4
Execution mode: llap
Reduce Operator Tree:
@@ -14988,11 +15020,11 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Reducer 5
Execution mode: uber
@@ -15000,10 +15032,10 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 119 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 75 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -15015,17 +15047,17 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 10 Data size: 104 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 239 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 161 Basic stats: COMPLETE Column stats: NONE
Union 3
Vertex: Union 3
Union 8
[26/51] [abbrv] hive git commit: HIVE-12001 : LLAP: update some out
files (Sergey Shelukhin)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/7273a4c4/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out b/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
new file mode 100644
index 0000000..c0103a8
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/vector_auto_smb_mapjoin_14.q.out
@@ -0,0 +1,1918 @@
+PREHOOK: query: -- SORT_QUERY_RESULTS
+
+CREATE TABLE tbl1(key int, value string) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORC
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tbl1
+POSTHOOK: query: -- SORT_QUERY_RESULTS
+
+CREATE TABLE tbl1(key int, value string) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORC
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tbl1
+PREHOOK: query: CREATE TABLE tbl2(key int, value string) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORC
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@tbl2
+POSTHOOK: query: CREATE TABLE tbl2(key int, value string) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORC
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@tbl2
+PREHOOK: query: insert overwrite table tbl1
+select * from src where key < 10
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Output: default@tbl1
+POSTHOOK: query: insert overwrite table tbl1
+select * from src where key < 10
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Output: default@tbl1
+POSTHOOK: Lineage: tbl1.key EXPRESSION [(src)src.FieldSchema(name:key, type:string, comment:default), ]
+POSTHOOK: Lineage: tbl1.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
+PREHOOK: query: insert overwrite table tbl2
+select * from src where key < 10
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Output: default@tbl2
+POSTHOOK: query: insert overwrite table tbl2
+select * from src where key < 10
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Output: default@tbl2
+POSTHOOK: Lineage: tbl2.key EXPRESSION [(src)src.FieldSchema(name:key, type:string, comment:default), ]
+POSTHOOK: Lineage: tbl2.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
+PREHOOK: query: -- The join is being performed as part of sub-query. It should be converted to a sort-merge join
+explain
+select count(*) from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The join is being performed as part of sub-query. It should be converted to a sort-merge join
+explain
+select count(*) from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+22
+PREHOOK: query: -- The join is being performed as part of more than one sub-query. It should be converted to a sort-merge join
+explain
+select count(*) from
+(
+ select key, count(*) from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1
+ group by key
+) subq2
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The join is being performed as part of more than one sub-query. It should be converted to a sort-merge join
+explain
+select count(*) from
+(
+ select key, count(*) from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1
+ group by key
+) subq2
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ keys: _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Reducer 3
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+(
+ select key, count(*) from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1
+ group by key
+) subq2
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+(
+ select key, count(*) from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1
+ group by key
+) subq2
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+6
+PREHOOK: query: -- A join is being performed across different sub-queries, where a join is being performed in each of them.
+-- Each sub-query should be converted to a sort-merge join.
+explain
+select src1.key, src1.cnt1, src2.cnt1 from
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1 group by key
+) src1
+join
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq2 group by key
+) src2
+on src1.key = src2.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- A join is being performed across different sub-queries, where a join is being performed in each of them.
+-- Each sub-query should be converted to a sort-merge join.
+explain
+select src1.key, src1.cnt1, src2.cnt1 from
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1 group by key
+) src1
+join
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq2 group by key
+) src2
+on src1.key = src2.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ keys: _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint)
+ Execution mode: llap
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ keys: _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint)
+ Reducer 3
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1, _col3
+ Statistics: Num rows: 2 Data size: 224 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: bigint), _col3 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 2 Data size: 224 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 2 Data size: 224 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Reducer 6
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint)
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select src1.key, src1.cnt1, src2.cnt1 from
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1 group by key
+) src1
+join
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq2 group by key
+) src2
+on src1.key = src2.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select src1.key, src1.cnt1, src2.cnt1 from
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq1 group by key
+) src1
+join
+(
+ select key, count(*) as cnt1 from
+ (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+ ) subq2 group by key
+) src2
+on src1.key = src2.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+0 9 9
+2 1 1
+4 1 1
+5 9 9
+8 1 1
+9 1 1
+PREHOOK: query: -- The subquery itself is being joined. Since the sub-query only contains selects and filters, it should
+-- be converted to a sort-merge join.
+explain
+select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on subq1.key = subq2.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The subquery itself is being joined. Since the sub-query only contains selects and filters, it should
+-- be converted to a sort-merge join.
+explain
+select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on subq1.key = subq2.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 6) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 6) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ Statistics: Num rows: 3 Data size: 306 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on subq1.key = subq2.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on subq1.key = subq2.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+20
+PREHOOK: query: -- The subquery itself is being joined. Since the sub-query only contains selects and filters, it should
+-- be converted to a sort-merge join, although there is more than one level of sub-query
+explain
+select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join tbl2 b
+ on subq2.key = b.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The subquery itself is being joined. Since the sub-query only contains selects and filters, it should
+-- be converted to a sort-merge join, although there is more than one level of sub-query
+explain
+select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join tbl2 b
+ on subq2.key = b.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (((key < 8) and (key < 6)) and key is not null) (type: boolean)
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 key (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join tbl2 b
+ on subq2.key = b.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join tbl2 b
+ on subq2.key = b.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+20
+PREHOOK: query: -- Both the tables are nested sub-queries i.e more then 1 level of sub-query.
+-- The join should be converted to a sort-merge join
+explain
+select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq3
+ where key < 6
+ ) subq4
+ on subq2.key = subq4.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- Both the tables are nested sub-queries i.e more then 1 level of sub-query.
+-- The join should be converted to a sort-merge join
+explain
+select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq3
+ where key < 6
+ ) subq4
+ on subq2.key = subq4.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (((key < 8) and (key < 6)) and key is not null) (type: boolean)
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (((key < 8) and (key < 6)) and key is not null) (type: boolean)
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq3
+ where key < 6
+ ) subq4
+ on subq2.key = subq4.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+ join
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq3
+ where key < 6
+ ) subq4
+ on subq2.key = subq4.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+#### A masked pattern was here ####
+20
+PREHOOK: query: -- The subquery itself is being joined. Since the sub-query only contains selects and filters and the join key
+-- is not getting modified, it should be converted to a sort-merge join. Note that the sub-query modifies one
+-- item, but that is not part of the join key.
+explain
+select count(*) from
+ (select a.key as key, concat(a.value, a.value) as value from tbl1 a where key < 8) subq1
+ join
+ (select a.key as key, concat(a.value, a.value) as value from tbl2 a where key < 8) subq2
+ on subq1.key = subq2.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The subquery itself is being joined. Since the sub-query only contains selects and filters and the join key
+-- is not getting modified, it should be converted to a sort-merge join. Note that the sub-query modifies one
+-- item, but that is not part of the join key.
+explain
+select count(*) from
+ (select a.key as key, concat(a.value, a.value) as value from tbl1 a where key < 8) subq1
+ join
+ (select a.key as key, concat(a.value, a.value) as value from tbl2 a where key < 8) subq2
+ on subq1.key = subq2.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 8) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 8) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ Statistics: Num rows: 3 Data size: 306 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (select a.key as key, concat(a.value, a.value) as value from tbl1 a where key < 8) subq1
+ join
+ (select a.key as key, concat(a.value, a.value) as value from tbl2 a where key < 8) subq2
+ on subq1.key = subq2.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (select a.key as key, concat(a.value, a.value) as value from tbl1 a where key < 8) subq1
+ join
+ (select a.key as key, concat(a.value, a.value) as value from tbl2 a where key < 8) subq2
+ on subq1.key = subq2.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+20
+PREHOOK: query: -- Since the join key is modified by the sub-query, neither sort-merge join not bucketized map-side
+-- join should be performed
+explain
+select count(*) from
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl1 a) subq1
+ join
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl2 a) subq2
+ on subq1.key = subq2.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- Since the join key is modified by the sub-query, neither sort-merge join not bucketized map-side
+-- join should be performed
+explain
+select count(*) from
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl1 a) subq1
+ join
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl2 a) subq2
+ on subq1.key = subq2.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: (key + 1) (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: _col0 is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: vectorized, llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: (key + 1) (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: _col0 is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: vectorized, llap
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Reducer 3
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl1 a) subq1
+ join
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl2 a) subq2
+ on subq1.key = subq2.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl1 a) subq1
+ join
+ (select a.key +1 as key, concat(a.value, a.value) as value from tbl2 a) subq2
+ on subq1.key = subq2.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+22
+PREHOOK: query: -- One of the tables is a sub-query and the other is not.
+-- It should be converted to a sort-merge join.
+explain
+select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join tbl2 a on subq1.key = a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- One of the tables is a sub-query and the other is not.
+-- It should be converted to a sort-merge join.
+explain
+select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join tbl2 a on subq1.key = a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 6) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 key (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join tbl2 a on subq1.key = a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join tbl2 a on subq1.key = a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+20
+PREHOOK: query: -- There are more than 2 inputs to the join, all of them being sub-queries.
+-- It should be converted to to a sort-merge join
+explain
+select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on (subq1.key = subq2.key)
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq3
+ on (subq1.key = subq3.key)
+PREHOOK: type: QUERY
+POSTHOOK: query: -- There are more than 2 inputs to the join, all of them being sub-queries.
+-- It should be converted to to a sort-merge join
+explain
+select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on (subq1.key = subq2.key)
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq3
+ on (subq1.key = subq3.key)
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 6) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 6) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 6) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ Inner Join 0 to 2
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ 2 _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on subq1.key = subq2.key
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq3
+ on (subq1.key = subq3.key)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from
+ (select a.key as key, a.value as value from tbl1 a where key < 6) subq1
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq2
+ on subq1.key = subq2.key
+ join
+ (select a.key as key, a.value as value from tbl2 a where key < 6) subq3
+ on (subq1.key = subq3.key)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+56
+PREHOOK: query: -- The join is being performed on a nested sub-query, and an aggregation is performed after that.
+-- The join should be converted to a sort-merge join
+explain
+select count(*) from (
+ select subq2.key as key, subq2.value as value1, b.value as value2 from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+join tbl2 b
+on subq2.key = b.key) a
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The join is being performed on a nested sub-query, and an aggregation is performed after that.
+-- The join should be converted to a sort-merge join
+explain
+select count(*) from (
+ select subq2.key as key, subq2.value as value1, b.value as value2 from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+join tbl2 b
+on subq2.key = b.key) a
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (((key < 8) and (key < 6)) and key is not null) (type: boolean)
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 key (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select count(*) from (
+ select subq2.key as key, subq2.value as value1, b.value as value2 from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+join tbl2 b
+on subq2.key = b.key) a
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+POSTHOOK: query: select count(*) from (
+ select subq2.key as key, subq2.value as value1, b.value as value2 from
+ (
+ select * from
+ (
+ select a.key as key, a.value as value from tbl1 a where key < 8
+ ) subq1
+ where key < 6
+ ) subq2
+join tbl2 b
+on subq2.key = b.key) a
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+#### A masked pattern was here ####
+20
+PREHOOK: query: CREATE TABLE dest1(key int, value string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@dest1
+POSTHOOK: query: CREATE TABLE dest1(key int, value string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@dest1
+PREHOOK: query: CREATE TABLE dest2(key int, val1 string, val2 string)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@dest2
+POSTHOOK: query: CREATE TABLE dest2(key int, val1 string, val2 string)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@dest2
+PREHOOK: query: -- The join is followed by a multi-table insert. It should be converted to
+-- a sort-merge join
+explain
+from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, val1, val2
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The join is followed by a multi-table insert. It should be converted to
+-- a sort-merge join
+explain
+from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, val1, val2
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-2 is a root stage
+ Stage-3 depends on stages: Stage-2
+ Stage-0 depends on stages: Stage-3
+ Stage-4 depends on stages: Stage-0
+ Stage-1 depends on stages: Stage-3
+ Stage-5 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-2
+ Tez
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0, _col1, _col6
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string), _col6 (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest1
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest2
+ Execution mode: llap
+
+ Stage: Stage-3
+ Dependency Collection
+
+ Stage: Stage-0
+ Move Operator
+ tables:
+ replace: true
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest1
+
+ Stage: Stage-4
+ Stats-Aggr Operator
+
+ Stage: Stage-1
+ Move Operator
+ tables:
+ replace: true
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest2
+
+ Stage: Stage-5
+ Stats-Aggr Operator
+
+PREHOOK: query: from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, val1, val2
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+PREHOOK: Output: default@dest1
+PREHOOK: Output: default@dest2
+POSTHOOK: query: from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, val1, val2
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+POSTHOOK: Output: default@dest1
+POSTHOOK: Output: default@dest2
+POSTHOOK: Lineage: dest1.key SIMPLE [(tbl1)a.FieldSchema(name:key, type:int, comment:null), ]
+POSTHOOK: Lineage: dest1.value SIMPLE [(tbl1)a.FieldSchema(name:value, type:string, comment:null), ]
+POSTHOOK: Lineage: dest2.key SIMPLE [(tbl1)a.FieldSchema(name:key, type:int, comment:null), ]
+POSTHOOK: Lineage: dest2.val1 SIMPLE [(tbl1)a.FieldSchema(name:value, type:string, comment:null), ]
+POSTHOOK: Lineage: dest2.val2 SIMPLE [(tbl2)b.FieldSchema(name:value, type:string, comment:null), ]
+PREHOOK: query: select * from dest1
+PREHOOK: type: QUERY
+PREHOOK: Input: default@dest1
+#### A masked pattern was here ####
+POSTHOOK: query: select * from dest1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@dest1
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+2 val_2
+4 val_4
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+8 val_8
+9 val_9
+PREHOOK: query: select * from dest2
+PREHOOK: type: QUERY
+PREHOOK: Input: default@dest2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from dest2
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@dest2
+#### A masked pattern was here ####
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+0 val_0 val_0
+2 val_2 val_2
+4 val_4 val_4
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+5 val_5 val_5
+8 val_8 val_8
+9 val_9 val_9
+PREHOOK: query: DROP TABLE dest2
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@dest2
+PREHOOK: Output: default@dest2
+POSTHOOK: query: DROP TABLE dest2
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@dest2
+POSTHOOK: Output: default@dest2
+PREHOOK: query: CREATE TABLE dest2(key int, cnt int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@dest2
+POSTHOOK: query: CREATE TABLE dest2(key int, cnt int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@dest2
+PREHOOK: query: -- The join is followed by a multi-table insert, and one of the inserts involves a reducer.
+-- It should be converted to a sort-merge join
+explain
+from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, count(*) group by key
+PREHOOK: type: QUERY
+POSTHOOK: query: -- The join is followed by a multi-table insert, and one of the inserts involves a reducer.
+-- It should be converted to a sort-merge join
+explain
+from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, count(*) group by key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-2 is a root stage
+ Stage-3 depends on stages: Stage-2
+ Stage-0 depends on stages: Stage-3
+ Stage-4 depends on stages: Stage-0
+ Stage-1 depends on stages: Stage-3
+ Stage-5 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-2
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 10 Data size: 930 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 5 Data size: 465 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest1
+ Select Operator
+ expressions: _col0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ keys: _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 5 Data size: 511 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint)
+ Execution mode: llap
+ Reducer 2
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), UDFToInteger(_col1) (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest2
+
+ Stage: Stage-3
+ Dependency Collection
+
+ Stage: Stage-0
+ Move Operator
+ tables:
+ replace: true
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest1
+
+ Stage: Stage-4
+ Stats-Aggr Operator
+
+ Stage: Stage-1
+ Move Operator
+ tables:
+ replace: true
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.dest2
+
+ Stage: Stage-5
+ Stats-Aggr Operator
+
+PREHOOK: query: from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, count(*) group by key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@tbl1
+PREHOOK: Input: default@tbl2
+PREHOOK: Output: default@dest1
+PREHOOK: Output: default@dest2
+POSTHOOK: query: from (
+ select a.key as key, a.value as val1, b.value as val2 from tbl1 a join tbl2 b on a.key = b.key
+) subq1
+insert overwrite table dest1 select key, val1
+insert overwrite table dest2 select key, count(*) group by key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@tbl1
+POSTHOOK: Input: default@tbl2
+POSTHOOK: Output: default@dest1
+POSTHOOK: Output: default@dest2
+POSTHOOK: Lineage: dest1.key SIMPLE [(tbl1)a.FieldSchema(name:key, type:int, comment:null), ]
+POSTHOOK: Lineage: dest1.value SIMPLE [(tbl1)a.FieldSchema(name:value, type:string, comment:null), ]
+POSTHOOK: Lineage: dest2.cnt EXPRESSION [(tbl1)a.null, (tbl2)b.null, ]
+POSTHOOK: Lineage: dest2.key SIMPLE [(tbl1)a.FieldSchema(name:key, type:int, comment:null), ]
+PREHOOK: query: select * from dest1
+PREHOOK: type: QUERY
+PREHOOK: Input: default@dest1
+#### A masked pattern was here ####
+POSTHOOK: query: select * from dest1
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@dest1
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+2 val_2
+4 val_4
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+5 val_5
+8 val_8
+9 val_9
+PREHOOK: query: select * from dest2
+PREHOOK: type: QUERY
+PREHOOK: Input: default@dest2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from dest2
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@dest2
+#### A masked pattern was here ####
+0 9
+2 1
+4 1
+5 9
+8 1
+9 1
[46/51] [abbrv] hive git commit: HIVE-12081: LLAP: Make
explainuser_1.q test consistent (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-12081: LLAP: Make explainuser_1.q test consistent (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/1d9574ad
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/1d9574ad
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/1d9574ad
Branch: refs/heads/master
Commit: 1d9574ad099c6364de8b7f987218124774de5c80
Parents: 49972ef
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Sat Oct 10 01:44:49 2015 +0530
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Sat Oct 10 01:44:49 2015 +0530
----------------------------------------------------------------------
ql/src/test/queries/clientpositive/explainuser_1.q | 2 ++
ql/src/test/results/clientpositive/llap/explainuser_1.q.out | 8 ++++++++
ql/src/test/results/clientpositive/tez/explainuser_1.q.out | 8 ++++++++
3 files changed, 18 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/1d9574ad/ql/src/test/queries/clientpositive/explainuser_1.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/explainuser_1.q b/ql/src/test/queries/clientpositive/explainuser_1.q
index 2862de4..7ae1ed6 100644
--- a/ql/src/test/queries/clientpositive/explainuser_1.q
+++ b/ql/src/test/queries/clientpositive/explainuser_1.q
@@ -348,6 +348,8 @@ SET hive.optimize.bucketmapjoin = true;
SET hive.optimize.bucketmapjoin.sortedmerge = true;
SET hive.input.format = org.apache.hadoop.hive.ql.io.BucketizedHiveInputFormat;
+analyze table smb_input1 compute statistics;
+
explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key;
explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key AND a.value <=> b.value;
explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a RIGHT OUTER JOIN smb_input1 b ON a.key <=> b.key;
http://git-wip-us.apache.org/repos/asf/hive/blob/1d9574ad/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 141a80b..4b8ec3f 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -6015,6 +6015,14 @@ POSTHOOK: Lineage: smb_input1.key SIMPLE [(smb_input)smb_input.FieldSchema(name:
POSTHOOK: Lineage: smb_input1.value SIMPLE [(smb_input)smb_input.FieldSchema(name:value, type:int, comment:null), ]
POSTHOOK: Lineage: smb_input2.key SIMPLE [(smb_input)smb_input.FieldSchema(name:key, type:int, comment:null), ]
POSTHOOK: Lineage: smb_input2.value SIMPLE [(smb_input)smb_input.FieldSchema(name:value, type:int, comment:null), ]
+PREHOOK: query: analyze table smb_input1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@smb_input1
+PREHOOK: Output: default@smb_input1
+POSTHOOK: query: analyze table smb_input1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@smb_input1
+POSTHOOK: Output: default@smb_input1
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
POSTHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
http://git-wip-us.apache.org/repos/asf/hive/blob/1d9574ad/ql/src/test/results/clientpositive/tez/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/explainuser_1.q.out b/ql/src/test/results/clientpositive/tez/explainuser_1.q.out
index 141a80b..4b8ec3f 100644
--- a/ql/src/test/results/clientpositive/tez/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/tez/explainuser_1.q.out
@@ -6015,6 +6015,14 @@ POSTHOOK: Lineage: smb_input1.key SIMPLE [(smb_input)smb_input.FieldSchema(name:
POSTHOOK: Lineage: smb_input1.value SIMPLE [(smb_input)smb_input.FieldSchema(name:value, type:int, comment:null), ]
POSTHOOK: Lineage: smb_input2.key SIMPLE [(smb_input)smb_input.FieldSchema(name:key, type:int, comment:null), ]
POSTHOOK: Lineage: smb_input2.value SIMPLE [(smb_input)smb_input.FieldSchema(name:value, type:int, comment:null), ]
+PREHOOK: query: analyze table smb_input1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@smb_input1
+PREHOOK: Output: default@smb_input1
+POSTHOOK: query: analyze table smb_input1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@smb_input1
+POSTHOOK: Output: default@smb_input1
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
POSTHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
[48/51] [abbrv] hive git commit: HIVE-12078 : LLAP: document config
settings (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12078 : LLAP: document config settings (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/3cfcad66
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/3cfcad66
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/3cfcad66
Branch: refs/heads/master
Commit: 3cfcad660b905a0791638e82649b27cf3be7053a
Parents: 4e53bfd
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 12 11:35:17 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 12 11:35:17 2015 -0700
----------------------------------------------------------------------
.../org/apache/hadoop/hive/conf/HiveConf.java | 66 ++++++++++++--------
1 file changed, 39 insertions(+), 27 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3cfcad66/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
----------------------------------------------------------------------
diff --git a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
index 109b19b..0fcd39b 100644
--- a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
+++ b/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
@@ -891,10 +891,10 @@ public class HiveConf extends Configuration {
HIVEDEFAULTFILEFORMAT("hive.default.fileformat", "TextFile", new StringSet("TextFile", "SequenceFile", "RCfile", "ORC"),
"Default file format for CREATE TABLE statement. Users can explicitly override it by CREATE TABLE ... STORED AS [FORMAT]"),
HIVEDEFAULTMANAGEDFILEFORMAT("hive.default.fileformat.managed", "none",
- new StringSet("none", "TextFile", "SequenceFile", "RCfile", "ORC"),
- "Default file format for CREATE TABLE statement applied to managed tables only. External tables will be \n" +
- "created with format specified by hive.default.fileformat. Leaving this null will result in using hive.default.fileformat \n" +
- "for all tables."),
+ new StringSet("none", "TextFile", "SequenceFile", "RCfile", "ORC"),
+ "Default file format for CREATE TABLE statement applied to managed tables only. External tables will be \n" +
+ "created with format specified by hive.default.fileformat. Leaving this null will result in using hive.default.fileformat \n" +
+ "for all tables."),
HIVEQUERYRESULTFILEFORMAT("hive.query.result.fileformat", "TextFile", new StringSet("TextFile", "SequenceFile", "RCfile"),
"Default file format for storing result of the query."),
HIVECHECKFILEFORMAT("hive.fileformat.check", true, "Whether to check file format or not when loading data files"),
@@ -2225,8 +2225,8 @@ public class HiveConf extends Configuration {
"Whether to send the query plan via local resource or RPC"),
HIVE_AM_SPLIT_GENERATION("hive.compute.splits.in.am", true,
"Whether to generate the splits locally or in the AM (tez only)"),
- HIVE_TEZ_GENERATE_CONSISTENT_SPLITS("hive.tez.input.generate.consistent.splits", true, "Whether to generate consisten split" +
- "locations when generating splits in the AM"),
+ HIVE_TEZ_GENERATE_CONSISTENT_SPLITS("hive.tez.input.generate.consistent.splits", true,
+ "Whether to generate consistent split locations when generating splits in the AM"),
HIVE_PREWARM_ENABLED("hive.prewarm.enabled", false, "Enables container prewarm for Tez/Spark (Hadoop 2 only)"),
HIVE_PREWARM_NUM_CONTAINERS("hive.prewarm.numcontainers", 10, "Controls the number of containers to prewarm for Tez/Spark (Hadoop 2 only)"),
@@ -2302,37 +2302,49 @@ public class HiveConf extends Configuration {
"hive.tez.exec.inplace.progress",
true,
"Updates tez job execution progress in-place in the terminal."),
- LLAP_IO_ENABLED("hive.llap.io.enabled", false, ""),
- LLAP_LOW_LEVEL_CACHE("hive.llap.io.use.lowlevel.cache", true, ""),
- LLAP_ORC_CACHE_MIN_ALLOC("hive.llap.io.cache.orc.alloc.min", 128 * 1024, ""),
- LLAP_ORC_CACHE_MAX_ALLOC("hive.llap.io.cache.orc.alloc.max", 16 * 1024 * 1024, ""),
- LLAP_ORC_CACHE_ARENA_SIZE("hive.llap.io.cache.orc.arena.size", 128 * 1024 * 1024, ""),
- LLAP_ORC_CACHE_MAX_SIZE("hive.llap.io.cache.orc.size", 1024L * 1024 * 1024, ""),
- LLAP_ORC_CACHE_ALLOCATE_DIRECT("hive.llap.io.cache.direct", true, ""),
- LLAP_USE_LRFU("hive.llap.io.use.lrfu", false, ""),
- LLAP_LRFU_LAMBDA("hive.llap.io.lrfu.lambda", 0.01f, ""),
- LLAP_ORC_ENABLE_TIME_COUNTERS("hive.llap.io.orc.time.counters", true, ""),
+ LLAP_IO_ENABLED("hive.llap.io.enabled", false, "Whether the LLAP IO layer is enabled."),
+ LLAP_LOW_LEVEL_CACHE("hive.llap.io.use.lowlevel.cache", true, "Must always be true for now"),
+ LLAP_ORC_CACHE_MIN_ALLOC("hive.llap.io.cache.orc.alloc.min", 128 * 1024,
+ "Minimum allocation possible from LLAP low-level cache for ORC. Allocations below that\n" +
+ "will be padded to minimum allocation. Should generally be the same as expected ORC\n" +
+ "compression buffer size, or next lowest power of 2. Must be power of 2."),
+ LLAP_ORC_CACHE_MAX_ALLOC("hive.llap.io.cache.orc.alloc.max", 16 * 1024 * 1024,
+ "Maximum allocation possible from LLAP low-level cache for ORC. Should be as large as\n" +
+ "the largest expected ORC compression buffer size. Must be power of 2."),
+ LLAP_ORC_CACHE_ARENA_SIZE("hive.llap.io.cache.orc.arena.size", 128 * 1024 * 1024,
+ "Arena size for ORC low-level cache; cache will be allocated in arena-sized steps.\n" +
+ "Must presently be a power of two."),
+ LLAP_ORC_CACHE_MAX_SIZE("hive.llap.io.cache.orc.size", 1024L * 1024 * 1024,
+ "Maximum size for ORC low-level cache; must be a multiple of arena size."),
+ LLAP_ORC_CACHE_ALLOCATE_DIRECT("hive.llap.io.cache.direct", true,
+ "Whether ORC low-level cache should use direct allocation."),
+ LLAP_USE_LRFU("hive.llap.io.use.lrfu", false,
+ "Whether ORC low-level cache should use LRFU cache policy instead of default (FIFO)."),
+ LLAP_LRFU_LAMBDA("hive.llap.io.lrfu.lambda", 0.01f,
+ "Lambda for ORC low-level cache LRFU cache policy."),
+ LLAP_ORC_ENABLE_TIME_COUNTERS("hive.llap.io.orc.time.counters", true,
+ "Whether to enable time counters for LLAP IO layer (time spent in HDFS, etc.)"),
LLAP_AUTO_ALLOW_UBER("hive.llap.auto.allow.uber", true,
- "Whether or not to allow the planner to run vertices in the AM"),
+ "Whether or not to allow the planner to run vertices in the AM."),
LLAP_AUTO_ENFORCE_TREE("hive.llap.auto.enforce.tree", true,
- "Enforce that all parents are in llap, before considering vertex"),
+ "Enforce that all parents are in llap, before considering vertex"),
LLAP_AUTO_ENFORCE_VECTORIZED("hive.llap.auto.enforce.vectorized", true,
- "Enforce that inputs are vectorized, before considering vertex"),
+ "Enforce that inputs are vectorized, before considering vertex"),
LLAP_AUTO_ENFORCE_STATS("hive.llap.auto.enforce.stats", true,
- "Enforce that col stats are available, before considering vertex"),
+ "Enforce that col stats are available, before considering vertex"),
LLAP_AUTO_MAX_INPUT("hive.llap.auto.max.input.size", 10*1024*1024*1024L,
- "Check input size, before considering vertex (-1 disables check)"),
+ "Check input size, before considering vertex (-1 disables check)"),
LLAP_AUTO_MAX_OUTPUT("hive.llap.auto.max.output.size", 1*1024*1024*1024L,
- "Check output size, before considering vertex (-1 disables check)"),
+ "Check output size, before considering vertex (-1 disables check)"),
LLAP_EXECUTION_MODE("hive.llap.execution.mode", "none",
- new StringSet("auto", "none", "all", "map"),
+ new StringSet("auto", "none", "all", "map"),
"Chooses whether query fragments will run in container or in llap"),
LLAP_OBJECT_CACHE_ENABLED("hive.llap.object.cache.enabled", true,
- "Cache objects (plans, hashtables, etc) in llap"),
+ "Cache objects (plans, hashtables, etc) in llap"),
LLAP_QUEUE_METRICS_PERCENTILE_INTERVALS("hive.llap.queue.metrics.percentiles.intervals", "",
- "Comma-delimited set of integers denoting the desired rollover intervals (in seconds) for" +
- "percentile latency metrics on the LLAP daemon producer-consumer queue. By default," +
- "percentile latency metrics are disabled."),
+ "Comma-delimited set of integers denoting the desired rollover intervals (in seconds)\n" +
+ "for percentile latency metrics on the LLAP daemon producer-consumer queue.\n" +
+ "By default, percentile latency metrics are disabled."),
LLAP_IO_THREADPOOL_SIZE("hive.llap.io.threadpool.size", 10,
"Specify the number of threads to use for low-level IO thread pool."),
[43/51] [abbrv] hive git commit: HIVE-12042 : LLAP: update some out
files (Sergey Shelukhin) ADDENDUM3
Posted by se...@apache.org.
HIVE-12042 : LLAP: update some out files (Sergey Shelukhin) ADDENDUM3
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/3cd915b8
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/3cd915b8
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/3cd915b8
Branch: refs/heads/master
Commit: 3cd915b82f6e3096be3b321aa87e97729c0b70c8
Parents: cf76e6b
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 8 13:01:17 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 8 13:01:17 2015 -0700
----------------------------------------------------------------------
.../clientpositive/llap/explainuser_1.q.out | 96 ++++++++++----------
1 file changed, 48 insertions(+), 48 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3cd915b8/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 3ca2f50..d57c271 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -548,10 +548,10 @@ Stage-0
Select Operator [SEL_37]
outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_36]
+ Filter Operator [FIL_51]
predicate:((_col1 > 0) or (_col6 >= 0)) (type: boolean)
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_55]
+ Merge Join Operator [MERGEJOIN_57]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col6"]
@@ -566,7 +566,7 @@ Stage-0
| Select Operator [SEL_30]
| outputColumnNames:["_col0","_col1"]
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_53]
+ | Filter Operator [FIL_55]
| predicate:key is not null (type: boolean)
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_29]
@@ -582,10 +582,10 @@ Stage-0
Select Operator [SEL_28]
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_27]
+ Filter Operator [FIL_52]
predicate:((_col1 + _col4) >= 0) (type: boolean)
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_54]
+ Merge Join Operator [MERGEJOIN_56]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col0","_col1","_col2","_col4"]
@@ -626,7 +626,7 @@ Stage-0
| keys:key (type: string), c_int (type: int), c_float (type: float)
| outputColumnNames:["_col0","_col1","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_52]
+ | Filter Operator [FIL_54]
| predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
| Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_11]
@@ -668,7 +668,7 @@ Stage-0
keys:key (type: string), c_int (type: int), c_float (type: float)
outputColumnNames:["_col0","_col1","_col2","_col3"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_51]
+ Filter Operator [FIL_53]
predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_0]
@@ -731,10 +731,10 @@ Stage-0
Select Operator [SEL_34]
outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_48]
+ Filter Operator [FIL_49]
predicate:((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_col6) + _col2) >= 0)) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_53]
+ Merge Join Operator [MERGEJOIN_55]
| condition map:[{"":"Left Outer Join0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col6"]
@@ -762,10 +762,10 @@ Stage-0
Select Operator [SEL_27]
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_26]
+ Filter Operator [FIL_50]
predicate:((_col1 + _col4) >= 0) (type: boolean)
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_52]
+ Merge Join Operator [MERGEJOIN_54]
| condition map:[{"":"Left Outer Join0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col0","_col1","_col2","_col4"]
@@ -809,7 +809,7 @@ Stage-0
| Select Operator [SEL_2]
| outputColumnNames:["_col0","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_49]
+ | Filter Operator [FIL_51]
| predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_0]
@@ -842,7 +842,7 @@ Stage-0
Select Operator [SEL_15]
outputColumnNames:["_col0","_col2","_col3"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_50]
+ Filter Operator [FIL_52]
predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_13]
@@ -894,10 +894,10 @@ Stage-0
Select Operator [SEL_30]
outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_29]
+ Filter Operator [FIL_38]
predicate:(((_col1 + _col4) >= 2) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_40]
+ Merge Join Operator [MERGEJOIN_41]
| condition map:[{"":"Right Outer Join0 to 1"},{"":"Right Outer Join0 to 2"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col4","_col6"]
@@ -954,7 +954,7 @@ Stage-0
| Select Operator [SEL_2]
| outputColumnNames:["_col0","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_38]
+ | Filter Operator [FIL_39]
| predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_0]
@@ -987,7 +987,7 @@ Stage-0
Select Operator [SEL_15]
outputColumnNames:["_col0","_col2","_col3"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_39]
+ Filter Operator [FIL_40]
predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_13]
@@ -1050,10 +1050,10 @@ Stage-0
Select Operator [SEL_33]
outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_44]
+ Filter Operator [FIL_45]
predicate:(((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_col6) + _col2) >= 0)) and ((_col1 + _col4) >= 0)) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_47]
+ Merge Join Operator [MERGEJOIN_48]
| condition map:[{"":"Outer Join 0 to 1"},{"":"Outer Join 0 to 2"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col4","_col6"]
@@ -1110,7 +1110,7 @@ Stage-0
| Select Operator [SEL_2]
| outputColumnNames:["_col0","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_45]
+ | Filter Operator [FIL_46]
| predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_0]
@@ -1155,7 +1155,7 @@ Stage-0
Select Operator [SEL_15]
outputColumnNames:["_col0","_col2","_col3"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_46]
+ Filter Operator [FIL_47]
predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_13]
@@ -1207,10 +1207,10 @@ Stage-0
Select Operator [SEL_35]
outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_34]
+ Filter Operator [FIL_46]
predicate:((_col1 > 0) or (_col6 >= 0)) (type: boolean)
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_50]
+ Merge Join Operator [MERGEJOIN_52]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col6"]
@@ -1225,7 +1225,7 @@ Stage-0
| Select Operator [SEL_28]
| outputColumnNames:["_col0","_col1"]
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_48]
+ | Filter Operator [FIL_50]
| predicate:key is not null (type: boolean)
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_27]
@@ -1241,10 +1241,10 @@ Stage-0
Select Operator [SEL_26]
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_25]
+ Filter Operator [FIL_47]
predicate:((_col1 + _col4) >= 0) (type: boolean)
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_49]
+ Merge Join Operator [MERGEJOIN_51]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col0","_col1","_col2","_col4"]
@@ -1279,7 +1279,7 @@ Stage-0
| Select Operator [SEL_2]
| outputColumnNames:["_col0","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_46]
+ | Filter Operator [FIL_48]
| predicate:(((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) and key is not null) (type: boolean)
| Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_0]
@@ -1312,7 +1312,7 @@ Stage-0
Select Operator [SEL_12]
outputColumnNames:["_col0","_col2","_col3"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_47]
+ Filter Operator [FIL_49]
predicate:(((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) and key is not null) (type: boolean)
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_10]
@@ -1954,10 +1954,10 @@ Stage-0
Select Operator [SEL_19]
outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
Statistics:Num rows: 4 Data size: 404 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_18]
+ Filter Operator [FIL_26]
predicate:(((_col4 + 1) = 2) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
Statistics:Num rows: 4 Data size: 404 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_32]
+ Merge Join Operator [MERGEJOIN_34]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col3","_col4","_col6"]
@@ -1972,7 +1972,7 @@ Stage-0
| Select Operator [SEL_12]
| outputColumnNames:["_col0","_col1"]
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_30]
+ | Filter Operator [FIL_32]
| predicate:key is not null (type: boolean)
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_11]
@@ -1985,10 +1985,10 @@ Stage-0
sort order:+
Statistics:Num rows: 4 Data size: 728 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col1 (type: int), _col2 (type: float), _col3 (type: string), _col4 (type: int)
- Filter Operator [FIL_27]
+ Filter Operator [FIL_29]
predicate:((((_col1 + _col4) = 2) and _col0 is not null) and ((_col4 + 1) = 2)) (type: boolean)
Statistics:Num rows: 4 Data size: 728 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_31]
+ Merge Join Operator [MERGEJOIN_33]
| condition map:[{"":"Outer Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
@@ -2003,7 +2003,7 @@ Stage-0
| Select Operator [SEL_2]
| outputColumnNames:["_col0","_col1","_col2"]
| Statistics:Num rows: 6 Data size: 465 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_28]
+ | Filter Operator [FIL_30]
| predicate:(((c_int + 1) = 2) and ((c_int > 0) or (c_float >= 0.0))) (type: boolean)
| Statistics:Num rows: 6 Data size: 465 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_0]
@@ -2019,7 +2019,7 @@ Stage-0
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1"]
Statistics:Num rows: 6 Data size: 445 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_29]
+ Filter Operator [FIL_31]
predicate:(((c_int + 1) = 2) and ((c_int > 0) or (c_float >= 0.0))) (type: boolean)
Statistics:Num rows: 6 Data size: 465 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_3]
@@ -2047,10 +2047,10 @@ Stage-0
Select Operator [SEL_13]
outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
Statistics:Num rows: 12 Data size: 1212 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_20]
+ Filter Operator [FIL_21]
predicate:((((_col4 + 1) = 2) and ((_col1 > 0) or (_col6 >= 0))) and ((_col1 + _col4) = 2)) (type: boolean)
Statistics:Num rows: 12 Data size: 1212 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_23]
+ Merge Join Operator [MERGEJOIN_24]
| condition map:[{"":"Right Outer Join0 to 1"},{"":"Right Outer Join0 to 2"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col3","_col4","_col6"]
@@ -2065,7 +2065,7 @@ Stage-0
| Select Operator [SEL_2]
| outputColumnNames:["_col0","_col1","_col2"]
| Statistics:Num rows: 6 Data size: 465 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_21]
+ | Filter Operator [FIL_22]
| predicate:(((c_int + 1) = 2) and ((c_int > 0) or (c_float >= 0.0))) (type: boolean)
| Statistics:Num rows: 6 Data size: 465 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_0]
@@ -2081,7 +2081,7 @@ Stage-0
| Select Operator [SEL_5]
| outputColumnNames:["_col0","_col1"]
| Statistics:Num rows: 6 Data size: 445 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_22]
+ | Filter Operator [FIL_23]
| predicate:(((c_int + 1) = 2) and ((c_int > 0) or (c_float >= 0.0))) (type: boolean)
| Statistics:Num rows: 6 Data size: 465 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_3]
@@ -2418,10 +2418,10 @@ Stage-0
Select Operator [SEL_39]
outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_38]
+ Filter Operator [FIL_54]
predicate:((_col1 > 0) or (_col6 >= 0)) (type: boolean)
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_60]
+ Merge Join Operator [MERGEJOIN_62]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col1","_col2","_col6"]
@@ -2436,7 +2436,7 @@ Stage-0
| Select Operator [SEL_32]
| outputColumnNames:["_col0","_col1"]
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_58]
+ | Filter Operator [FIL_60]
| predicate:key is not null (type: boolean)
| Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_31]
@@ -2452,10 +2452,10 @@ Stage-0
Select Operator [SEL_30]
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_29]
+ Filter Operator [FIL_55]
predicate:((_col1 + _col4) >= 0) (type: boolean)
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_59]
+ Merge Join Operator [MERGEJOIN_61]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
| outputColumnNames:["_col0","_col1","_col2","_col4"]
@@ -2467,7 +2467,7 @@ Stage-0
| sort order:+
| Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
| value expressions:_col1 (type: int)
- | Filter Operator [FIL_56]
+ | Filter Operator [FIL_58]
| predicate:_col0 is not null (type: boolean)
| Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
| Limit [LIM_22]
@@ -2502,7 +2502,7 @@ Stage-0
| keys:key (type: string), c_int (type: int), c_float (type: float)
| outputColumnNames:["_col0","_col1","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_57]
+ | Filter Operator [FIL_59]
| predicate:(((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) (type: boolean)
| Statistics:Num rows: 4 Data size: 372 Basic stats: COMPLETE Column stats: COMPLETE
| TableScan [TS_12]
@@ -2515,7 +2515,7 @@ Stage-0
sort order:+
Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col1 (type: int), _col2 (type: bigint)
- Filter Operator [FIL_54]
+ Filter Operator [FIL_56]
predicate:_col0 is not null (type: boolean)
Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
Limit [LIM_10]
@@ -2550,7 +2550,7 @@ Stage-0
keys:key (type: string), c_int (type: int), c_float (type: float)
outputColumnNames:["_col0","_col1","_col2","_col3"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_55]
+ Filter Operator [FIL_57]
predicate:(((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) (type: boolean)
Statistics:Num rows: 4 Data size: 372 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_0]
[10/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/union_fast_stats.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/union_fast_stats.q.out b/ql/src/test/results/clientpositive/llap/union_fast_stats.q.out
new file mode 100644
index 0000000..d0d82a0
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/union_fast_stats.q.out
@@ -0,0 +1,526 @@
+PREHOOK: query: drop table small_alltypesorc1a
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table small_alltypesorc1a
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: drop table small_alltypesorc2a
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table small_alltypesorc2a
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: drop table small_alltypesorc3a
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table small_alltypesorc3a
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: drop table small_alltypesorc4a
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table small_alltypesorc4a
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: drop table small_alltypesorc_a
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: drop table small_alltypesorc_a
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: create table small_alltypesorc1a as select * from alltypesorc where cint is not null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc1a
+POSTHOOK: query: create table small_alltypesorc1a as select * from alltypesorc where cint is not null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc1a
+PREHOOK: query: create table small_alltypesorc2a as select * from alltypesorc where cint is null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc2a
+POSTHOOK: query: create table small_alltypesorc2a as select * from alltypesorc where cint is null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc2a
+PREHOOK: query: create table small_alltypesorc3a as select * from alltypesorc where cint is not null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc3a
+POSTHOOK: query: create table small_alltypesorc3a as select * from alltypesorc where cint is not null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc3a
+PREHOOK: query: create table small_alltypesorc4a as select * from alltypesorc where cint is null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc4a
+POSTHOOK: query: create table small_alltypesorc4a as select * from alltypesorc where cint is null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc4a
+PREHOOK: query: create table small_alltypesorc_a stored as orc as select * from
+(select * from (select * from small_alltypesorc1a) sq1
+ union all
+ select * from (select * from small_alltypesorc2a) sq2
+ union all
+ select * from (select * from small_alltypesorc3a) sq3
+ union all
+ select * from (select * from small_alltypesorc4a) sq4) q
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@small_alltypesorc1a
+PREHOOK: Input: default@small_alltypesorc2a
+PREHOOK: Input: default@small_alltypesorc3a
+PREHOOK: Input: default@small_alltypesorc4a
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: create table small_alltypesorc_a stored as orc as select * from
+(select * from (select * from small_alltypesorc1a) sq1
+ union all
+ select * from (select * from small_alltypesorc2a) sq2
+ union all
+ select * from (select * from small_alltypesorc3a) sq3
+ union all
+ select * from (select * from small_alltypesorc4a) sq4) q
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@small_alltypesorc1a
+POSTHOOK: Input: default@small_alltypesorc2a
+POSTHOOK: Input: default@small_alltypesorc3a
+POSTHOOK: Input: default@small_alltypesorc4a
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc_a
+PREHOOK: query: desc formatted small_alltypesorc_a
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: query: desc formatted small_alltypesorc_a
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+# col_name data_type comment
+
+ctinyint tinyint
+csmallint smallint
+cint int
+cbigint bigint
+cfloat float
+cdouble double
+cstring1 string
+cstring2 string
+ctimestamp1 timestamp
+ctimestamp2 timestamp
+cboolean1 boolean
+cboolean2 boolean
+
+# Detailed Table Information
+Database: default
+#### A masked pattern was here ####
+Retention: 0
+#### A masked pattern was here ####
+Table Type: MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE true
+ numFiles 4
+ numRows 0
+ rawDataSize 0
+ totalSize 3915
+#### A masked pattern was here ####
+
+# Storage Information
+SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+Compressed: No
+Num Buckets: -1
+Bucket Columns: []
+Sort Columns: []
+Storage Desc Params:
+ serialization.format 1
+PREHOOK: query: ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS
+PREHOOK: type: QUERY
+PREHOOK: Input: default@small_alltypesorc_a
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: Output: default@small_alltypesorc_a
+PREHOOK: query: desc formatted small_alltypesorc_a
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: query: desc formatted small_alltypesorc_a
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+# col_name data_type comment
+
+ctinyint tinyint
+csmallint smallint
+cint int
+cbigint bigint
+cfloat float
+cdouble double
+cstring1 string
+cstring2 string
+ctimestamp1 timestamp
+ctimestamp2 timestamp
+cboolean1 boolean
+cboolean2 boolean
+
+# Detailed Table Information
+Database: default
+#### A masked pattern was here ####
+Retention: 0
+#### A masked pattern was here ####
+Table Type: MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE true
+ numFiles 4
+ numRows 15
+ rawDataSize 3483
+ totalSize 3915
+#### A masked pattern was here ####
+
+# Storage Information
+SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+Compressed: No
+Num Buckets: -1
+Bucket Columns: []
+Sort Columns: []
+Storage Desc Params:
+ serialization.format 1
+PREHOOK: query: insert into table small_alltypesorc_a select * from small_alltypesorc1a
+PREHOOK: type: QUERY
+PREHOOK: Input: default@small_alltypesorc1a
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: insert into table small_alltypesorc_a select * from small_alltypesorc1a
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@small_alltypesorc1a
+POSTHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: Lineage: small_alltypesorc_a.cbigint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cbigint, type:bigint, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cboolean1 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cboolean1, type:boolean, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cboolean2 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cboolean2, type:boolean, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cdouble SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cdouble, type:double, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cfloat SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cfloat, type:float, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cint, type:int, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.csmallint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:csmallint, type:smallint, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cstring1 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cstring1, type:string, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cstring2 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cstring2, type:string, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.ctimestamp1 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:ctimestamp1, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.ctimestamp2 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:ctimestamp2, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.ctinyint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:ctinyint, type:tinyint, comment:null), ]
+PREHOOK: query: desc formatted small_alltypesorc_a
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: query: desc formatted small_alltypesorc_a
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+# col_name data_type comment
+
+ctinyint tinyint
+csmallint smallint
+cint int
+cbigint bigint
+cfloat float
+cdouble double
+cstring1 string
+cstring2 string
+ctimestamp1 timestamp
+ctimestamp2 timestamp
+cboolean1 boolean
+cboolean2 boolean
+
+# Detailed Table Information
+Database: default
+#### A masked pattern was here ####
+Retention: 0
+#### A masked pattern was here ####
+Table Type: MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE true
+ numFiles 5
+ numRows 20
+ rawDataSize 4552
+ totalSize 5225
+#### A masked pattern was here ####
+
+# Storage Information
+SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+Compressed: No
+Num Buckets: -1
+Bucket Columns: []
+Sort Columns: []
+Storage Desc Params:
+ serialization.format 1
+PREHOOK: query: drop table small_alltypesorc1a
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@small_alltypesorc1a
+PREHOOK: Output: default@small_alltypesorc1a
+POSTHOOK: query: drop table small_alltypesorc1a
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@small_alltypesorc1a
+POSTHOOK: Output: default@small_alltypesorc1a
+PREHOOK: query: drop table small_alltypesorc2a
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@small_alltypesorc2a
+PREHOOK: Output: default@small_alltypesorc2a
+POSTHOOK: query: drop table small_alltypesorc2a
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@small_alltypesorc2a
+POSTHOOK: Output: default@small_alltypesorc2a
+PREHOOK: query: drop table small_alltypesorc3a
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@small_alltypesorc3a
+PREHOOK: Output: default@small_alltypesorc3a
+POSTHOOK: query: drop table small_alltypesorc3a
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@small_alltypesorc3a
+POSTHOOK: Output: default@small_alltypesorc3a
+PREHOOK: query: drop table small_alltypesorc4a
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@small_alltypesorc4a
+PREHOOK: Output: default@small_alltypesorc4a
+POSTHOOK: query: drop table small_alltypesorc4a
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@small_alltypesorc4a
+POSTHOOK: Output: default@small_alltypesorc4a
+PREHOOK: query: drop table small_alltypesorc_a
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: drop table small_alltypesorc_a
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: Output: default@small_alltypesorc_a
+PREHOOK: query: create table small_alltypesorc1a as select * from alltypesorc where cint is not null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc1a
+POSTHOOK: query: create table small_alltypesorc1a as select * from alltypesorc where cint is not null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc1a
+PREHOOK: query: create table small_alltypesorc2a as select * from alltypesorc where cint is null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc2a
+POSTHOOK: query: create table small_alltypesorc2a as select * from alltypesorc where cint is null and ctinyint is not null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc2a
+PREHOOK: query: create table small_alltypesorc3a as select * from alltypesorc where cint is not null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc3a
+POSTHOOK: query: create table small_alltypesorc3a as select * from alltypesorc where cint is not null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc3a
+PREHOOK: query: create table small_alltypesorc4a as select * from alltypesorc where cint is null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc4a
+POSTHOOK: query: create table small_alltypesorc4a as select * from alltypesorc where cint is null and ctinyint is null order by ctinyint, csmallint, cint, cbigint, cfloat, cdouble, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2 limit 5
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc4a
+PREHOOK: query: create table small_alltypesorc_a stored as orc as select * from
+(select * from (select * from small_alltypesorc1a) sq1
+ union all
+ select * from (select * from small_alltypesorc2a) sq2
+ union all
+ select * from (select * from small_alltypesorc3a) sq3
+ union all
+ select * from (select * from small_alltypesorc4a) sq4) q
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@small_alltypesorc1a
+PREHOOK: Input: default@small_alltypesorc2a
+PREHOOK: Input: default@small_alltypesorc3a
+PREHOOK: Input: default@small_alltypesorc4a
+PREHOOK: Output: database:default
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: create table small_alltypesorc_a stored as orc as select * from
+(select * from (select * from small_alltypesorc1a) sq1
+ union all
+ select * from (select * from small_alltypesorc2a) sq2
+ union all
+ select * from (select * from small_alltypesorc3a) sq3
+ union all
+ select * from (select * from small_alltypesorc4a) sq4) q
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@small_alltypesorc1a
+POSTHOOK: Input: default@small_alltypesorc2a
+POSTHOOK: Input: default@small_alltypesorc3a
+POSTHOOK: Input: default@small_alltypesorc4a
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@small_alltypesorc_a
+PREHOOK: query: desc formatted small_alltypesorc_a
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: query: desc formatted small_alltypesorc_a
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+# col_name data_type comment
+
+ctinyint tinyint
+csmallint smallint
+cint int
+cbigint bigint
+cfloat float
+cdouble double
+cstring1 string
+cstring2 string
+ctimestamp1 timestamp
+ctimestamp2 timestamp
+cboolean1 boolean
+cboolean2 boolean
+
+# Detailed Table Information
+Database: default
+#### A masked pattern was here ####
+Retention: 0
+#### A masked pattern was here ####
+Table Type: MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE true
+ numFiles 1
+ numRows 5
+ rawDataSize 1069
+ totalSize 3177
+#### A masked pattern was here ####
+
+# Storage Information
+SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+Compressed: No
+Num Buckets: -1
+Bucket Columns: []
+Sort Columns: []
+Storage Desc Params:
+ serialization.format 1
+PREHOOK: query: ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS
+PREHOOK: type: QUERY
+PREHOOK: Input: default@small_alltypesorc_a
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: Output: default@small_alltypesorc_a
+PREHOOK: query: desc formatted small_alltypesorc_a
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: query: desc formatted small_alltypesorc_a
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+# col_name data_type comment
+
+ctinyint tinyint
+csmallint smallint
+cint int
+cbigint bigint
+cfloat float
+cdouble double
+cstring1 string
+cstring2 string
+ctimestamp1 timestamp
+ctimestamp2 timestamp
+cboolean1 boolean
+cboolean2 boolean
+
+# Detailed Table Information
+Database: default
+#### A masked pattern was here ####
+Retention: 0
+#### A masked pattern was here ####
+Table Type: MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE true
+ numFiles 1
+ numRows 15
+ rawDataSize 3320
+ totalSize 3177
+#### A masked pattern was here ####
+
+# Storage Information
+SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+Compressed: No
+Num Buckets: -1
+Bucket Columns: []
+Sort Columns: []
+Storage Desc Params:
+ serialization.format 1
+PREHOOK: query: insert into table small_alltypesorc_a select * from small_alltypesorc1a
+PREHOOK: type: QUERY
+PREHOOK: Input: default@small_alltypesorc1a
+PREHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: query: insert into table small_alltypesorc_a select * from small_alltypesorc1a
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@small_alltypesorc1a
+POSTHOOK: Output: default@small_alltypesorc_a
+POSTHOOK: Lineage: small_alltypesorc_a.cbigint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cbigint, type:bigint, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cboolean1 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cboolean1, type:boolean, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cboolean2 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cboolean2, type:boolean, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cdouble SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cdouble, type:double, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cfloat SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cfloat, type:float, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cint, type:int, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.csmallint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:csmallint, type:smallint, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cstring1 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cstring1, type:string, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.cstring2 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:cstring2, type:string, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.ctimestamp1 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:ctimestamp1, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.ctimestamp2 SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:ctimestamp2, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: small_alltypesorc_a.ctinyint SIMPLE [(small_alltypesorc1a)small_alltypesorc1a.FieldSchema(name:ctinyint, type:tinyint, comment:null), ]
+PREHOOK: query: desc formatted small_alltypesorc_a
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@small_alltypesorc_a
+POSTHOOK: query: desc formatted small_alltypesorc_a
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@small_alltypesorc_a
+# col_name data_type comment
+
+ctinyint tinyint
+csmallint smallint
+cint int
+cbigint bigint
+cfloat float
+cdouble double
+cstring1 string
+cstring2 string
+ctimestamp1 timestamp
+ctimestamp2 timestamp
+cboolean1 boolean
+cboolean2 boolean
+
+# Detailed Table Information
+Database: default
+#### A masked pattern was here ####
+Retention: 0
+#### A masked pattern was here ####
+Table Type: MANAGED_TABLE
+Table Parameters:
+ COLUMN_STATS_ACCURATE true
+ numFiles 2
+ numRows 20
+ rawDataSize 4389
+ totalSize 4487
+#### A masked pattern was here ####
+
+# Storage Information
+SerDe Library: org.apache.hadoop.hive.ql.io.orc.OrcSerde
+InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+Compressed: No
+Num Buckets: -1
+Bucket Columns: []
+Sort Columns: []
+Storage Desc Params:
+ serialization.format 1
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out b/ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out
index 3624854..c086bcb 100644
--- a/ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_aggregate_9.q.out
@@ -125,10 +125,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: dc (type: decimal(38,18))
- outputColumnNames: _col0
+ outputColumnNames: dc
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0), max(_col0), sum(_col0), avg(_col0)
+ aggregations: min(dc), max(dc), sum(dc), avg(dc)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 1 Data size: 336 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out b/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
index 5671343..acf7149 100644
--- a/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_binary_join_groupby.q.out
@@ -223,11 +223,11 @@ STAGE PLANS:
Statistics: Num rows: 100 Data size: 29638 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: bin (type: binary)
- outputColumnNames: _col0
+ outputColumnNames: bin
Statistics: Num rows: 100 Data size: 29638 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: binary)
+ keys: bin (type: binary)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 100 Data size: 29638 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_char_cast.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_char_cast.q.out b/ql/src/test/results/clientpositive/llap/vector_char_cast.q.out
new file mode 100644
index 0000000..dbeae74
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/vector_char_cast.q.out
@@ -0,0 +1,35 @@
+PREHOOK: query: create table s1(id smallint) stored as orc
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@s1
+POSTHOOK: query: create table s1(id smallint) stored as orc
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@s1
+PREHOOK: query: insert into table s1 values (1000),(1001),(1002),(1003),(1000)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@values__tmp__table__1
+PREHOOK: Output: default@s1
+POSTHOOK: query: insert into table s1 values (1000),(1001),(1002),(1003),(1000)
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@values__tmp__table__1
+POSTHOOK: Output: default@s1
+POSTHOOK: Lineage: s1.id EXPRESSION [(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col1, type:string, comment:), ]
+PREHOOK: query: select count(1) from s1 where cast(id as char(4))='1000'
+PREHOOK: type: QUERY
+PREHOOK: Input: default@s1
+#### A masked pattern was here ####
+POSTHOOK: query: select count(1) from s1 where cast(id as char(4))='1000'
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@s1
+#### A masked pattern was here ####
+2
+PREHOOK: query: select count(1) from s1 where cast(id as char(4))='1000'
+PREHOOK: type: QUERY
+PREHOOK: Input: default@s1
+#### A masked pattern was here ####
+POSTHOOK: query: select count(1) from s1 where cast(id as char(4))='1000'
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@s1
+#### A masked pattern was here ####
+2
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out b/ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out
index c8f9c2c..95c2bc5 100644
--- a/ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_count_distinct.q.out
@@ -1258,10 +1258,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 3504000 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ws_order_number (type: int)
- outputColumnNames: _col0
+ outputColumnNames: ws_order_number
Statistics: Num rows: 2000 Data size: 3504000 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: ws_order_number (type: int)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 3504000 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_decimal_aggregate.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_decimal_aggregate.q.out b/ql/src/test/results/clientpositive/llap/vector_decimal_aggregate.q.out
index c0df9ab..8bcac50 100644
--- a/ql/src/test/results/clientpositive/llap/vector_decimal_aggregate.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_decimal_aggregate.q.out
@@ -54,11 +54,11 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2165060 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cint (type: int), cdecimal1 (type: decimal(20,10)), cdecimal2 (type: decimal(23,14))
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: cint, cdecimal1, cdecimal2
Statistics: Num rows: 12288 Data size: 2165060 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1), max(_col1), min(_col1), sum(_col1), count(_col2), max(_col2), min(_col2), sum(_col2), count()
- keys: _col0 (type: int)
+ aggregations: count(cdecimal1), max(cdecimal1), min(cdecimal1), sum(cdecimal1), count(cdecimal2), max(cdecimal2), min(cdecimal2), sum(cdecimal2), count()
+ keys: cint (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9
Statistics: Num rows: 12288 Data size: 2165060 Basic stats: COMPLETE Column stats: NONE
@@ -159,11 +159,11 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2165060 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cint (type: int), cdecimal1 (type: decimal(20,10)), cdecimal2 (type: decimal(23,14))
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: cint, cdecimal1, cdecimal2
Statistics: Num rows: 12288 Data size: 2165060 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1), max(_col1), min(_col1), sum(_col1), avg(_col1), stddev_pop(_col1), stddev_samp(_col1), count(_col2), max(_col2), min(_col2), sum(_col2), avg(_col2), stddev_pop(_col2), stddev_samp(_col2), count()
- keys: _col0 (type: int)
+ aggregations: count(cdecimal1), max(cdecimal1), min(cdecimal1), sum(cdecimal1), avg(cdecimal1), stddev_pop(cdecimal1), stddev_samp(cdecimal1), count(cdecimal2), max(cdecimal2), min(cdecimal2), sum(cdecimal2), avg(cdecimal2), stddev_pop(cdecimal2), stddev_samp(cdecimal2), count()
+ keys: cint (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15
Statistics: Num rows: 12288 Data size: 2165060 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out b/ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out
index 2cf76a3..4287a73 100644
--- a/ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_decimal_precision.q.out
@@ -567,10 +567,10 @@ STAGE PLANS:
Statistics: Num rows: 75 Data size: 3472 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: dec (type: decimal(20,10))
- outputColumnNames: _col0
+ outputColumnNames: dec
Statistics: Num rows: 75 Data size: 3472 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(_col0), sum(_col0)
+ aggregations: avg(dec), sum(dec)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_decimal_udf.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_decimal_udf.q.out b/ql/src/test/results/clientpositive/llap/vector_decimal_udf.q.out
index 15957e6..66303c3 100644
--- a/ql/src/test/results/clientpositive/llap/vector_decimal_udf.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_decimal_udf.q.out
@@ -1640,11 +1640,11 @@ STAGE PLANS:
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: int), key (type: decimal(20,10))
- outputColumnNames: _col0, _col1
+ outputColumnNames: value, key
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col1), count(_col1), avg(_col1)
- keys: _col0 (type: int)
+ aggregations: sum(key), count(key), avg(key)
+ keys: value (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
@@ -2320,11 +2320,11 @@ STAGE PLANS:
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: int), key (type: decimal(20,10))
- outputColumnNames: _col0, _col1
+ outputColumnNames: value, key
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: stddev(_col1), variance(_col1)
- keys: _col0 (type: int)
+ aggregations: stddev(key), variance(key)
+ keys: value (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
@@ -2407,11 +2407,11 @@ STAGE PLANS:
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: int), key (type: decimal(20,10))
- outputColumnNames: _col0, _col1
+ outputColumnNames: value, key
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: stddev_samp(_col1), var_samp(_col1)
- keys: _col0 (type: int)
+ aggregations: stddev_samp(key), var_samp(key)
+ keys: value (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
@@ -2561,10 +2561,10 @@ STAGE PLANS:
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: decimal(20,10))
- outputColumnNames: _col0
+ outputColumnNames: key
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(key)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: NONE
@@ -2628,10 +2628,10 @@ STAGE PLANS:
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: decimal(20,10))
- outputColumnNames: _col0
+ outputColumnNames: key
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(key)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column stats: NONE
@@ -2695,10 +2695,10 @@ STAGE PLANS:
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: decimal(20,10))
- outputColumnNames: _col0
+ outputColumnNames: key
Statistics: Num rows: 38 Data size: 4296 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(key)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_distinct_2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_distinct_2.q.out b/ql/src/test/results/clientpositive/llap/vector_distinct_2.q.out
index 3b8e5cd..4a758cb 100644
--- a/ql/src/test/results/clientpositive/llap/vector_distinct_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_distinct_2.q.out
@@ -128,35 +128,39 @@ STAGE PLANS:
alias: vectortab2korc
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: s (type: string), t (type: tinyint)
- outputColumnNames: _col0, _col1
+ expressions: t (type: tinyint), s (type: string)
+ outputColumnNames: t, s
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string), _col1 (type: tinyint)
+ keys: t (type: tinyint), s (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: tinyint)
+ key expressions: _col0 (type: tinyint), _col1 (type: string)
sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: tinyint)
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: string)
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
Reducer 2
Execution mode: vectorized, uber
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_groupby_3.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_groupby_3.q.out b/ql/src/test/results/clientpositive/llap/vector_groupby_3.q.out
index 93c5ce1..7c34c09 100644
--- a/ql/src/test/results/clientpositive/llap/vector_groupby_3.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_groupby_3.q.out
@@ -128,19 +128,19 @@ STAGE PLANS:
alias: vectortab2korc
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: s (type: string), t (type: tinyint), b (type: bigint)
- outputColumnNames: _col0, _col1, _col2
+ expressions: t (type: tinyint), s (type: string), b (type: bigint)
+ outputColumnNames: t, s, b
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col2)
- keys: _col0 (type: string), _col1 (type: tinyint)
+ aggregations: max(b)
+ keys: t (type: tinyint), s (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: tinyint)
+ key expressions: _col0 (type: tinyint), _col1 (type: string)
sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: tinyint)
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: string)
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: bigint)
Execution mode: vectorized, llap
@@ -149,17 +149,21 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0)
- keys: KEY._col0 (type: string), KEY._col1 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: tinyint), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
index c344a97..9a8379a 100644
--- a/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
@@ -245,10 +245,10 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ss_ticket_number (type: int)
- outputColumnNames: _col0
+ outputColumnNames: ss_ticket_number
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: ss_ticket_number (type: int)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
@@ -369,10 +369,10 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ss_ticket_number (type: int)
- outputColumnNames: _col0
+ outputColumnNames: ss_ticket_number
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: ss_ticket_number (type: int)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_grouping_sets.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_grouping_sets.q.out b/ql/src/test/results/clientpositive/llap/vector_grouping_sets.q.out
index d092c92..81fc754 100644
--- a/ql/src/test/results/clientpositive/llap/vector_grouping_sets.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_grouping_sets.q.out
@@ -126,10 +126,10 @@ STAGE PLANS:
Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: s_store_id (type: string)
- outputColumnNames: _col0
+ outputColumnNames: s_store_id
Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string), '0' (type: string)
+ keys: s_store_id (type: string), '0' (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
@@ -209,10 +209,10 @@ STAGE PLANS:
Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: s_store_id (type: string)
- outputColumnNames: _col0
+ outputColumnNames: s_store_id
Statistics: Num rows: 12 Data size: 25632 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string), '0' (type: string)
+ keys: s_store_id (type: string), '0' (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 24 Data size: 51264 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out b/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
index 4770f53..e4946aa 100644
--- a/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
@@ -111,20 +111,16 @@ STAGE PLANS:
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 4
Execution mode: vectorized, llap
@@ -257,20 +253,16 @@ STAGE PLANS:
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 4
Execution mode: vectorized, uber
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_orderby_5.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_orderby_5.q.out b/ql/src/test/results/clientpositive/llap/vector_orderby_5.q.out
index 4c79485..4ff8e99 100644
--- a/ql/src/test/results/clientpositive/llap/vector_orderby_5.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_orderby_5.q.out
@@ -126,11 +126,11 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: bo (type: boolean), b (type: bigint)
- outputColumnNames: _col0, _col1
+ outputColumnNames: bo, b
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1)
- keys: _col0 (type: boolean)
+ aggregations: max(b)
+ keys: bo (type: boolean)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out b/ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out
index 448c1cc..6904db2 100644
--- a/ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_outer_join2.q.out
@@ -227,19 +227,15 @@ STAGE PLANS:
1 Map 4
Statistics: Num rows: 24 Data size: 5361 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
- Select Operator
- expressions: _col1 (type: bigint)
- outputColumnNames: _col0
- Statistics: Num rows: 24 Data size: 5361 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), sum(_col0)
- mode: hash
- outputColumnNames: _col0, _col1
+ Group By Operator
+ aggregations: count(), sum(_col1)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Execution mode: vectorized, llap
Map 3
Map Operator Tree:
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_partition_diff_num_cols.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_partition_diff_num_cols.q.out b/ql/src/test/results/clientpositive/llap/vector_partition_diff_num_cols.q.out
index 230aba2..dd519fe 100644
--- a/ql/src/test/results/clientpositive/llap/vector_partition_diff_num_cols.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_partition_diff_num_cols.q.out
@@ -100,10 +100,10 @@ STAGE PLANS:
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: inv_quantity_on_hand (type: int)
- outputColumnNames: _col0
+ outputColumnNames: inv_quantity_on_hand
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(inv_quantity_on_hand)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -226,10 +226,10 @@ STAGE PLANS:
Statistics: Num rows: 200 Data size: 11876 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: inv_quantity_on_hand (type: int)
- outputColumnNames: _col0
+ outputColumnNames: inv_quantity_on_hand
Statistics: Num rows: 200 Data size: 11876 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(inv_quantity_on_hand)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -354,10 +354,10 @@ STAGE PLANS:
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: inv_quantity_on_hand (type: int)
- outputColumnNames: _col0
+ outputColumnNames: inv_quantity_on_hand
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(inv_quantity_on_hand)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -463,10 +463,10 @@ STAGE PLANS:
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: inv_quantity_on_hand (type: int)
- outputColumnNames: _col0
+ outputColumnNames: inv_quantity_on_hand
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(inv_quantity_on_hand)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -578,10 +578,10 @@ STAGE PLANS:
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: inv_quantity_on_hand (type: int)
- outputColumnNames: _col0
+ outputColumnNames: inv_quantity_on_hand
Statistics: Num rows: 200 Data size: 3176 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(inv_quantity_on_hand)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
index 87d75c2..a79aa17 100644
--- a/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_partitioned_date_time.q.out
@@ -379,11 +379,11 @@ STAGE PLANS:
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: fl_date (type: date)
- outputColumnNames: _col0
+ outputColumnNames: fl_date
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: date)
+ keys: fl_date (type: date)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 137 Data size: 39456 Basic stats: COMPLETE Column stats: NONE
@@ -1159,11 +1159,11 @@ STAGE PLANS:
Statistics: Num rows: 137 Data size: 31776 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: fl_date (type: date)
- outputColumnNames: _col0
+ outputColumnNames: fl_date
Statistics: Num rows: 137 Data size: 31776 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: date)
+ keys: fl_date (type: date)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 137 Data size: 31776 Basic stats: COMPLETE Column stats: NONE
@@ -1963,11 +1963,11 @@ STAGE PLANS:
Statistics: Num rows: 137 Data size: 33968 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: fl_time (type: timestamp)
- outputColumnNames: _col0
+ outputColumnNames: fl_time
Statistics: Num rows: 137 Data size: 33968 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: timestamp)
+ keys: fl_time (type: timestamp)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 137 Data size: 33968 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out b/ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out
index 55d0544..a32c246 100644
--- a/ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_reduce_groupby_decimal.q.out
@@ -44,22 +44,18 @@ STAGE PLANS:
Filter Operator
predicate: (cdecimal1 is not null and cdecimal2 is not null) (type: boolean)
Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: cint (type: int), cdouble (type: double), cdecimal1 (type: decimal(20,10)), cdecimal2 (type: decimal(23,14))
- outputColumnNames: _col0, _col1, _col2, _col3
+ Group By Operator
+ aggregations: min(cdecimal1)
+ keys: cint (type: int), cdouble (type: double), cdecimal1 (type: decimal(20,10)), cdecimal2 (type: decimal(23,14))
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(_col2)
- keys: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
+ sort order: ++++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
- sort order: ++++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: double), _col2 (type: decimal(20,10)), _col3 (type: decimal(23,14))
- Statistics: Num rows: 1526 Data size: 360136 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col4 (type: decimal(20,10))
+ value expressions: _col4 (type: decimal(20,10))
Execution mode: vectorized, llap
Reducer 2
Execution mode: vectorized, llap
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_0.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_0.q.out b/ql/src/test/results/clientpositive/llap/vectorization_0.q.out
index 269c6ac..85ed627 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_0.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_0.q.out
@@ -133,10 +133,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint)
- outputColumnNames: _col0
+ outputColumnNames: ctinyint
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(ctinyint)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -440,10 +440,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cbigint (type: bigint)
- outputColumnNames: _col0
+ outputColumnNames: cbigint
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(cbigint)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -747,10 +747,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cfloat (type: float)
- outputColumnNames: _col0
+ outputColumnNames: cfloat
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(cfloat)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -1001,10 +1001,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cbigint (type: bigint), cfloat (type: float), ctinyint (type: tinyint)
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: cbigint, cfloat, ctinyint
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(_col0), stddev_pop(_col0), var_samp(_col0), count(), sum(_col1), min(_col2)
+ aggregations: avg(cbigint), stddev_pop(cbigint), var_samp(cbigint), count(), sum(cfloat), min(ctinyint)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: NONE
[14/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-11893: LLAP: Update llap golden files after master merge (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/73eb9239
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/73eb9239
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/73eb9239
Branch: refs/heads/master
Commit: 73eb9239a7cd37c16b9eaff6af6201d3a1711b18
Parents: 1373ce7
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Fri Sep 18 19:28:56 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Fri Sep 18 19:28:56 2015 -0500
----------------------------------------------------------------------
.../queries/clientpositive/unionDistinct_1.q | 5 +
ql/src/test/queries/clientpositive/union_view.q | 2 +
.../acid_vectorization_partition.q.out | 14 +-
.../results/clientpositive/llap/count.q.out | 14 +-
.../llap/dynamic_partition_pruning.q.out | 88 +-
.../llap/dynpart_sort_opt_vectorization.q.out | 92 +-
.../llap/dynpart_sort_optimization.q.out | 92 +-
.../clientpositive/llap/explainuser_1.q.out | 2319 +++++++++---------
.../clientpositive/llap/explainuser_2.q.out | 782 +++---
.../results/clientpositive/llap/having.q.out | 62 +-
.../clientpositive/llap/limit_pushdown.q.out | 34 +-
.../clientpositive/llap/mapjoin_mapjoin.q.out | 24 +-
.../llap/metadata_only_queries.q.out | 4 +-
.../clientpositive/llap/metadataonly1.q.out | 44 +-
.../test/results/clientpositive/llap/mrr.q.out | 94 +-
.../llap/selectDistinctStar.q.out | 44 +-
.../llap/show_create_database.q.out | 19 +
.../clientpositive/llap/stats_only_null.q.out | 8 +-
.../clientpositive/llap/subquery_in.q.out | 36 +-
.../results/clientpositive/llap/tez_dml.q.out | 6 +-
.../results/clientpositive/llap/union5.q.out | 44 +-
.../results/clientpositive/llap/union7.q.out | 28 +-
.../clientpositive/llap/unionDistinct_1.q.out | 142 +-
.../clientpositive/llap/union_fast_stats.q.out | 526 ++++
.../llap/vector_aggregate_9.q.out | 4 +-
.../llap/vector_binary_join_groupby.q.out | 4 +-
.../clientpositive/llap/vector_char_cast.q.out | 35 +
.../llap/vector_count_distinct.q.out | 4 +-
.../llap/vector_decimal_aggregate.q.out | 12 +-
.../llap/vector_decimal_precision.q.out | 4 +-
.../llap/vector_decimal_udf.q.out | 30 +-
.../clientpositive/llap/vector_distinct_2.q.out | 28 +-
.../clientpositive/llap/vector_groupby_3.q.out | 30 +-
.../llap/vector_groupby_reduce.q.out | 8 +-
.../llap/vector_grouping_sets.q.out | 8 +-
.../llap/vector_mapjoin_reduce.q.out | 36 +-
.../clientpositive/llap/vector_orderby_5.q.out | 6 +-
.../llap/vector_outer_join2.q.out | 20 +-
.../llap/vector_partition_diff_num_cols.q.out | 20 +-
.../llap/vector_partitioned_date_time.q.out | 12 +-
.../llap/vector_reduce_groupby_decimal.q.out | 24 +-
.../clientpositive/llap/vectorization_0.q.out | 16 +-
.../clientpositive/llap/vectorization_13.q.out | 32 +-
.../clientpositive/llap/vectorization_15.q.out | 16 +-
.../clientpositive/llap/vectorization_16.q.out | 16 +-
.../clientpositive/llap/vectorization_9.q.out | 16 +-
.../llap/vectorization_limit.q.out | 14 +-
.../llap/vectorization_pushdown.q.out | 4 +-
.../llap/vectorization_short_regress.q.out | 74 +-
.../llap/vectorized_distinct_gby.q.out | 8 +-
.../vectorized_dynamic_partition_pruning.q.out | 88 +-
.../llap/vectorized_nested_mapjoin.q.out | 18 +-
.../llap/vectorized_parquet.q.out | 6 +-
.../llap/vectorized_timestamp_funcs.q.out | 12 +-
.../clientpositive/tez/unionDistinct_1.q.out | 32 +
.../results/clientpositive/tez/union_view.q.out | 167 ++
.../clientpositive/unionDistinct_1.q.out | 32 +
.../vector_leftsemi_mapjoin.q.out | 8 +
.../clientpositive/vector_multi_insert.q.out | 8 +
59 files changed, 3071 insertions(+), 2304 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/queries/clientpositive/unionDistinct_1.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/unionDistinct_1.q b/ql/src/test/queries/clientpositive/unionDistinct_1.q
index 984ce2e..a86b37a 100644
--- a/ql/src/test/queries/clientpositive/unionDistinct_1.q
+++ b/ql/src/test/queries/clientpositive/unionDistinct_1.q
@@ -995,6 +995,11 @@ insert overwrite table src10_2 select *
insert overwrite table src10_3 select *
insert overwrite table src10_4 select *;
+analyze table src10_1 compute statistics;
+analyze table src10_2 compute statistics;
+analyze table src10_3 compute statistics;
+analyze table src10_4 compute statistics;
+
set hive.auto.convert.join=true;
-- When we convert the Join of sub1 and sub0 into a MapJoin,
-- we can use a single MR job to evaluate this entire query.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/queries/clientpositive/union_view.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/union_view.q b/ql/src/test/queries/clientpositive/union_view.q
index b727199..733ed3e 100644
--- a/ql/src/test/queries/clientpositive/union_view.q
+++ b/ql/src/test/queries/clientpositive/union_view.q
@@ -1,4 +1,6 @@
set hive.stats.dbclass=fs;
+set hive.explain.user=false;
+
CREATE TABLE src_union_1 (key int, value string) PARTITIONED BY (ds string);
CREATE INDEX src_union_1_key_idx ON TABLE src_union_1(key) AS 'COMPACT' WITH DEFERRED REBUILD;
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/acid_vectorization_partition.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/acid_vectorization_partition.q.out b/ql/src/test/results/clientpositive/acid_vectorization_partition.q.out
index ee97cc9..f608b10 100644
--- a/ql/src/test/results/clientpositive/acid_vectorization_partition.q.out
+++ b/ql/src/test/results/clientpositive/acid_vectorization_partition.q.out
@@ -26,13 +26,13 @@ POSTHOOK: Input: default@alltypesorc
POSTHOOK: Output: default@acid_vectorized_part@ds=tomorrow
POSTHOOK: Lineage: acid_vectorized_part PARTITION(ds=tomorrow).a SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cint, type:int, comment:null), ]
POSTHOOK: Lineage: acid_vectorized_part PARTITION(ds=tomorrow).b SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cstring1, type:string, comment:null), ]
-PREHOOK: query: select * from acid_vectorized_part order by a, b
+PREHOOK: query: select * from acid_vectorized_part order by a, b, ds
PREHOOK: type: QUERY
PREHOOK: Input: default@acid_vectorized_part
PREHOOK: Input: default@acid_vectorized_part@ds=today
PREHOOK: Input: default@acid_vectorized_part@ds=tomorrow
#### A masked pattern was here ####
-POSTHOOK: query: select * from acid_vectorized_part order by a, b
+POSTHOOK: query: select * from acid_vectorized_part order by a, b, ds
POSTHOOK: type: QUERY
POSTHOOK: Input: default@acid_vectorized_part
POSTHOOK: Input: default@acid_vectorized_part@ds=today
@@ -40,21 +40,21 @@ POSTHOOK: Input: default@acid_vectorized_part@ds=tomorrow
#### A masked pattern was here ####
-1073279343 oj1YrV5Wa today
-1073279343 oj1YrV5Wa tomorrow
--1073051226 A34p7oRr2WvUJNf tomorrow
-1073051226 A34p7oRr2WvUJNf today
--1072910839 0iqrc5 tomorrow
+-1073051226 A34p7oRr2WvUJNf tomorrow
-1072910839 0iqrc5 today
+-1072910839 0iqrc5 tomorrow
-1072081801 dPkN74F7 today
-1072081801 dPkN74F7 tomorrow
-1072076362 2uLyD28144vklju213J1mr today
-1072076362 2uLyD28144vklju213J1mr tomorrow
--1071480828 aw724t8c5558x2xneC624 tomorrow
-1071480828 aw724t8c5558x2xneC624 today
+-1071480828 aw724t8c5558x2xneC624 tomorrow
-1071363017 Anj0oF today
-1071363017 Anj0oF tomorrow
--1070883071 0ruyd6Y50JpdGRf6HqD tomorrow
-1070883071 0ruyd6Y50JpdGRf6HqD today
+-1070883071 0ruyd6Y50JpdGRf6HqD tomorrow
-1070551679 iUR3Q today
-1070551679 iUR3Q tomorrow
--1069736047 k17Am8uPHWk02cEf1jet tomorrow
-1069736047 k17Am8uPHWk02cEf1jet today
+-1069736047 k17Am8uPHWk02cEf1jet tomorrow
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/count.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/count.q.out b/ql/src/test/results/clientpositive/llap/count.q.out
index d59b063..e8d5a8a 100644
--- a/ql/src/test/results/clientpositive/llap/count.q.out
+++ b/ql/src/test/results/clientpositive/llap/count.q.out
@@ -53,11 +53,11 @@ STAGE PLANS:
Statistics: Num rows: 4 Data size: 78 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: a (type: int), b (type: int), c (type: int), d (type: int)
- outputColumnNames: _col0, _col1, _col2, _col3
+ outputColumnNames: a, b, c, d
Statistics: Num rows: 4 Data size: 78 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(DISTINCT _col1), count(DISTINCT _col2), sum(_col3)
- keys: _col0 (type: int), _col1 (type: int), _col2 (type: int)
+ aggregations: count(DISTINCT b), count(DISTINCT c), sum(d)
+ keys: a (type: int), b (type: int), c (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 4 Data size: 78 Basic stats: COMPLETE Column stats: NONE
@@ -192,14 +192,14 @@ STAGE PLANS:
Statistics: Num rows: 4 Data size: 78 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: a (type: int), b (type: int), c (type: int), d (type: int)
- outputColumnNames: _col0, _col1, _col2, _col3
+ outputColumnNames: a, b, c, d
Statistics: Num rows: 4 Data size: 78 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int), _col2 (type: int)
+ key expressions: a (type: int), b (type: int), c (type: int)
sort order: +++
- Map-reduce partition columns: _col0 (type: int)
+ Map-reduce partition columns: a (type: int)
Statistics: Num rows: 4 Data size: 78 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: int)
+ value expressions: d (type: int)
Execution mode: llap
Reducer 2
Execution mode: uber
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
index 5a67250..cdb11b6 100644
--- a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
@@ -59,10 +59,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
@@ -1831,7 +1831,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11
#### A masked pattern was here ####
1000
-Warning: Shuffle Join MERGEJOIN[23][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[21][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: -- parent is reduce tasks
EXPLAIN select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
@@ -1871,10 +1871,10 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: '2008-04-08' (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
@@ -1938,7 +1938,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[23][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[21][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
PREHOOK: Input: default@srcpart
@@ -2835,10 +2835,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -2854,10 +2854,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3039,10 +3039,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3058,10 +3058,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3230,20 +3230,16 @@ STAGE PLANS:
alias: srcpart
filterExpr: ds is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: ds (type: string)
+ Group By Operator
+ keys: ds (type: string)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 10
Map Operator Tree:
@@ -3252,10 +3248,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3270,20 +3266,16 @@ STAGE PLANS:
alias: srcpart
filterExpr: ds is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: ds (type: string)
+ Group By Operator
+ keys: ds (type: string)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 7
Map Operator Tree:
@@ -3292,10 +3284,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -4270,7 +4262,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11
#### A masked pattern was here ####
1000
-Warning: Map Join MAPJOIN[23][bigTable=?] in task 'Map 1' is a cross product
+Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
PREHOOK: query: -- parent is reduce tasks
EXPLAIN select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
@@ -4326,10 +4318,10 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: '2008-04-08' (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
@@ -4374,7 +4366,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Map Join MAPJOIN[23][bigTable=?] in task 'Map 1' is a cross product
+Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
PREHOOK: Input: default@srcpart
@@ -4984,10 +4976,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -5003,10 +4995,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
index 4f291dd..07a87c8 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_opt_vectorization.q.out
@@ -1591,37 +1591,37 @@ STAGE PLANS:
Filter Operator
predicate: (t is null or (t = 27)) (type: boolean)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: si (type: smallint), i (type: int), b (type: bigint), f (type: float), t (type: tinyint)
+ Group By Operator
+ keys: t (type: tinyint), si (type: smallint), i (type: int), b (type: bigint), f (type: float)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
+ sort order: +++++
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- sort order: +++++
- Map-reduce partition columns: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
Reducer 2
Execution mode: vectorized, uber
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: smallint), KEY._col1 (type: int), KEY._col2 (type: bigint), KEY._col3 (type: float), KEY._col4 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: smallint), KEY._col2 (type: int), KEY._col3 (type: bigint), KEY._col4 (type: float)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.over1k_part2_orc
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.over1k_part2_orc
Stage: Stage-2
Dependency Collection
@@ -1659,7 +1659,6 @@ STAGE PLANS:
Tez
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1670,50 +1669,37 @@ STAGE PLANS:
Filter Operator
predicate: (t is null or (t = 27)) (type: boolean)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: si (type: smallint), i (type: int), b (type: bigint), f (type: float), t (type: tinyint)
+ Group By Operator
+ keys: t (type: tinyint), si (type: smallint), i (type: int), b (type: bigint), f (type: float)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
+ sort order: +++++
+ Map-reduce partition columns: _col0 (type: tinyint)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- sort order: +++++
- Map-reduce partition columns: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
Reducer 2
- Execution mode: vectorized, llap
+ Execution mode: vectorized, uber
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: smallint), KEY._col1 (type: int), KEY._col2 (type: bigint), KEY._col3 (type: float), KEY._col4 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: smallint), KEY._col2 (type: int), KEY._col3 (type: bigint), KEY._col4 (type: float)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col4 (type: tinyint)
- sort order: +
- Map-reduce partition columns: _col4 (type: tinyint)
- Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Reducer 3
- Execution mode: vectorized, uber
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: smallint), VALUE._col1 (type: int), VALUE._col2 (type: bigint), VALUE._col3 (type: float), VALUE._col4 (type: tinyint)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
- Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.over1k_part2_orc
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.over1k_part2_orc
Stage: Stage-2
Dependency Collection
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
index 356aafc..2534254 100644
--- a/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynpart_sort_optimization.q.out
@@ -1522,37 +1522,37 @@ STAGE PLANS:
Filter Operator
predicate: (t is null or (t = 27)) (type: boolean)
Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: si (type: smallint), i (type: int), b (type: bigint), f (type: float), t (type: tinyint)
+ Group By Operator
+ keys: t (type: tinyint), si (type: smallint), i (type: int), b (type: bigint), f (type: float)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
+ sort order: +++++
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- sort order: +++++
- Map-reduce partition columns: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
Execution mode: uber
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: smallint), KEY._col1 (type: int), KEY._col2 (type: bigint), KEY._col3 (type: float), KEY._col4 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: smallint), KEY._col2 (type: int), KEY._col3 (type: bigint), KEY._col4 (type: float)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.over1k_part2
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.over1k_part2
Stage: Stage-2
Dependency Collection
@@ -1590,7 +1590,6 @@ STAGE PLANS:
Tez
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1601,50 +1600,37 @@ STAGE PLANS:
Filter Operator
predicate: (t is null or (t = 27)) (type: boolean)
Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: si (type: smallint), i (type: int), b (type: bigint), f (type: float), t (type: tinyint)
+ Group By Operator
+ keys: t (type: tinyint), si (type: smallint), i (type: int), b (type: bigint), f (type: float)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
+ sort order: +++++
+ Map-reduce partition columns: _col0 (type: tinyint)
Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- sort order: +++++
- Map-reduce partition columns: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Statistics: Num rows: 4442 Data size: 106611 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Reducer 2
- Execution mode: llap
+ Execution mode: uber
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: smallint), KEY._col1 (type: int), KEY._col2 (type: bigint), KEY._col3 (type: float), KEY._col4 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: smallint), KEY._col2 (type: int), KEY._col3 (type: bigint), KEY._col4 (type: float)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col4 (type: tinyint)
- sort order: +
- Map-reduce partition columns: _col4 (type: tinyint)
- Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Reducer 3
- Execution mode: uber
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: smallint), VALUE._col1 (type: int), VALUE._col2 (type: bigint), VALUE._col3 (type: float), VALUE._col4 (type: tinyint)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
- Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.over1k_part2
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 2221 Data size: 53305 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.over1k_part2
Stage: Stage-2
Dependency Collection
[25/51] [abbrv] hive git commit: HIVE-12001 : LLAP: update some out
files (Sergey Shelukhin)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/7273a4c4/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
index 9a8379a..b7b9a1b 100644
--- a/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_groupby_reduce.q.out
@@ -321,12 +321,9 @@ POSTHOOK: Input: default@store_sales
18
19
20
-PREHOOK: query: -- The Reduce task has 2 MergePartial GROUP BY operators in a row. Currently,
--- we don't issue startGroup with keys out of the 1st vectorized GROUP BY, so we can't
--- vectorize the 2nd GROUP BY...
-explain
+PREHOOK: query: explain
select
- min(ss_ticket_number)
+ min(ss_ticket_number) m
from
(select
ss_ticket_number
@@ -334,14 +331,11 @@ from
store_sales
group by ss_ticket_number) a
group by ss_ticket_number
-limit 20
+order by m
PREHOOK: type: QUERY
-POSTHOOK: query: -- The Reduce task has 2 MergePartial GROUP BY operators in a row. Currently,
--- we don't issue startGroup with keys out of the 1st vectorized GROUP BY, so we can't
--- vectorize the 2nd GROUP BY...
-explain
+POSTHOOK: query: explain
select
- min(ss_ticket_number)
+ min(ss_ticket_number) m
from
(select
ss_ticket_number
@@ -349,7 +343,7 @@ from
store_sales
group by ss_ticket_number) a
group by ss_ticket_number
-limit 20
+order by m
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
@@ -360,6 +354,7 @@ STAGE PLANS:
Tez
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -383,7 +378,7 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized, llap
Reducer 2
- Execution mode: uber
+ Execution mode: vectorized, llap
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -400,25 +395,33 @@ STAGE PLANS:
expressions: _col1 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
- Limit
- Number of rows: 20
- Statistics: Num rows: 20 Data size: 1760 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 20 Data size: 1760 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Reducer 3
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
- limit: 20
+ limit: -1
Processor Tree:
ListSink
PREHOOK: query: select
- min(ss_ticket_number)
+ min(ss_ticket_number) m
from
(select
ss_ticket_number
@@ -426,12 +429,12 @@ from
store_sales
group by ss_ticket_number) a
group by ss_ticket_number
-limit 20
+order by m
PREHOOK: type: QUERY
PREHOOK: Input: default@store_sales
#### A masked pattern was here ####
POSTHOOK: query: select
- min(ss_ticket_number)
+ min(ss_ticket_number) m
from
(select
ss_ticket_number
@@ -439,7 +442,7 @@ from
store_sales
group by ss_ticket_number) a
group by ss_ticket_number
-limit 20
+order by m
POSTHOOK: type: QUERY
POSTHOOK: Input: default@store_sales
#### A masked pattern was here ####
@@ -463,3 +466,1417 @@ POSTHOOK: Input: default@store_sales
18
19
20
+21
+22
+23
+24
+25
+26
+27
+28
+29
+30
+31
+32
+33
+34
+35
+36
+37
+38
+39
+40
+41
+42
+43
+44
+45
+46
+47
+48
+49
+50
+51
+52
+53
+54
+55
+56
+57
+58
+59
+60
+61
+62
+63
+64
+65
+66
+67
+68
+69
+70
+71
+72
+73
+74
+75
+76
+77
+78
+79
+80
+81
+82
+PREHOOK: query: explain
+select
+ ss_ticket_number, sum(ss_item_sk), sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number
+order by ss_ticket_number
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select
+ ss_ticket_number, sum(ss_item_sk), sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number
+order by ss_ticket_number
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: store_sales
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: ss_item_sk (type: int), ss_ticket_number (type: int), ss_quantity (type: int)
+ outputColumnNames: ss_item_sk, ss_ticket_number, ss_quantity
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: min(ss_quantity)
+ keys: ss_item_sk (type: int), ss_ticket_number (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: int)
+ Execution mode: vectorized, llap
+ Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: min(VALUE._col0)
+ keys: KEY._col0 (type: int), KEY._col1 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: sum(_col0), sum(_col2)
+ keys: _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Reducer 3
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: sum(VALUE._col0), sum(VALUE._col1)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Reducer 4
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 (type: bigint), VALUE._col1 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select
+ ss_ticket_number, sum(ss_item_sk), sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number
+order by ss_ticket_number
+PREHOOK: type: QUERY
+PREHOOK: Input: default@store_sales
+#### A masked pattern was here ####
+POSTHOOK: query: select
+ ss_ticket_number, sum(ss_item_sk), sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number
+order by ss_ticket_number
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@store_sales
+#### A masked pattern was here ####
+1 85411 816
+2 157365 812
+3 147948 710
+4 69545 411
+5 163232 840
+6 86307 627
+7 114874 563
+8 117953 662
+9 173250 690
+10 60338 602
+11 138545 657
+12 97181 586
+13 109484 555
+14 137333 442
+15 176829 652
+16 115004 654
+17 105008 460
+18 165135 738
+19 128252 831
+20 104789 374
+21 72771 469
+22 128153 449
+23 110253 603
+24 100662 1029
+25 118714 760
+26 81596 502
+27 164068 871
+28 58632 409
+29 133777 417
+30 130451 772
+31 114967 586
+32 142021 592
+33 151818 691
+34 112559 662
+35 137027 780
+36 118285 538
+37 94528 401
+38 81368 521
+39 101064 937
+40 84435 480
+41 112444 688
+42 95731 840
+43 57298 410
+44 159880 839
+45 68919 474
+46 111212 374
+47 78210 416
+48 94459 445
+49 90879 589
+50 37821 407
+51 124927 612
+52 98099 489
+53 138706 609
+54 87478 354
+55 90290 406
+56 78812 372
+57 101175 597
+58 88044 202
+59 104582 753
+60 99218 900
+61 66514 392
+62 126713 527
+63 98778 648
+64 131659 380
+65 86990 494
+66 108808 492
+67 75250 711
+68 91671 548
+69 92821 405
+70 75021 319
+71 124484 748
+72 161470 744
+73 104358 621
+74 88609 688
+75 92940 649
+76 75853 580
+77 124755 873
+78 98285 573
+79 160595 581
+80 151471 704
+81 105109 429
+82 55611 254
+PREHOOK: query: explain
+select
+ ss_ticket_number, ss_item_sk, sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number, ss_item_sk
+order by ss_ticket_number, ss_item_sk
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select
+ ss_ticket_number, ss_item_sk, sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number, ss_item_sk
+order by ss_ticket_number, ss_item_sk
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: store_sales
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: ss_item_sk (type: int), ss_ticket_number (type: int), ss_quantity (type: int)
+ outputColumnNames: ss_item_sk, ss_ticket_number, ss_quantity
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: min(ss_quantity)
+ keys: ss_item_sk (type: int), ss_ticket_number (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: int)
+ Execution mode: vectorized, llap
+ Reducer 2
+ Execution mode: vectorized, llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: min(VALUE._col0)
+ keys: KEY._col0 (type: int), KEY._col1 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: sum(_col2)
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: complete
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: int), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: bigint)
+ Reducer 3
+ Execution mode: vectorized, uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: int), VALUE._col0 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select
+ ss_ticket_number, ss_item_sk, sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number, ss_item_sk
+order by ss_ticket_number, ss_item_sk
+PREHOOK: type: QUERY
+PREHOOK: Input: default@store_sales
+#### A masked pattern was here ####
+POSTHOOK: query: select
+ ss_ticket_number, ss_item_sk, sum(q)
+from
+ (select
+ ss_ticket_number, ss_item_sk, min(ss_quantity) q
+ from
+ store_sales
+ group by ss_ticket_number, ss_item_sk) a
+group by ss_ticket_number, ss_item_sk
+order by ss_ticket_number, ss_item_sk
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@store_sales
+#### A masked pattern was here ####
+1 49 5
+1 173 65
+1 1553 50
+1 3248 58
+1 3617 79
+1 4553 100
+1 4583 72
+1 4682 44
+1 5527 88
+1 5981 14
+1 10993 91
+1 13283 37
+1 13538 14
+1 13631 99
+2 1363 4
+2 2930 36
+2 3740 49
+2 6928 65
+2 7654 25
+2 9436 79
+2 10768 30
+2 12068 74
+2 12223 78
+2 13340 71
+2 13927 93
+2 14701 58
+2 15085 88
+2 15782 62
+2 17420 NULL
+3 246 96
+3 1531 NULL
+3 3525 42
+3 4698 98
+3 5355 53
+3 10693 27
+3 12447 82
+3 13021 64
+3 14100 79
+3 14443 4
+3 15786 56
+3 16869 4
+3 17263 17
+3 17971 88
+4 163 17
+4 1576 74
+4 5350 86
+4 5515 23
+4 6988 23
+4 7990 56
+4 8452 27
+4 9685 21
+4 11036 41
+4 12790 43
+5 1808 NULL
+5 1940 60
+5 5842 50
+5 6068 76
+5 6466 36
+5 11324 52
+5 11590 15
+5 12650 66
+5 13562 64
+5 13958 60
+5 14599 83
+5 14686 91
+5 15752 66
+5 16195 50
+5 16792 71
+6 2549 62
+6 2647 100
+6 3049 31
+6 3291 100
+6 6437 72
+6 8621 NULL
+6 10355 94
+6 10895 1
+6 11705 61
+6 13245 64
+6 13513 42
+7 4627 9
+7 4795 73
+7 4833 88
+7 5183 51
+7 5905 69
+7 8955 54
+7 9751 4
+7 10487 52
+7 12571 82
+7 15179 12
+7 15333 NULL
+7 17255 69
+8 665 31
+8 4183 90
+8 5929 83
+8 7115 54
+8 11365 7
+8 11893 95
+8 12041 95
+8 13427 87
+8 16671 20
+8 17119 51
+8 17545 49
+9 69 11
+9 889 6
+9 1185 62
+9 4623 34
+9 7945 83
+9 8334 71
+9 12027 27
+9 12969 59
+9 13483 NULL
+9 13717 53
+9 15133 15
+9 16083 32
+9 16363 54
+9 16461 66
+9 16659 84
+9 17310 33
+10 755 74
+10 1425 92
+10 1511 76
+10 3433 83
+10 3933 52
+10 4357 17
+10 5863 47
+10 9811 28
+10 13803 66
+10 15447 67
+11 157 84
+11 1315 70
+11 7519 68
+11 7608 66
+11 9901 57
+11 10699 33
+11 11490 NULL
+11 11991 38
+11 12438 16
+11 15157 96
+11 15649 33
+11 17226 11
+11 17395 85
+12 373 57
+12 1591 82
+12 4888 56
+12 6148 36
+12 6248 36
+12 9616 66
+12 9788 73
+12 13399 46
+12 14746 26
+12 14944 9
+12 15440 99
+13 868 NULL
+13 1760 12
+13 1898 NULL
+13 2108 9
+13 2191 NULL
+13 4430 73
+13 5971 80
+13 6085 58
+13 6140 15
+13 6682 80
+13 7640 48
+13 7723 27
+13 10096 12
+13 11758 34
+13 16894 87
+13 17240 20
+14 177 41
+14 769 20
+14 4507 4
+14 10175 19
+14 11549 6
+14 11653 60
+14 11817 81
+14 12587 NULL
+14 13069 77
+14 13515 57
+14 13845 17
+14 16741 46
+14 16929 14
+15 4241 21
+15 4505 59
+15 4777 28
+15 7391 98
+15 8336 15
+15 8353 NULL
+15 8690 32
+15 8707 21
+15 10361 39
+15 11659 80
+15 13172 25
+15 16619 81
+15 17267 7
+15 17330 82
+15 17564 26
+15 17857 38
+16 457 60
+16 1888 4
+16 4144 94
+16 6008 59
+16 7504 51
+16 8887 35
+16 9769 42
+16 9790 17
+16 9997 94
+16 11168 86
+16 11920 29
+16 16226 13
+16 17246 70
+17 2092 37
+17 4678 34
+17 6811 70
+17 9214 57
+17 10543 54
+17 11203 21
+17 13177 45
+17 13826 32
+17 15781 76
+17 17683 34
+18 2440 40
+18 5251 41
+18 7378 94
+18 8779 9
+18 8884 18
+18 9886 62
+18 11584 76
+18 11890 7
+18 12602 81
+18 12826 93
+18 12860 18
+18 14011 95
+18 14372 76
+18 14377 15
+18 17995 13
+19 1094 48
+19 3133 96
+19 3376 84
+19 4882 84
+19 6772 97
+19 7087 1
+19 7814 29
+19 8662 97
+19 9094 49
+19 9346 39
+19 10558 82
+19 10651 46
+19 11914 59
+19 16330 NULL
+19 17539 20
+20 1451 89
+20 2618 4
+20 5312 9
+20 5425 15
+20 5483 8
+20 6026 21
+20 7207 90
+20 8714 NULL
+20 9086 4
+20 9800 32
+20 13601 17
+20 14935 NULL
+20 15131 85
+21 230 48
+21 1810 59
+21 2870 50
+21 5170 45
+21 5998 51
+21 6476 49
+21 9187 14
+21 12266 47
+21 14368 18
+21 14396 88
+22 9985 70
+22 10474 31
+22 11599 66
+22 12415 10
+22 15310 15
+22 16396 85
+22 16922 88
+22 17392 14
+22 17660 70
+23 319 86
+23 7242 37
+23 8181 13
+23 8413 1
+23 9093 38
+23 9097 81
+23 11220 91
+23 11257 64
+23 12397 80
+23 15403 96
+23 17631 16
+24 407 53
+24 1389 72
+24 1795 21
+24 2497 85
+24 3103 73
+24 4425 57
+24 4749 28
+24 4873 41
+24 5653 92
+24 6043 1
+24 6751 82
+24 7375 97
+24 10265 93
+24 11551 48
+24 13303 97
+24 16483 89
+25 1333 55
+25 2150 100
+25 2608 76
+25 3454 100
+25 4880 29
+25 5954 34
+25 6955 40
+25 7874 65
+25 9472 48
+25 10159 24
+25 14488 26
+25 14635 68
+25 17000 40
+25 17752 55
+26 1989 26
+26 5053 4
+26 5385 97
+26 5721 81
+26 6647 64
+26 7337 45
+26 9679 18
+26 11895 77
+26 12851 56
+26 15039 34
+27 1305 44
+27 2137 96
+27 2671 92
+27 5831 61
+27 7139 59
+27 8167 28
+27 10757 15
+27 11441 15
+27 11509 65
+27 12237 89
+27 12749 31
+27 13885 66
+27 15025 26
+27 16029 59
+27 16419 65
+27 16767 60
+28 1807 98
+28 2817 8
+28 2967 29
+28 4483 78
+28 5437 15
+28 6411 3
+28 7965 93
+28 8043 58
+28 8407 14
+28 10295 13
+29 20 18
+29 1363 75
+29 2930 23
+29 3740 5
+29 7654 20
+29 9458 33
+29 10795 33
+29 12068 37
+29 12223 59
+29 13340 21
+29 13693 NULL
+29 15085 40
+29 15626 NULL
+29 15782 53
+30 217 91
+30 1951 59
+30 3238 16
+30 3506 15
+30 3928 87
+30 5431 77
+30 6752 69
+30 7870 7
+30 8666 21
+30 12572 33
+30 12670 20
+30 13579 75
+30 14848 62
+30 17348 62
+30 17875 78
+31 913 54
+31 4963 67
+31 6617 11
+31 6917 4
+31 7513 82
+31 11739 95
+31 14575 97
+31 14727 41
+31 15341 31
+31 15411 53
+31 16251 51
+32 1115 61
+32 2095 34
+32 2887 8
+32 4339 6
+32 4537 22
+32 4808 NULL
+32 5798 87
+32 7547 24
+32 9683 26
+32 11005 46
+32 11348 41
+32 12134 21
+32 15001 57
+32 15644 34
+32 16421 74
+32 17659 51
+33 4798 27
+33 7300 3
+33 9649 36
+33 10376 21
+33 11119 92
+33 11756 26
+33 12643 89
+33 12760 54
+33 12964 80
+33 14125 66
+33 14158 82
+33 14692 93
+33 15478 22
+34 1526 91
+34 1717 53
+34 2312 6
+34 4118 88
+34 5197 63
+34 5449 9
+34 6193 61
+34 9325 3
+34 9766 83
+34 12016 42
+34 12290 53
+34 12512 60
+34 13814 20
+34 16324 30
+35 411 51
+35 2377 52
+35 3667 97
+35 4325 56
+35 5179 83
+35 11635 87
+35 11661 81
+35 14239 55
+35 15619 45
+35 15757 9
+35 17341 92
+35 17365 65
+35 17451 7
+36 1115 80
+36 2095 43
+36 2887 31
+36 7547 46
+36 11005 49
+36 11349 80
+36 15001 54
+36 15645 23
+36 16421 25
+36 17561 16
+36 17659 91
+37 2997 94
+37 7283 87
+37 10715 52
+37 10929 88
+37 13171 6
+37 15337 62
+37 16971 12
+37 17125 NULL
+38 757 2
+38 2164 17
+38 3439 84
+38 4154 35
+38 5113 73
+38 6220 98
+38 7018 15
+38 7784 56
+38 8870 15
+38 9710 7
+38 10441 62
+38 15698 57
+39 386 89
+39 1598 64
+39 3476 73
+39 3943 64
+39 4190 86
+39 4957 24
+39 5393 98
+39 7097 78
+39 7118 67
+39 7604 49
+39 7697 24
+39 8078 54
+39 8411 96
+39 15491 54
+39 15625 17
+40 2854 71
+40 3490 65
+40 3985 63
+40 5098 35
+40 5318 87
+40 10094 80
+40 10912 23
+40 12050 NULL
+40 13658 53
+40 16976 3
+41 10 50
+41 64 29
+41 3380 88
+41 5566 11
+41 6310 90
+41 7402 69
+41 7603 94
+41 9322 8
+41 10915 81
+41 14788 15
+41 15242 87
+41 15328 46
+41 16514 20
+42 619 69
+42 976 100
+42 1436 94
+42 2314 74
+42 2392 14
+42 2602 30
+42 3346 74
+42 3613 30
+42 6058 30
+42 6134 92
+42 8462 23
+42 9740 52
+42 10016 57
+42 10471 19
+42 12550 41
+42 15002 41
+43 2923 16
+43 3344 22
+43 3911 26
+43 4364 77
+43 4691 41
+43 5773 85
+43 5852 16
+43 11771 30
+43 14669 97
+44 2351 56
+44 2623 18
+44 7303 14
+44 7527 67
+44 9059 68
+44 11707 83
+44 12341 20
+44 13331 98
+44 13449 45
+44 14149 80
+44 15803 81
+44 16491 56
+44 16837 92
+44 16909 61
+45 811 62
+45 1479 49
+45 3265 98
+45 5309 18
+45 7363 87
+45 10115 68
+45 11095 40
+45 13133 46
+45 16349 6
+46 1960 12
+46 3010 67
+46 7040 33
+46 8065 NULL
+46 11426 72
+46 13042 58
+46 15595 32
+46 16540 30
+46 17150 57
+46 17384 13
+47 254 NULL
+47 481 30
+47 1132 66
+47 1916 71
+47 3085 51
+47 3202 7
+47 3878 NULL
+47 4774 11
+47 5008 82
+47 5305 NULL
+47 5468 7
+47 7214 1
+47 9770 33
+47 13246 47
+47 13477 10
+48 1761 22
+48 2820 4
+48 2829 65
+48 4431 39
+48 5971 29
+48 6085 1
+48 6684 44
+48 9199 88
+48 11259 NULL
+48 12468 62
+48 13153 74
+48 17799 17
+49 749 60
+49 2135 4
+49 5342 69
+49 5852 47
+49 6805 40
+49 7141 94
+49 9049 68
+49 9553 71
+49 12737 48
+49 15155 84
+49 16361 4
+50 1280 69
+50 1312 30
+50 1909 53
+50 1984 40
+50 3097 64
+50 5023 NULL
+50 7135 69
+50 16081 82
+51 422 21
+51 3091 28
+51 4687 6
+51 5029 12
+51 5059 51
+51 6565 33
+51 8384 79
+51 9311 90
+51 10133 54
+51 11234 NULL
+51 12625 53
+51 13199 97
+51 17483 22
+51 17705 66
+52 2420 90
+52 3334 73
+52 6098 NULL
+52 7606 45
+52 11488 76
+52 15649 29
+52 16646 48
+52 17402 91
+52 17456 37
+53 1114 40
+53 2095 62
+53 2786 70
+53 2887 39
+53 7546 58
+53 11348 38
+53 13220 76
+53 13795 38
+53 15991 37
+53 16420 14
+53 16648 79
+53 17296 43
+53 17560 15
+54 702 40
+54 825 50
+54 1165 62
+54 3861 NULL
+54 6517 40
+54 9159 75
+54 14737 38
+54 16059 15
+54 16974 NULL
+54 17479 34
+55 1339 16
+55 3001 7
+55 5137 33
+55 9703 44
+55 12170 92
+55 12205 90
+55 14135 36
+55 14923 71
+55 17677 17
+56 4242 2
+56 4506 57
+56 8353 35
+56 8691 59
+56 8707 68
+56 10362 54
+56 16620 23
+56 17331 74
+57 3253 71
+57 4028 88
+57 4933 22
+57 12596 91
+57 12721 62
+57 12740 52
+57 15182 86
+57 17729 26
+57 17993 99
+58 1829 52
+58 3848 6
+58 5117 2
+58 7649 19
+58 9743 62
+58 10802 14
+58 15635 6
+58 16472 6
+58 16949 35
+59 3133 92
+59 3546 22
+59 5772 70
+59 7087 80
+59 8010 46
+59 8335 36
+59 9348 62
+59 9397 92
+59 10651 100
+59 11916 19
+59 12858 90
+59 14529 44
+60 97 50
+60 555 62
+60 633 71
+60 999 43
+60 1117 78
+60 1573 90
+60 4041 25
+60 4235 28
+60 4513 72
+60 4937 22
+60 7231 95
+60 10277 62
+60 10393 75
+60 13975 14
+60 16887 25
+60 17755 88
+61 1106 4
+61 2264 36
+61 3362 48
+61 4567 26
+61 5528 78
+61 6380 77
+61 7591 78
+61 8924 11
+61 10330 8
+61 16462 26
+62 4093 94
+62 6403 NULL
+62 8457 37
+62 10149 75
+62 12163 29
+62 12199 5
+62 12407 NULL
+62 13559 80
+62 15399 74
+62 15733 40
+62 16151 93
+63 4488 73
+63 5079 79
+63 5217 66
+63 5658 99
+63 9319 80
+63 11370 38
+63 11946 85
+63 13339 19
+63 15793 40
+63 16569 69
+64 1213 NULL
+64 3090 87
+64 3963 NULL
+64 11835 82
+64 13224 NULL
+64 14407 8
+64 15867 59
+64 15936 30
+64 16921 19
+64 17586 78
+64 17617 17
+65 2287 100
+65 4227 42
+65 9625 51
+65 9847 54
+65 13897 40
+65 14905 85
+65 15177 55
+65 17025 67
+66 6507 76
+66 7033 65
+66 7227 66
+66 8197 41
+66 9237 29
+66 10019 10
+66 11419 66
+66 15629 20
+66 16745 91
+66 16795 28
+67 757 77
+67 2133 74
+67 3439 73
+67 4155 87
+67 5113 NULL
+67 7020 79
+67 7507 77
+67 8469 59
+67 8871 71
+67 12087 70
+67 15699 44
+68 1387 74
+68 1603 57
+68 1820 54
+68 2035 22
+68 2296 52
+68 2564 83
+68 5162 23
+68 6763 77
+68 7765 NULL
+68 12526 3
+68 12724 88
+68 17426 2
+68 17600 13
+69 322 45
+69 337 34
+69 4208 9
+69 4267 10
+69 6136 7
+69 7264 67
+69 7822 30
+69 8599 53
+69 11137 68
+69 13489 66
+69 13792 NULL
+69 15448 16
+70 1592 53
+70 2462 NULL
+70 3296 48
+70 3947 NULL
+70 6185 82
+70 6425 NULL
+70 8893 17
+70 9857 20
+70 14549 4
+70 17815 95
+71 457 75
+71 1888 4
+71 2098 51
+71 4144 49
+71 5858 NULL
+71 6008 54
+71 7504 3
+71 8887 10
+71 9274 36
+71 9769 79
+71 9790 96
+71 9997 26
+71 10108 66
+71 10288 30
+71 11168 79
+71 17246 90
+72 1535 9
+72 5917 85
+72 6113 45
+72 6671 13
+72 9860 26
+72 10427 66
+72 10753 16
+72 11741 62
+72 12788 29
+72 12901 57
+72 13085 94
+72 13423 62
+72 13904 37
+72 15587 87
+72 16765 56
+73 247 53
+73 1063 37
+73 3205 82
+73 4946 54
+73 6862 58
+73 10051 49
+73 12502 75
+73 15109 38
+73 16519 97
+73 16585 38
+73 17269 40
+74 326 29
+74 3104 78
+74 3175 23
+74 3278 NULL
+74 3542 96
+74 3754 26
+74 5492 54
+74 7694 17
+74 8653 12
+74 9620 95
+74 10069 99
+74 13208 87
+74 16694 72
+75 607 20
+75 2948 25
+75 4625 73
+75 6938 89
+75 6953 71
+75 8726 6
+75 9905 54
+75 10217 85
+75 11039 70
+75 14186 63
+75 16796 93
+76 257 5
+76 465 2
+76 1107 16
+76 1503 97
+76 2265 98
+76 2869 32
+76 3363 25
+76 4237 48
+76 4567 40
+76 5529 78
+76 6381 50
+76 7591 27
+76 8925 6
+76 10331 3
+76 16463 53
+77 992 62
+77 1399 34
+77 2713 85
+77 3868 89
+77 6289 30
+77 7339 88
+77 7448 95
+77 7486 49
+77 8686 38
+77 9220 90
+77 11918 36
+77 12439 95
+77 13456 48
+77 14815 18
+77 16687 16
+78 901 3
+78 3304 50
+78 3856 27
+78 5965 78
+78 6044 59
+78 6110 43
+78 6500 76
+78 7576 87
+78 8611 79
+78 10507 6
+78 11209 7
+78 12706 19
+78 14996 39
+79 247 NULL
+79 1063 85
+79 3205 48
+79 4947 35
+79 6864 1
+79 10051 10
+79 10524 36
+79 12504 81
+79 14322 41
+79 15109 NULL
+79 15498 3
+79 15888 58
+79 16519 9
+79 16585 93
+79 17269 81
+80 998 93
+80 1519 25
+80 1573 40
+80 4040 66
+80 4513 NULL
+80 4622 1
+80 7231 49
+80 7610 37
+80 10393 5
+80 12968 NULL
+80 13717 91
+80 13975 13
+80 16363 84
+80 16886 77
+80 17308 29
+80 17755 94
+81 4486 31
+81 5078 75
+81 5216 64
+81 5656 24
+81 7166 7
+81 7663 79
+81 8918 37
+81 9319 36
+81 11107 36
+81 11368 26
+81 13339 6
+81 15793 8
+82 2572 53
+82 7862 75
+82 13138 59
+82 14998 49
+82 17041 18
http://git-wip-us.apache.org/repos/asf/hive/blob/7273a4c4/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out b/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
index 1d2f8dd..afaf2d6 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
@@ -320,10 +320,10 @@ NULL
-47
-46
PREHOOK: query: explain
-select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint limit 20
+select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint order by ctinyint limit 20
PREHOOK: type: QUERY
POSTHOOK: query: explain
-select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint limit 20
+select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint order by ctinyint limit 20
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
@@ -387,11 +387,11 @@ STAGE PLANS:
Processor Tree:
ListSink
-PREHOOK: query: select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint limit 20
+PREHOOK: query: select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint order by ctinyint limit 20
PREHOOK: type: QUERY
PREHOOK: Input: default@alltypesorc
#### A masked pattern was here ####
-POSTHOOK: query: select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint limit 20
+POSTHOOK: query: select ctinyint, count(distinct(cdouble)) from alltypesorc group by ctinyint order by ctinyint limit 20
POSTHOOK: type: QUERY
POSTHOOK: Input: default@alltypesorc
#### A masked pattern was here ####
[06/51] [abbrv] hive git commit: HIVE-11853: LLAP: Fix tests related
to stats difference (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-11853: LLAP: Fix tests related to stats difference (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/c453c0ed
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/c453c0ed
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/c453c0ed
Branch: refs/heads/master
Commit: c453c0ed807625c94aee54c0465bb962346b5e31
Parents: 4fa4e40
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Fri Sep 18 17:22:14 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Fri Sep 18 17:22:14 2015 -0500
----------------------------------------------------------------------
.../clientpositive/vector_leftsemi_mapjoin.q | 4 +-
.../clientpositive/vector_multi_insert.q | 2 +
.../llap/vector_leftsemi_mapjoin.q.out | 13973 +++++++++++++++++
.../llap/vector_multi_insert.q.out | 28 +-
.../tez/vector_leftsemi_mapjoin.q.out | 8 +
.../tez/vector_multi_insert.q.out | 8 +
6 files changed, 14012 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c453c0ed/ql/src/test/queries/clientpositive/vector_leftsemi_mapjoin.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/vector_leftsemi_mapjoin.q b/ql/src/test/queries/clientpositive/vector_leftsemi_mapjoin.q
index 680d2c9..dfb8405 100644
--- a/ql/src/test/queries/clientpositive/vector_leftsemi_mapjoin.q
+++ b/ql/src/test/queries/clientpositive/vector_leftsemi_mapjoin.q
@@ -17,6 +17,8 @@ select * from t2 sort by key;
create table t3 stored as orc as select * from (select * from t1 union all select * from t2) b;
select * from t3 sort by key, value;
+analyze table t3 compute statistics;
+
create table t4 (key int, value string) stored as orc;
select * from t4;
@@ -401,4 +403,4 @@ explain select a.key from t3 a left semi join t2 b on a.key = b.key left outer j
select a.key from t3 a left semi join t2 b on a.key = b.key left outer join t1 c on a.value = c.value sort by a.key;
explain select a.key from t3 a left semi join t2 b on a.value = b.value where a.key > 100;
-select a.key from t3 a left semi join t2 b on a.value = b.value where a.key > 100;
\ No newline at end of file
+select a.key from t3 a left semi join t2 b on a.value = b.value where a.key > 100;
http://git-wip-us.apache.org/repos/asf/hive/blob/c453c0ed/ql/src/test/queries/clientpositive/vector_multi_insert.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/vector_multi_insert.q b/ql/src/test/queries/clientpositive/vector_multi_insert.q
index 8c92cda..374a0da 100644
--- a/ql/src/test/queries/clientpositive/vector_multi_insert.q
+++ b/ql/src/test/queries/clientpositive/vector_multi_insert.q
@@ -20,6 +20,8 @@ create table orc_rn1 (rn int);
create table orc_rn2 (rn int);
create table orc_rn3 (rn int);
+analyze table orc1 compute statistics;
+
explain from orc1 a
insert overwrite table orc_rn1 select a.* where a.rn < 100
insert overwrite table orc_rn2 select a.* where a.rn >= 100 and a.rn < 1000
[42/51] [abbrv] hive git commit: HIVE-12066 : Add javadoc for methods
added to public APIs (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12066 : Add javadoc for methods added to public APIs (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/cf76e6b5
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/cf76e6b5
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/cf76e6b5
Branch: refs/heads/master
Commit: cf76e6b5d89dfa2d44b96c852a10ccd0252a4fe8
Parents: ca99b0c
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 8 12:52:04 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 8 12:52:04 2015 -0700
----------------------------------------------------------------------
.../apache/hadoop/hive/ql/io/orc/Reader.java | 27 ++++++++++++++++++--
1 file changed, 25 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/cf76e6b5/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
index 251e1f8..cf81782 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/Reader.java
@@ -22,7 +22,6 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.List;
-import org.apache.hadoop.hive.ql.io.orc.OrcProto.Footer;
import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
@@ -351,20 +350,44 @@ public interface Reader {
boolean[] include, SearchArgument sarg,
String[] neededColumns) throws IOException;
+ /**
+ * @return Metadata reader used to read file metadata.
+ */
MetadataReader metadata() throws IOException;
+ /**
+ * @return List of integers representing version of the file, in order from major to minor.
+ */
List<Integer> getVersionList();
+ /**
+ * @return Gets the size of metadata, in bytes.
+ */
int getMetadataSize();
+ /**
+ * @return Stripe statistics, in original protobuf form.
+ */
List<OrcProto.StripeStatistics> getOrcProtoStripeStatistics();
+ /**
+ * @return Stripe statistics.
+ */
List<StripeStatistics> getStripeStatistics();
+ /**
+ * @return File statistics, in original protobuf form.
+ */
List<OrcProto.ColumnStatistics> getOrcProtoFileStatistics();
+ /**
+ * @param useZeroCopy Whether zero-copy read should be used.
+ * @return The default data reader that ORC is using to read bytes from disk.
+ */
DataReader createDefaultDataReader(boolean useZeroCopy);
- /** Gets serialized file metadata read from disk for the purposes of caching, etc. */
+ /**
+ * @return Serialized file metadata read from disk for the purposes of caching, etc.
+ */
ByteBuffer getSerializedFileFooter();
}
[24/51] [abbrv] hive git commit: HIVE-11923: allow qtests to run via
a single client session for tez and llap (Siddharth Seth reviewed by Prasanth
Jayachandran)
Posted by se...@apache.org.
HIVE-11923: allow qtests to run via a single client session for tez and llap (Siddharth Seth reviewed by Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/7a3e7457
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/7a3e7457
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/7a3e7457
Branch: refs/heads/master
Commit: 7a3e74579f33ef96abcdf7f45c507c6d47b912c7
Parents: bf8acfb
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Mon Sep 28 19:40:05 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Mon Sep 28 19:40:05 2015 -0500
----------------------------------------------------------------------
data/conf/hive-log4j2.xml | 27 ++++++++++++++++++++
data/conf/llap/tez-site.xml | 6 +++++
data/conf/tez/tez-site.xml | 6 +++++
.../org/apache/hadoop/hive/ql/QTestUtil.java | 16 ++++++++++++
4 files changed, 55 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/7a3e7457/data/conf/hive-log4j2.xml
----------------------------------------------------------------------
diff --git a/data/conf/hive-log4j2.xml b/data/conf/hive-log4j2.xml
index 48a584f..860d756 100644
--- a/data/conf/hive-log4j2.xml
+++ b/data/conf/hive-log4j2.xml
@@ -72,6 +72,33 @@
<AppenderRef ref="EventCounter" />
</Root>
+ <Logger name="org.apache.hadoop.ipc" level="WARN">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.hadoop.security" level="INFO">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.hadoop.hdfs" level="INFO">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.hadoop.hdfs.server" level="WARN">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.hadoop.metrics2" level="INFO">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.mortbay" level="INFO">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.hadoop.yarn" level="INFO">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.hadoop.yarn.server" level="WARN">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
+ <Logger name="org.apache.tez" level="INFO">
+ <AppenderRef ref="${sys:hive.root.logger}"/>
+ </Logger>
<Logger name="org.apache.hadoop.conf.Configuration" level="ERROR">
<AppenderRef ref="${sys:hive.root.logger}"/>
</Logger>
http://git-wip-us.apache.org/repos/asf/hive/blob/7a3e7457/data/conf/llap/tez-site.xml
----------------------------------------------------------------------
diff --git a/data/conf/llap/tez-site.xml b/data/conf/llap/tez-site.xml
new file mode 100644
index 0000000..940f390
--- /dev/null
+++ b/data/conf/llap/tez-site.xml
@@ -0,0 +1,6 @@
+<configuration>
+ <property>
+ <name>tez.am.dag.scheduler.class</name>
+ <value>org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/hive/blob/7a3e7457/data/conf/tez/tez-site.xml
----------------------------------------------------------------------
diff --git a/data/conf/tez/tez-site.xml b/data/conf/tez/tez-site.xml
new file mode 100644
index 0000000..940f390
--- /dev/null
+++ b/data/conf/tez/tez-site.xml
@@ -0,0 +1,6 @@
+<configuration>
+ <property>
+ <name>tez.am.dag.scheduler.class</name>
+ <value>org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrderControlled</value>
+ </property>
+</configuration>
http://git-wip-us.apache.org/repos/asf/hive/blob/7a3e7457/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
----------------------------------------------------------------------
diff --git a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
index 4cab161..477a08d 100644
--- a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
+++ b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
@@ -81,6 +81,7 @@ import org.apache.hadoop.hive.ql.exec.Task;
import org.apache.hadoop.hive.ql.exec.Utilities;
import org.apache.hadoop.hive.ql.exec.spark.session.SparkSession;
import org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl;
+import org.apache.hadoop.hive.ql.exec.tez.TezSessionState;
import org.apache.hadoop.hive.ql.lockmgr.zookeeper.CuratorFrameworkSingleton;
import org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager;
import org.apache.hadoop.hive.ql.metadata.Hive;
@@ -158,6 +159,7 @@ public class QTestUtil {
private final boolean miniMr = false;
private String hadoopVer = null;
private QTestSetup setup = null;
+ private TezSessionState tezSessionState = null;
private SparkSession sparkSession = null;
private boolean isSessionStateStarted = false;
private static final String javaVersion = getJavaVersion();
@@ -428,10 +430,16 @@ public class QTestUtil {
String uriString = WindowsPathUtil.getHdfsUriString(fs.getUri().toString());
if (clusterType == MiniClusterType.tez) {
+ if (confDir != null && !confDir.isEmpty()) {
+ conf.addResource(new URL("file://" + new File(confDir).toURI().getPath()
+ + "/tez-site.xml"));
+ }
mr = shims.getMiniTezCluster(conf, 4, uriString, false);
} else if (clusterType == MiniClusterType.llap) {
if (confDir != null && !confDir.isEmpty()) {
conf.addResource(new URL("file://" + new File(confDir).toURI().getPath()
+ + "/tez-site.xml"));
+ conf.addResource(new URL("file://" + new File(confDir).toURI().getPath()
+ "/llap-daemon-site.xml"));
}
mr = shims.getMiniTezCluster(conf, 2, uriString, true);
@@ -952,6 +960,10 @@ public class QTestUtil {
sparkSession = oldSs.getSparkSession();
ss.setSparkSession(sparkSession);
oldSs.setSparkSession(null);
+ // Copy the tezSessionState from the old CliSessionState.
+ tezSessionState = oldSs.getTezSession();
+ ss.setTezSession(tezSessionState);
+ oldSs.setTezSession(null);
oldSs.close();
}
@@ -1016,6 +1028,10 @@ public class QTestUtil {
sparkSession = oldSs.getSparkSession();
ss.setSparkSession(sparkSession);
oldSs.setSparkSession(null);
+ // Copy the tezSessionState from the old CliSessionState.
+ tezSessionState = oldSs.getTezSession();
+ ss.setTezSession(tezSessionState);
+ oldSs.setTezSession(null);
oldSs.close();
}
if (oldSs != null && oldSs.out != null && oldSs.out != System.out) {
[15/51] [abbrv] hive git commit: HIVE-11904: LLAP: Typo in webapps
causing initialization error for MiniLlap tests (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-11904: LLAP: Typo in webapps causing initialization error for MiniLlap tests (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b6ad7a2f
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b6ad7a2f
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b6ad7a2f
Branch: refs/heads/master
Commit: b6ad7a2f2b251e36b511c6f7013b444de307ee61
Parents: 73eb923
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Mon Sep 21 13:46:05 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Mon Sep 21 13:46:05 2015 -0500
----------------------------------------------------------------------
llap-server/src/test/resources/webapp/llap/.keep | 0
llap-server/src/test/resources/webapps/llap/.keep | 0
2 files changed, 0 insertions(+), 0 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b6ad7a2f/llap-server/src/test/resources/webapp/llap/.keep
----------------------------------------------------------------------
diff --git a/llap-server/src/test/resources/webapp/llap/.keep b/llap-server/src/test/resources/webapp/llap/.keep
deleted file mode 100644
index e69de29..0000000
http://git-wip-us.apache.org/repos/asf/hive/blob/b6ad7a2f/llap-server/src/test/resources/webapps/llap/.keep
----------------------------------------------------------------------
diff --git a/llap-server/src/test/resources/webapps/llap/.keep b/llap-server/src/test/resources/webapps/llap/.keep
new file mode 100644
index 0000000..e69de29
[50/51] [abbrv] hive git commit: HIVE-12096 : LLAP: merge master into
branch (Sergey Shelukhin) ADDENDUM merge
Posted by se...@apache.org.
HIVE-12096 : LLAP: merge master into branch (Sergey Shelukhin) ADDENDUM merge
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/eb28deb6
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/eb28deb6
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/eb28deb6
Branch: refs/heads/master
Commit: eb28deb61d927b43bd82d06aee2802b0571c6832
Parents: b6de889 9b4826e
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 12 16:53:35 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 12 16:53:35 2015 -0700
----------------------------------------------------------------------
.../hive/ql/stats/DummyStatsAggregator.java | 12 +-
.../hive/ql/stats/DummyStatsPublisher.java | 15 +-
.../ql/stats/KeyVerifyingStatsAggregator.java | 10 +-
pom.xml | 1 -
.../hadoop/hive/ql/exec/FileSinkOperator.java | 8 +-
.../apache/hadoop/hive/ql/exec/StatsTask.java | 31 ++-
.../hadoop/hive/ql/exec/TableScanOperator.java | 7 +-
.../apache/hadoop/hive/ql/exec/Utilities.java | 31 ++-
.../hadoop/hive/ql/exec/mr/ExecDriver.java | 14 +-
.../hive/ql/exec/spark/SparkPlanGenerator.java | 16 +-
.../hadoop/hive/ql/exec/tez/DagUtils.java | 5 +-
.../hive/ql/index/AggregateIndexHandler.java | 1 -
.../hive/ql/index/TableBasedIndexHandler.java | 7 -
.../ql/index/bitmap/BitmapIndexHandler.java | 1 -
.../ql/index/compact/CompactIndexHandler.java | 1 -
.../ql/io/rcfile/stats/PartialScanMapper.java | 7 +-
.../ql/io/rcfile/stats/PartialScanTask.java | 11 +-
.../ql/io/rcfile/stats/PartialScanWork.java | 14 ++
.../hive/ql/optimizer/GenMRTableScan1.java | 3 +
.../hive/ql/optimizer/GenMapRedUtils.java | 2 +-
.../hive/ql/parse/ProcessAnalyzeTable.java | 4 +-
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 8 +-
.../parse/spark/SparkProcessAnalyzeTable.java | 2 +
.../hadoop/hive/ql/plan/FileSinkDesc.java | 16 +-
.../apache/hadoop/hive/ql/plan/StatsWork.java | 15 +-
.../hadoop/hive/ql/plan/TableScanDesc.java | 12 +-
.../hive/ql/stats/CounterStatsAggregator.java | 8 +-
.../ql/stats/CounterStatsAggregatorSpark.java | 6 +-
.../ql/stats/CounterStatsAggregatorTez.java | 10 +-
.../hive/ql/stats/CounterStatsPublisher.java | 7 +-
.../hadoop/hive/ql/stats/StatsAggregator.java | 7 +-
.../hive/ql/stats/StatsCollectionContext.java | 63 ++++++
.../hadoop/hive/ql/stats/StatsPublisher.java | 8 +-
.../hive/ql/stats/fs/FSStatsAggregator.java | 23 +-
.../hive/ql/stats/fs/FSStatsPublisher.java | 32 +--
.../hive/ql/stats/jdbc/JDBCStatsAggregator.java | 18 +-
.../hive/ql/stats/jdbc/JDBCStatsPublisher.java | 22 +-
.../hive/ql/exec/TestFileSinkOperator.java | 13 +-
.../ql/exec/TestStatsPublisherEnhanced.java | 61 +++---
.../infer_bucket_sort_multi_insert.q | 1 +
.../test/queries/clientpositive/multi_insert.q | 2 +-
.../queries/clientpositive/multi_insert_gby2.q | 2 +-
.../queries/clientpositive/multi_insert_gby3.q | 2 +-
.../clientpositive/multi_insert_lateral_view.q | 1 +
.../queries/clientpositive/multi_insert_mixed.q | 2 +-
...multi_insert_move_tasks_share_dependencies.q | 2 +-
.../clientpositive/multi_insert_union_src.q | 2 +-
.../spark/column_access_stats.q.out | 46 ++--
.../test/results/clientpositive/spark/pcr.q.out | 16 +-
.../clientpositive/spark/ppd_join5.q.out | 58 ++---
.../clientpositive/spark/smb_mapjoin_12.q.out | 6 +-
.../clientpositive/spark/smb_mapjoin_13.q.out | 36 ++--
.../clientpositive/spark/smb_mapjoin_15.q.out | 12 +-
.../clientpositive/spark/smb_mapjoin_16.q.out | 2 +-
.../results/clientpositive/spark/union34.q.out | 68 +++---
service/pom.xml | 6 -
.../auth/TestLdapAtnProviderWithLdapServer.java | 215 -------------------
.../org/apache/hive/service/auth/ldapdata.ldif | 59 -----
58 files changed, 486 insertions(+), 584 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/eb28deb6/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
----------------------------------------------------------------------
[41/51] [abbrv] hive git commit: HIVE-12042 : LLAP: update some out
files (Sergey Shelukhin) ADDENDUM2
Posted by se...@apache.org.
HIVE-12042 : LLAP: update some out files (Sergey Shelukhin) ADDENDUM2
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/ca99b0c6
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/ca99b0c6
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/ca99b0c6
Branch: refs/heads/master
Commit: ca99b0c619fcdb5ad9a8f41bfe8e0a2dcf73ef25
Parents: be00a03
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 8 12:51:10 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 8 12:51:10 2015 -0700
----------------------------------------------------------------------
.../llap/dynamic_partition_pruning.q.out | 4 +-
.../llap/dynamic_partition_pruning_2.q.out | 54 ++++++++++----------
.../vectorized_dynamic_partition_pruning.q.out | 4 +-
3 files changed, 31 insertions(+), 31 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/ca99b0c6/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
index 705e31d..3ebd690 100644
--- a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning.q.out
@@ -1965,7 +1965,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
#### A masked pattern was here ####
1000
-Warning: Shuffle Join MERGEJOIN[13][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[14][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: -- non-equi join
EXPLAIN select count(*) from srcpart, srcpart_date_hour where (srcpart_date_hour.`date` = '2008-04-08' and srcpart_date_hour.hour = 11) and (srcpart.ds = srcpart_date_hour.ds or srcpart.hr = srcpart_date_hour.hr)
PREHOOK: type: QUERY
@@ -2055,7 +2055,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[13][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[14][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: select count(*) from srcpart, srcpart_date_hour where (srcpart_date_hour.`date` = '2008-04-08' and srcpart_date_hour.hour = 11) and (srcpart.ds = srcpart_date_hour.ds or srcpart.hr = srcpart_date_hour.hr)
PREHOOK: type: QUERY
PREHOOK: Input: default@srcpart
http://git-wip-us.apache.org/repos/asf/hive/blob/ca99b0c6/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
index 22e0f29..cec4219 100644
--- a/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/dynamic_partition_pruning_2.q.out
@@ -178,23 +178,23 @@ STAGE PLANS:
Statistics: Num rows: 9 Data size: 29 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
Filter Operator
- predicate: ((_col1 = _col5) and (_col6) IN ('foo', 'bar')) (type: boolean)
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ predicate: (_col1 = _col5) (type: boolean)
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col6 (type: string), _col0 (type: decimal(10,0))
outputColumnNames: _col6, _col0
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(), sum(_col0)
keys: _col6 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: decimal(20,0))
Execution mode: llap
Map 4
@@ -236,11 +236,11 @@ STAGE PLANS:
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: decimal(20,0))
Reducer 3
Execution mode: uber
@@ -248,10 +248,10 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: bigint), VALUE._col1 (type: decimal(20,0))
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -344,23 +344,23 @@ STAGE PLANS:
Statistics: Num rows: 9 Data size: 29 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
Filter Operator
- predicate: ((_col1 = _col5) and (_col6) IN ('foo', 'bar')) (type: boolean)
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ predicate: (_col1 = _col5) (type: boolean)
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col6 (type: string), _col0 (type: decimal(10,0))
outputColumnNames: _col6, _col0
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(), sum(_col0)
keys: _col6 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: decimal(20,0))
Execution mode: llap
Map 4
@@ -387,11 +387,11 @@ STAGE PLANS:
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: decimal(20,0))
Reducer 3
Execution mode: uber
@@ -399,10 +399,10 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: bigint), VALUE._col1 (type: decimal(20,0))
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -695,23 +695,23 @@ STAGE PLANS:
Statistics: Num rows: 9 Data size: 29 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
Filter Operator
- predicate: ((_col1 = _col5) and (_col6) IN ('foo', 'bar')) (type: boolean)
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ predicate: (_col1 = _col5) (type: boolean)
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col6 (type: string), _col0 (type: decimal(10,0))
outputColumnNames: _col6, _col0
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(), sum(_col0)
keys: _col6 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 12 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: decimal(20,0))
Execution mode: llap
Map 4
@@ -753,11 +753,11 @@ STAGE PLANS:
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: decimal(20,0))
Reducer 3
Execution mode: uber
@@ -765,10 +765,10 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: bigint), VALUE._col1 (type: decimal(20,0))
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
http://git-wip-us.apache.org/repos/asf/hive/blob/ca99b0c6/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out b/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
index 44f4568..3e82696 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
@@ -1965,7 +1965,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
#### A masked pattern was here ####
1000
-Warning: Shuffle Join MERGEJOIN[13][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[14][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: -- non-equi join
EXPLAIN select count(*) from srcpart, srcpart_date_hour where (srcpart_date_hour.`date` = '2008-04-08' and srcpart_date_hour.hour = 11) and (srcpart.ds = srcpart_date_hour.ds or srcpart.hr = srcpart_date_hour.hr)
PREHOOK: type: QUERY
@@ -2055,7 +2055,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[13][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[14][tables = [srcpart, srcpart_date_hour]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: select count(*) from srcpart, srcpart_date_hour where (srcpart_date_hour.`date` = '2008-04-08' and srcpart_date_hour.hour = 11) and (srcpart.ds = srcpart_date_hour.ds or srcpart.hr = srcpart_date_hour.hr)
PREHOOK: type: QUERY
PREHOOK: Input: default@srcpart
[35/51] [abbrv] hive git commit: HIVE-12047 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/b87f63ca/ql/src/test/results/clientpositive/orc_llap.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/orc_llap.q.out
index b93771f,0000000..79c6281
mode 100644,000000..100644
--- a/ql/src/test/results/clientpositive/orc_llap.q.out
+++ b/ql/src/test/results/clientpositive/orc_llap.q.out
@@@ -1,1095 -1,0 +1,1095 @@@
+PREHOOK: query: DROP TABLE cross_numbers
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: DROP TABLE cross_numbers
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: DROP TABLE orc_llap
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: DROP TABLE orc_llap
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: DROP TABLE orc_llap_small
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: DROP TABLE orc_llap_small
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: CREATE TABLE orc_llap(
+ ctinyint TINYINT,
+ csmallint SMALLINT,
+ cint INT,
+ cbigint BIGINT,
+ cfloat FLOAT,
+ cdouble DOUBLE,
+ cstring1 STRING,
+ cstring2 STRING,
+ ctimestamp1 TIMESTAMP,
+ ctimestamp2 TIMESTAMP,
+ cboolean1 BOOLEAN,
+ cboolean2 BOOLEAN)
+ STORED AS ORC
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@orc_llap
+POSTHOOK: query: CREATE TABLE orc_llap(
+ ctinyint TINYINT,
+ csmallint SMALLINT,
+ cint INT,
+ cbigint BIGINT,
+ cfloat FLOAT,
+ cdouble DOUBLE,
+ cstring1 STRING,
+ cstring2 STRING,
+ ctimestamp1 TIMESTAMP,
+ ctimestamp2 TIMESTAMP,
+ cboolean1 BOOLEAN,
+ cboolean2 BOOLEAN)
+ STORED AS ORC
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@orc_llap
+PREHOOK: query: CREATE TABLE orc_llap_small(
+ ctinyint TINYINT,
+ csmallint SMALLINT,
+ cint INT)
+ STORED AS ORC
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@orc_llap_small
+POSTHOOK: query: CREATE TABLE orc_llap_small(
+ ctinyint TINYINT,
+ csmallint SMALLINT,
+ cint INT)
+ STORED AS ORC
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@orc_llap_small
+PREHOOK: query: create table cross_numbers(i int)
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@cross_numbers
+POSTHOOK: query: create table cross_numbers(i int)
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@cross_numbers
+PREHOOK: query: insert into table cross_numbers
+select distinct csmallint
+from alltypesorc where csmallint > 0 order by csmallint limit 10
+PREHOOK: type: QUERY
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: default@cross_numbers
+POSTHOOK: query: insert into table cross_numbers
+select distinct csmallint
+from alltypesorc where csmallint > 0 order by csmallint limit 10
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: default@cross_numbers
+POSTHOOK: Lineage: cross_numbers.i EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:csmallint, type:smallint, comment:null), ]
+Warning: Shuffle Join JOIN[7][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
+PREHOOK: query: insert into table orc_llap
+select ctinyint + i, csmallint + i, cint + i, cbigint + i, cfloat + i, cdouble + i, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2
+from alltypesorc cross join cross_numbers
+PREHOOK: type: QUERY
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Input: default@cross_numbers
+PREHOOK: Output: default@orc_llap
+POSTHOOK: query: insert into table orc_llap
+select ctinyint + i, csmallint + i, cint + i, cbigint + i, cfloat + i, cdouble + i, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2
+from alltypesorc cross join cross_numbers
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Input: default@cross_numbers
+POSTHOOK: Output: default@orc_llap
+POSTHOOK: Lineage: orc_llap.cbigint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cbigint, type:bigint, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cboolean1 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cboolean1, type:boolean, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cboolean2 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cboolean2, type:boolean, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cdouble EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cdouble, type:double, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cfloat EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cfloat, type:float, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cint, type:int, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.csmallint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:csmallint, type:smallint, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cstring1 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cstring1, type:string, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cstring2 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cstring2, type:string, comment:null), ]
+POSTHOOK: Lineage: orc_llap.ctimestamp1 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:ctimestamp1, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: orc_llap.ctimestamp2 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:ctimestamp2, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: orc_llap.ctinyint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:ctinyint, type:tinyint, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+PREHOOK: query: insert into table orc_llap_small
+select ctinyint, csmallint, cint from alltypesorc where ctinyint is not null and cint is not null limit 15
+PREHOOK: type: QUERY
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Output: default@orc_llap_small
+POSTHOOK: query: insert into table orc_llap_small
+select ctinyint, csmallint, cint from alltypesorc where ctinyint is not null and cint is not null limit 15
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Output: default@orc_llap_small
+POSTHOOK: Lineage: orc_llap_small.cint SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cint, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap_small.csmallint SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:csmallint, type:smallint, comment:null), ]
+POSTHOOK: Lineage: orc_llap_small.ctinyint SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:ctinyint, type:tinyint, comment:null), ]
+Warning: Map Join MAPJOIN[18][bigTable=?] in task 'Stage-2:MAPRED' is a cross product
+PREHOOK: query: -- Cross join with no projection - do it on small table
+explain
+select count(1) from orc_llap_small y join orc_llap_small x
+PREHOOK: type: QUERY
+POSTHOOK: query: -- Cross join with no projection - do it on small table
+explain
+select count(1) from orc_llap_small y join orc_llap_small x
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-5 is a root stage
+ Stage-2 depends on stages: Stage-5
+ Stage-0 depends on stages: Stage-2
+
+STAGE PLANS:
+ Stage: Stage-5
+ Map Reduce Local Work
+ Alias -> Map Local Tables:
+ $hdt$_0:$hdt$_0:y
+ Fetch Operator
+ limit: -1
+ Alias -> Map Local Operator Tree:
+ $hdt$_0:$hdt$_0:y
+ TableScan
+ alias: y
+ Statistics: Num rows: 15 Data size: 180 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 15 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
+ HashTable Sink Operator
+ keys:
+ 0
+ 1
+
+ Stage: Stage-2
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: y
+ Statistics: Num rows: 15 Data size: 180 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 15 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0
+ 1
+ Statistics: Num rows: 112 Data size: 0 Basic stats: PARTIAL Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(1)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: bigint)
+ Execution mode: vectorized
+ LLAP IO: all inputs
+ Local Work:
+ Map Reduce Local Work
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+Warning: Map Join MAPJOIN[18][bigTable=?] in task 'Stage-2:MAPRED' is a cross product
+PREHOOK: query: select count(1) from orc_llap_small y join orc_llap_small x
+PREHOOK: type: QUERY
+PREHOOK: Input: default@orc_llap_small
+#### A masked pattern was here ####
+POSTHOOK: query: select count(1) from orc_llap_small y join orc_llap_small x
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@orc_llap_small
+#### A masked pattern was here ####
+225
+PREHOOK: query: -- All row groups selected, no projection
+select count(*) from orc_llap_small
+PREHOOK: type: QUERY
+PREHOOK: Input: default@orc_llap_small
+#### A masked pattern was here ####
+POSTHOOK: query: -- All row groups selected, no projection
+select count(*) from orc_llap_small
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@orc_llap_small
+#### A masked pattern was here ####
+15
+PREHOOK: query: -- All row groups pruned
+select count(*) from orc_llap_small where cint < 60000000
+PREHOOK: type: QUERY
+PREHOOK: Input: default@orc_llap_small
+#### A masked pattern was here ####
+POSTHOOK: query: -- All row groups pruned
+select count(*) from orc_llap_small where cint < 60000000
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@orc_llap_small
+#### A masked pattern was here ####
+0
+PREHOOK: query: -- Hash cannot be vectorized, so run hash as the last step on a temp table
+drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+POSTHOOK: query: -- Hash cannot be vectorized, so run hash as the last step on a temp table
+drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+PREHOOK: query: explain
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ filterExpr: ((cint > 10) and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((cint > 10) and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 20480 Data size: 4846656 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cint (type: int), csmallint (type: smallint), cbigint (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 20480 Data size: 4846656 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 20480 Data size: 4846656 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-558222259686
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select * from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select * from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ filterExpr: ((cint > 10) and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((cint > 10) and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 20480 Data size: 4846656 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cbigint (type: bigint), cfloat (type: float), cdouble (type: double), cstring1 (type: string), cstring2 (type: string), ctimestamp1 (type: timestamp), ctimestamp2 (type: timestamp), cboolean1 (type: boolean), cboolean2 (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11
+ Statistics: Num rows: 20480 Data size: 4846656 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 20480 Data size: 4846656 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select * from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select * from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-197609091139
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select cstring2 from orc_llap where cint > 5 and cint < 10
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select cstring2 from orc_llap where cint > 5 and cint < 10
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ filterExpr: ((cint > 5) and (cint < 10)) (type: boolean)
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((cint > 5) and (cint < 10)) (type: boolean)
+ Statistics: Num rows: 13653 Data size: 3231025 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cstring2 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 13653 Data size: 3231025 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 13653 Data size: 3231025 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select cstring2 from orc_llap where cint > 5 and cint < 10
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select cstring2 from orc_llap where cint > 5 and cint < 10
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+NULL
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cstring1 (type: string), cstring2 (type: string)
+ outputColumnNames: cstring1, cstring2
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ keys: cstring1 (type: string), cstring2 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: bigint)
+ Execution mode: vectorized
+ LLAP IO: all inputs
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 61440 Data size: 14539970 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 61440 Data size: 14539970 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-201218541193
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-4 is a root stage
+ Stage-3 depends on stages: Stage-4
+ Stage-0 depends on stages: Stage-3
+
+STAGE PLANS:
+ Stage: Stage-4
+ Map Reduce Local Work
+ Alias -> Map Local Tables:
+ o1
+ Fetch Operator
+ limit: -1
+ Alias -> Map Local Operator Tree:
+ o1
+ TableScan
+ alias: o1
+ filterExpr: (csmallint is not null and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (csmallint is not null and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 30720 Data size: 7269985 Basic stats: COMPLETE Column stats: NONE
+ HashTable Sink Operator
+ keys:
+ 0 csmallint (type: smallint)
+ 1 csmallint (type: smallint)
+
+ Stage: Stage-3
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: o2
+ filterExpr: (csmallint is not null and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (csmallint is not null and cbigint is not null) (type: boolean)
+ Statistics: Num rows: 30720 Data size: 7269985 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 csmallint (type: smallint)
+ 1 csmallint (type: smallint)
+ outputColumnNames: _col6, _col22
+ Statistics: Num rows: 33792 Data size: 7996983 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col6 (type: string), _col22 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 33792 Data size: 7996983 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 33792 Data size: 7996983 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-735462183586256
+Warning: Map Join MAPJOIN[12][bigTable=?] in task 'Stage-4:MAPRED' is a cross product
+PREHOOK: query: -- multi-stripe test
+insert into table orc_llap
+select ctinyint + i, csmallint + i, cint + i, cbigint + i, cfloat + i, cdouble + i, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2
+from alltypesorc cross join cross_numbers
+PREHOOK: type: QUERY
+PREHOOK: Input: default@alltypesorc
+PREHOOK: Input: default@cross_numbers
+PREHOOK: Output: default@orc_llap
+POSTHOOK: query: -- multi-stripe test
+insert into table orc_llap
+select ctinyint + i, csmallint + i, cint + i, cbigint + i, cfloat + i, cdouble + i, cstring1, cstring2, ctimestamp1, ctimestamp2, cboolean1, cboolean2
+from alltypesorc cross join cross_numbers
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@alltypesorc
+POSTHOOK: Input: default@cross_numbers
+POSTHOOK: Output: default@orc_llap
+POSTHOOK: Lineage: orc_llap.cbigint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cbigint, type:bigint, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cboolean1 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cboolean1, type:boolean, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cboolean2 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cboolean2, type:boolean, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cdouble EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cdouble, type:double, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cfloat EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cfloat, type:float, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:cint, type:int, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.csmallint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:csmallint, type:smallint, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cstring1 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cstring1, type:string, comment:null), ]
+POSTHOOK: Lineage: orc_llap.cstring2 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:cstring2, type:string, comment:null), ]
+POSTHOOK: Lineage: orc_llap.ctimestamp1 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:ctimestamp1, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: orc_llap.ctimestamp2 SIMPLE [(alltypesorc)alltypesorc.FieldSchema(name:ctimestamp2, type:timestamp, comment:null), ]
+POSTHOOK: Lineage: orc_llap.ctinyint EXPRESSION [(alltypesorc)alltypesorc.FieldSchema(name:ctinyint, type:tinyint, comment:null), (cross_numbers)cross_numbers.FieldSchema(name:i, type:int, comment:null), ]
+PREHOOK: query: alter table orc_llap concatenate
+PREHOOK: type: ALTER_TABLE_MERGE
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: default@orc_llap
+POSTHOOK: query: alter table orc_llap concatenate
+POSTHOOK: type: ALTER_TABLE_MERGE
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: default@orc_llap
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ filterExpr: ((cint > 10) and cbigint is not null) (type: boolean)
- Statistics: Num rows: 99580 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 99583 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((cint > 10) and cbigint is not null) (type: boolean)
- Statistics: Num rows: 16597 Data size: 265554 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 16597 Data size: 265553 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cint (type: int), csmallint (type: smallint), cbigint (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 16597 Data size: 265554 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 16597 Data size: 265553 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
- Statistics: Num rows: 16597 Data size: 265554 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 16597 Data size: 265553 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select cint, csmallint, cbigint from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-1116444519372
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select * from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select * from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ filterExpr: ((cint > 10) and cbigint is not null) (type: boolean)
- Statistics: Num rows: 4979 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 4979 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((cint > 10) and cbigint is not null) (type: boolean)
- Statistics: Num rows: 830 Data size: 265602 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 830 Data size: 265609 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cbigint (type: bigint), cfloat (type: float), cdouble (type: double), cstring1 (type: string), cstring2 (type: string), ctimestamp1 (type: timestamp), ctimestamp2 (type: timestamp), cboolean1 (type: boolean), cboolean2 (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11
- Statistics: Num rows: 830 Data size: 265602 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 830 Data size: 265609 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
- Statistics: Num rows: 830 Data size: 265602 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 830 Data size: 265609 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select * from orc_llap where cint > 10 and cbigint is not null
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select * from orc_llap where cint > 10 and cbigint is not null
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-395218182278
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select cstring2 from orc_llap where cint > 5 and cint < 10
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select cstring2 from orc_llap where cint > 5 and cint < 10
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
+ filterExpr: ((cint > 5) and (cint < 10)) (type: boolean)
- Statistics: Num rows: 15320 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 15320 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((cint > 5) and (cint < 10)) (type: boolean)
- Statistics: Num rows: 1702 Data size: 177009 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 1702 Data size: 177014 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cstring2 (type: string)
+ outputColumnNames: _col0
- Statistics: Num rows: 1702 Data size: 177009 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 1702 Data size: 177014 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
- Statistics: Num rows: 1702 Data size: 177009 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 1702 Data size: 177014 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select cstring2 from orc_llap where cint > 5 and cint < 10
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select cstring2 from orc_llap where cint > 5 and cint < 10
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+NULL
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: orc_llap
- Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 7966 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cstring1 (type: string), cstring2 (type: string)
+ outputColumnNames: cstring1, cstring2
- Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 7966 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ keys: cstring1 (type: string), cstring2 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 7966 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 7966 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: bigint)
+ Execution mode: vectorized
+ LLAP IO: all inputs
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 3983 Data size: 796646 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3983 Data size: 796669 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
- Statistics: Num rows: 3983 Data size: 796646 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3983 Data size: 796669 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select cstring1, cstring2, count(*) from orc_llap group by cstring1, cstring2
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-201218418313
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: explain
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+PREHOOK: type: QUERY
+POSTHOOK: query: explain
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-4 is a root stage
+ Stage-3 depends on stages: Stage-4
+ Stage-0 depends on stages: Stage-3
+
+STAGE PLANS:
+ Stage: Stage-4
+ Map Reduce Local Work
+ Alias -> Map Local Tables:
+ o1
+ Fetch Operator
+ limit: -1
+ Alias -> Map Local Operator Tree:
+ o1
+ TableScan
+ alias: o1
+ filterExpr: (csmallint is not null and cbigint is not null) (type: boolean)
- Statistics: Num rows: 14225 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 14226 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (csmallint is not null and cbigint is not null) (type: boolean)
- Statistics: Num rows: 3557 Data size: 398407 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3557 Data size: 398390 Basic stats: COMPLETE Column stats: NONE
+ HashTable Sink Operator
+ keys:
+ 0 csmallint (type: smallint)
+ 1 csmallint (type: smallint)
+
+ Stage: Stage-3
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ alias: o2
+ filterExpr: (csmallint is not null and cbigint is not null) (type: boolean)
- Statistics: Num rows: 14225 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 14226 Data size: 1593339 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (csmallint is not null and cbigint is not null) (type: boolean)
- Statistics: Num rows: 3557 Data size: 398407 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3557 Data size: 398390 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 csmallint (type: smallint)
+ 1 csmallint (type: smallint)
+ outputColumnNames: _col6, _col22
- Statistics: Num rows: 3912 Data size: 438247 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3912 Data size: 438229 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col6 (type: string), _col22 (type: string)
+ outputColumnNames: _col0, _col1
- Statistics: Num rows: 3912 Data size: 438247 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3912 Data size: 438229 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
- Statistics: Num rows: 3912 Data size: 438247 Basic stats: COMPLETE Column stats: NONE
++ Statistics: Num rows: 3912 Data size: 438229 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
+ LLAP IO: all inputs
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: create table llap_temp_table as
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: database:default
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: create table llap_temp_table as
+select o1.cstring1, o2.cstring2 from orc_llap o1 inner join orc_llap o2 on o1.csmallint = o2.csmallint where o1.cbigint is not null and o2.cbigint is not null
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: select sum(hash(*)) from llap_temp_table
+PREHOOK: type: QUERY
+PREHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+POSTHOOK: query: select sum(hash(*)) from llap_temp_table
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@llap_temp_table
+#### A masked pattern was here ####
+-2941848734345024
+PREHOOK: query: drop table llap_temp_table
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@llap_temp_table
+PREHOOK: Output: default@llap_temp_table
+POSTHOOK: query: drop table llap_temp_table
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@llap_temp_table
+POSTHOOK: Output: default@llap_temp_table
+PREHOOK: query: DROP TABLE cross_numbers
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@cross_numbers
+PREHOOK: Output: default@cross_numbers
+POSTHOOK: query: DROP TABLE cross_numbers
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@cross_numbers
+POSTHOOK: Output: default@cross_numbers
+PREHOOK: query: DROP TABLE orc_llap
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@orc_llap
+PREHOOK: Output: default@orc_llap
+POSTHOOK: query: DROP TABLE orc_llap
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@orc_llap
+POSTHOOK: Output: default@orc_llap
+PREHOOK: query: DROP TABLE orc_llap_small
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@orc_llap_small
+PREHOOK: Output: default@orc_llap_small
+POSTHOOK: query: DROP TABLE orc_llap_small
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@orc_llap_small
+POSTHOOK: Output: default@orc_llap_small
[17/51] [abbrv] hive git commit: HIVE-9898: LLAP: Sort out issues
with UGI and cached FileSystems (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-9898: LLAP: Sort out issues with UGI and cached FileSystems (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/7148ea09
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/7148ea09
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/7148ea09
Branch: refs/heads/master
Commit: 7148ea0903059733b234359e600df479c82b9673
Parents: 683fb68
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Mon Sep 21 16:11:18 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Mon Sep 21 16:11:18 2015 -0500
----------------------------------------------------------------------
.../hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/7148ea09/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
----------------------------------------------------------------------
diff --git a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
index e0bd48a..157f9d3 100644
--- a/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
+++ b/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java
@@ -27,8 +27,8 @@ import java.util.concurrent.Executors;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
-import com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.hive.common.CallableWithNdc;
import org.apache.hadoop.hive.llap.daemon.FragmentCompletionHandler;
import org.apache.hadoop.hive.llap.daemon.HistoryLogger;
@@ -62,6 +62,7 @@ import org.apache.tez.runtime.task.TezTaskRunner2;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Stopwatch;
import com.google.common.collect.HashMultimap;
import com.google.common.collect.Multimap;
@@ -232,8 +233,7 @@ public class TaskRunnerCallable extends CallableWithNdc<TaskRunner2Result> {
isCompleted.set(true);
return result;
} finally {
- // TODO Fix UGI and FS Handling. Closing UGI here causes some errors right now.
- // FileSystem.closeAllForUGI(taskUgi);
+ FileSystem.closeAllForUGI(taskUgi);
LOG.info("ExecutionTime for Container: " + request.getContainerIdString() + "=" +
runtimeWatch.stop().elapsedMillis());
if (LOG.isDebugEnabled()) {
[18/51] [abbrv] hive git commit: HIVE-11909 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-11909 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/47187618
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/47187618
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/47187618
Branch: refs/heads/master
Commit: 47187618bbdd014c04465ca3eb4c7843132c9c25
Parents: 7148ea0 2a65989
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Sep 21 16:04:08 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Sep 21 16:04:08 2015 -0700
----------------------------------------------------------------------
.../apache/hadoop/hive/ql/log/PerfLogger.java | 196 +++++++++
.../HiveHBaseTableSnapshotInputFormat.java | 21 +-
.../queries/positive/hbase_handler_snapshot.q | 4 +
.../positive/hbase_handler_snapshot.q.out | 22 +
.../hive/thrift/TestHadoop20SAuthBridge.java | 420 ------------------
.../hive/thrift/TestHadoopAuthBridge23.java | 423 +++++++++++++++++++
.../hive/metastore/RetryingHMSHandler.java | 33 +-
.../java/org/apache/hadoop/hive/ql/Driver.java | 11 +-
.../hadoop/hive/ql/exec/MapJoinOperator.java | 3 +-
.../ql/exec/SparkHashTableSinkOperator.java | 3 +-
.../apache/hadoop/hive/ql/exec/Utilities.java | 11 +-
.../hadoop/hive/ql/exec/spark/SparkPlan.java | 3 +-
.../hive/ql/exec/spark/SparkPlanGenerator.java | 9 +-
.../hive/ql/exec/spark/SparkRecordHandler.java | 3 +-
.../hadoop/hive/ql/exec/spark/SparkTask.java | 2 +-
.../ql/exec/spark/status/SparkJobMonitor.java | 2 +-
.../hive/ql/exec/tez/RecordProcessor.java | 4 +-
.../hive/ql/exec/tez/ReduceRecordProcessor.java | 1 -
.../hive/ql/exec/tez/ReduceRecordSource.java | 3 +-
.../hadoop/hive/ql/exec/tez/TezJobMonitor.java | 17 +-
.../hadoop/hive/ql/exec/tez/TezProcessor.java | 3 +-
.../apache/hadoop/hive/ql/exec/tez/TezTask.java | 2 +-
.../hive/ql/io/CombineHiveInputFormat.java | 10 +-
.../hadoop/hive/ql/io/HiveInputFormat.java | 4 +-
.../apache/hadoop/hive/ql/log/PerfLogger.java | 195 ---------
.../hive/ql/optimizer/ppr/PartitionPruner.java | 7 +-
.../hive/ql/parse/spark/SparkCompiler.java | 3 +-
.../hadoop/hive/ql/session/SessionState.java | 37 +-
.../persistence/TestBytesBytesMultiHashMap.java | 3 +
.../ql/exec/persistence/TestHashPartition.java | 29 ++
.../clientpositive/parquet_ppd_boolean.q | 4 +-
.../queries/clientpositive/parquet_ppd_char.q | 12 +-
.../queries/clientpositive/parquet_ppd_date.q | 16 +-
.../clientpositive/parquet_ppd_decimal.q | 32 +-
.../clientpositive/parquet_ppd_timestamp.q | 16 +-
.../clientpositive/parquet_ppd_varchar.q | 12 +-
.../clientpositive/parquet_ppd_boolean.q.out | 28 +-
.../clientpositive/parquet_ppd_char.q.out | 84 ++--
.../clientpositive/parquet_ppd_date.q.out | 112 ++---
.../clientpositive/parquet_ppd_decimal.q.out | 224 +++++-----
.../clientpositive/parquet_ppd_timestamp.q.out | 112 ++---
.../clientpositive/parquet_ppd_varchar.q.out | 84 ++--
.../auth/LdapAuthenticationProviderImpl.java | 82 +++-
.../hadoop/hive/shims/Hadoop20SShims.java | 5 +-
.../apache/hadoop/hive/shims/Hadoop23Shims.java | 4 +-
.../hive/thrift/HadoopThriftAuthBridge.java | 3 +
46 files changed, 1263 insertions(+), 1051 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/RecordProcessor.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java
index f177f0d,91ba2bb..7c41cb6
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java
@@@ -53,10 -51,7 +53,9 @@@ import org.apache.tez.runtime.api.Logic
import org.apache.tez.runtime.api.LogicalOutput;
import org.apache.tez.runtime.api.ProcessorContext;
import org.apache.tez.runtime.api.Reader;
- import org.apache.tez.runtime.library.api.KeyValuesReader;
+import com.google.common.collect.Lists;
+
/**
* Process input from tez LogicalInput and write output - for a map plan
* Just pump the records through the query plan.
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezJobMonitor.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezProcessor.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
index 3501d19,1ac1669..45ee9c5
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java
@@@ -36,11 -36,7 +36,10 @@@ import org.apache.hadoop.conf.Configura
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
import org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil;
+import org.apache.hadoop.hive.llap.io.api.LlapIo;
+import org.apache.hadoop.hive.llap.io.api.LlapIoProxy;
- import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
import org.apache.hadoop.hive.ql.exec.spark.SparkDynamicPartitionPruner;
import org.apache.hadoop.hive.ql.plan.TableDesc;
import org.apache.hadoop.hive.ql.exec.Operator;
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
----------------------------------------------------------------------
diff --cc shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
index d47e3ff,93dcbd3..f60e8f0
--- a/shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
+++ b/shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
@@@ -68,11 -69,14 +68,13 @@@ import org.apache.hadoop.mapreduce.Task
import org.apache.hadoop.mapreduce.TaskID;
import org.apache.hadoop.security.Credentials;
import org.apache.hadoop.security.KerberosName;
-import org.apache.hadoop.security.token.Token;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
import org.apache.hadoop.tools.distcp2.DistCp;
import org.apache.hadoop.tools.distcp2.DistCpOptions;
+ import org.apache.hadoop.tools.distcp2.DistCpOptions.FileAttribute;
+
import org.apache.hadoop.util.Progressable;
-import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.VersionInfo;
http://git-wip-us.apache.org/repos/asf/hive/blob/47187618/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
----------------------------------------------------------------------
diff --cc shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
index fd6b17d,83369ee..5a136d8
--- a/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
+++ b/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
@@@ -95,11 -92,12 +95,12 @@@ import org.apache.hadoop.mapreduce.Task
import org.apache.hadoop.mapreduce.task.JobContextImpl;
import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.net.NetUtils;
-import org.apache.hadoop.security.authentication.util.KerberosName;
import org.apache.hadoop.security.Credentials;
import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.authentication.util.KerberosName;
import org.apache.hadoop.tools.DistCp;
import org.apache.hadoop.tools.DistCpOptions;
+ import org.apache.hadoop.tools.DistCpOptions.FileAttribute;
import org.apache.hadoop.util.Progressable;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.tez.test.MiniTezCluster;
[08/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/tez/union_view.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/union_view.q.out b/ql/src/test/results/clientpositive/tez/union_view.q.out
index ae6d7c8..51c2785 100644
--- a/ql/src/test/results/clientpositive/tez/union_view.q.out
+++ b/ql/src/test/results/clientpositive/tez/union_view.q.out
@@ -270,16 +270,21 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: ((key = 86) and (ds = '1')) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '1' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -289,16 +294,21 @@ STAGE PLANS:
TableScan
alias: src_union_2
filterExpr: ((key = 86) and (ds = '1')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '1')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '1' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -308,16 +318,21 @@ STAGE PLANS:
TableScan
alias: src_union_3
filterExpr: ((key = 86) and (ds = '1')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '1')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '1' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -349,16 +364,21 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: ((key = 86) and (ds = '2')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '2')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '2' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -368,16 +388,21 @@ STAGE PLANS:
TableScan
alias: src_union_2
filterExpr: ((key = 86) and (ds = '2')) (type: boolean)
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '2' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -387,16 +412,21 @@ STAGE PLANS:
TableScan
alias: src_union_3
filterExpr: ((key = 86) and (ds = '2')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '2')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '2' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -428,16 +458,21 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: ((key = 86) and (ds = '3')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '3')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '3' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -447,16 +482,21 @@ STAGE PLANS:
TableScan
alias: src_union_2
filterExpr: ((key = 86) and (ds = '3')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '3')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '3' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -466,16 +506,21 @@ STAGE PLANS:
TableScan
alias: src_union_3
filterExpr: ((key = 86) and (ds = '3')) (type: boolean)
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '3' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -508,51 +553,66 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: ((key = 86) and ds is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string), ds (type: string)
outputColumnNames: _col1, _col2
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col2 (type: string)
outputColumnNames: _col1, _col2
+ Statistics: Num rows: 1250 Data size: 13280 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
+ Statistics: Num rows: 1250 Data size: 13280 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Map 4
Map Operator Tree:
TableScan
alias: src_union_2
filterExpr: ((key = 86) and ds is not null) (type: boolean)
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string), ds (type: string)
outputColumnNames: _col1, _col2
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col2 (type: string)
outputColumnNames: _col1, _col2
+ Statistics: Num rows: 1250 Data size: 13280 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
+ Statistics: Num rows: 1250 Data size: 13280 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Map 5
Map Operator Tree:
TableScan
alias: src_union_3
filterExpr: ((key = 86) and ds is not null) (type: boolean)
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string), ds (type: string)
outputColumnNames: _col1, _col2
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col2 (type: string)
outputColumnNames: _col1, _col2
+ Statistics: Num rows: 1250 Data size: 13280 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
+ Statistics: Num rows: 1250 Data size: 13280 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Reducer 3
Reduce Operator Tree:
@@ -605,46 +665,69 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: (ds = '1') (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ expressions: '1' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 4
Map Operator Tree:
TableScan
alias: src_union_2
filterExpr: (ds = '1') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '1') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '1' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 5
Map Operator Tree:
TableScan
alias: src_union_3
filterExpr: (ds = '1') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '1') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '1' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Reducer 3
Reduce Operator Tree:
@@ -688,46 +771,69 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: (ds = '2') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '2') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '2' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 1002 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 4
Map Operator Tree:
TableScan
alias: src_union_2
filterExpr: (ds = '2') (type: boolean)
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ expressions: '2' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ Statistics: Num rows: 1002 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 5
Map Operator Tree:
TableScan
alias: src_union_3
filterExpr: (ds = '2') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '2') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '2' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 1002 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Reducer 3
Reduce Operator Tree:
@@ -771,46 +877,69 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: (ds = '3') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '3') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '3' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 1002 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 4
Map Operator Tree:
TableScan
alias: src_union_2
filterExpr: (ds = '3') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '3') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '3' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 1002 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 5
Map Operator Tree:
TableScan
alias: src_union_3
filterExpr: (ds = '3') (type: boolean)
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ expressions: '3' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ Statistics: Num rows: 1002 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Reducer 3
Reduce Operator Tree:
@@ -856,16 +985,21 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: ((key = 86) and (ds = '4')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '4')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '4' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -875,16 +1009,21 @@ STAGE PLANS:
TableScan
alias: src_union_2
filterExpr: ((key = 86) and (ds = '4')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: ((key = 86) and (ds = '4')) (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '4' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -894,16 +1033,21 @@ STAGE PLANS:
TableScan
alias: src_union_3
filterExpr: ((key = 86) and (ds = '4')) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (key = 86) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: value (type: string)
outputColumnNames: _col1
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: 86 (type: int), _col1 (type: string), '4' (type: string)
outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 252 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -937,46 +1081,69 @@ STAGE PLANS:
TableScan
alias: src_union_1
filterExpr: (ds = '4') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '4') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '4' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 4
Map Operator Tree:
TableScan
alias: src_union_2
filterExpr: (ds = '4') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
predicate: (ds = '4') (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ expressions: '4' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Map 5
Map Operator Tree:
TableScan
alias: src_union_3
filterExpr: (ds = '4') (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ expressions: '4' (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
+ Statistics: Num rows: 502 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(1)
mode: hash
outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
Reducer 3
Reduce Operator Tree:
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/unionDistinct_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/unionDistinct_1.q.out b/ql/src/test/results/clientpositive/unionDistinct_1.q.out
index 81c46da..4fb230d 100644
--- a/ql/src/test/results/clientpositive/unionDistinct_1.q.out
+++ b/ql/src/test/results/clientpositive/unionDistinct_1.q.out
@@ -15369,6 +15369,38 @@ POSTHOOK: Lineage: src10_3.key SIMPLE [(src)src.FieldSchema(name:key, type:strin
POSTHOOK: Lineage: src10_3.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
POSTHOOK: Lineage: src10_4.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ]
POSTHOOK: Lineage: src10_4.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
+PREHOOK: query: analyze table src10_1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_1
+PREHOOK: Output: default@src10_1
+POSTHOOK: query: analyze table src10_1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_1
+POSTHOOK: Output: default@src10_1
+PREHOOK: query: analyze table src10_2 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_2
+PREHOOK: Output: default@src10_2
+POSTHOOK: query: analyze table src10_2 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_2
+POSTHOOK: Output: default@src10_2
+PREHOOK: query: analyze table src10_3 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_3
+PREHOOK: Output: default@src10_3
+POSTHOOK: query: analyze table src10_3 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_3
+POSTHOOK: Output: default@src10_3
+PREHOOK: query: analyze table src10_4 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_4
+PREHOOK: Output: default@src10_4
+POSTHOOK: query: analyze table src10_4 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_4
+POSTHOOK: Output: default@src10_4
PREHOOK: query: -- When we convert the Join of sub1 and sub0 into a MapJoin,
-- we can use a single MR job to evaluate this entire query.
explain
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/vector_leftsemi_mapjoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/vector_leftsemi_mapjoin.q.out b/ql/src/test/results/clientpositive/vector_leftsemi_mapjoin.q.out
index 8502085..dcaabd2 100644
--- a/ql/src/test/results/clientpositive/vector_leftsemi_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/vector_leftsemi_mapjoin.q.out
@@ -102,6 +102,14 @@ POSTHOOK: Input: default@t3
8 val_4
8 val_8
9 val_9
+PREHOOK: query: analyze table t3 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t3
+PREHOOK: Output: default@t3
+POSTHOOK: query: analyze table t3 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t3
+POSTHOOK: Output: default@t3
PREHOOK: query: create table t4 (key int, value string) stored as orc
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/vector_multi_insert.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/vector_multi_insert.q.out b/ql/src/test/results/clientpositive/vector_multi_insert.q.out
index 4d10ee8..78456c7 100644
--- a/ql/src/test/results/clientpositive/vector_multi_insert.q.out
+++ b/ql/src/test/results/clientpositive/vector_multi_insert.q.out
@@ -56,6 +56,14 @@ POSTHOOK: query: create table orc_rn3 (rn int)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@orc_rn3
+PREHOOK: query: analyze table orc1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@orc1
+PREHOOK: Output: default@orc1
+POSTHOOK: query: analyze table orc1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@orc1
+POSTHOOK: Output: default@orc1
PREHOOK: query: explain from orc1 a
insert overwrite table orc_rn1 select a.* where a.rn < 100
insert overwrite table orc_rn2 select a.* where a.rn >= 100 and a.rn < 1000
[19/51] [abbrv] hive git commit: HIVE-11921 : merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-11921 : merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/f84eefa5
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/f84eefa5
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/f84eefa5
Branch: refs/heads/master
Commit: f84eefa526346b7b48adebfd48044df7aff04e29
Parents: 4718761 451381c
Author: Sergey Shelukhin <se...@apache.org>
Authored: Tue Sep 22 14:31:29 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Tue Sep 22 14:31:29 2015 -0700
----------------------------------------------------------------------
.../apache/hadoop/hive/ant/QTestGenTask.java | 11 +
bin/ext/hbaseimport.cmd | 35 +
bin/ext/hbaseimport.sh | 27 +
bin/ext/hbaseschematool.sh | 27 +
.../apache/hadoop/hive/common/ObjectPair.java | 5 +
.../org/apache/hadoop/hive/conf/HiveConf.java | 98 +-
.../apache/hive/common/util/BloomFilter.java | 20 +-
data/conf/tez/hive-site.xml | 10 +
.../antlr4/org/apache/hive/hplsql/Hplsql.g4 | 70 +-
.../main/java/org/apache/hive/hplsql/Cmp.java | 314 +
.../java/org/apache/hive/hplsql/Column.java | 29 +-
.../main/java/org/apache/hive/hplsql/Conn.java | 21 +
.../main/java/org/apache/hive/hplsql/Copy.java | 50 +-
.../main/java/org/apache/hive/hplsql/Exec.java | 66 +-
.../java/org/apache/hive/hplsql/Expression.java | 33 +-
.../main/java/org/apache/hive/hplsql/File.java | 18 +-
.../main/java/org/apache/hive/hplsql/Meta.java | 28 +-
.../main/java/org/apache/hive/hplsql/Query.java | 18 +
.../java/org/apache/hive/hplsql/Select.java | 23 +-
.../main/java/org/apache/hive/hplsql/Stmt.java | 8 +-
.../main/java/org/apache/hive/hplsql/Var.java | 110 +-
.../apache/hive/hplsql/functions/Function.java | 6 +-
.../hive/hplsql/functions/FunctionMisc.java | 121 +
.../org/apache/hive/hplsql/TestHplsqlLocal.java | 18 +
.../apache/hive/hplsql/TestHplsqlOffline.java | 5 +
hplsql/src/test/queries/db/cmp_row_count.sql | 4 +
hplsql/src/test/queries/db/cmp_sum.sql | 3 +
hplsql/src/test/queries/db/copy_to_file.sql | 2 +
hplsql/src/test/queries/db/copy_to_hdfs.sql | 2 +
hplsql/src/test/queries/db/copy_to_table.sql | 2 +
hplsql/src/test/queries/db/part_count.sql | 17 +
hplsql/src/test/queries/db/part_count_by.sql | 4 +
hplsql/src/test/queries/db/schema.sql | 32 +
hplsql/src/test/queries/db/select_into.sql | 20 +-
hplsql/src/test/queries/db/select_into2.sql | 17 +
.../test/queries/local/create_procedure2.sql | 16 +
hplsql/src/test/queries/local/if2.sql | 5 +
hplsql/src/test/queries/local/include.sql | 2 +
hplsql/src/test/queries/local/include_file.sql | 1 +
hplsql/src/test/queries/local/mult_div.sql | 8 +
hplsql/src/test/queries/offline/select_db2.sql | 5 +
.../src/test/results/db/cmp_row_count.out.txt | 12 +
hplsql/src/test/results/db/cmp_sum.out.txt | 320 +
hplsql/src/test/results/db/copy_to_file.out.txt | 6 +
hplsql/src/test/results/db/copy_to_hdfs.out.txt | 4 +
.../src/test/results/db/copy_to_table.out.txt | 2 +
hplsql/src/test/results/db/part_count.out.txt | 15 +
.../src/test/results/db/part_count_by.out.txt | 13 +
hplsql/src/test/results/db/select_into.out.txt | 58 +-
hplsql/src/test/results/db/select_into2.out.txt | 19 +
.../results/local/create_procedure2.out.txt | 10 +
hplsql/src/test/results/local/if2.out.txt | 4 +
hplsql/src/test/results/local/include.out.txt | 8 +
hplsql/src/test/results/local/mult_div.out.txt | 7 +
.../src/test/results/offline/select_db2.out.txt | 6 +
itests/hive-unit/pom.xml | 35 +
.../hadoop/hive/metastore/TestAdminUser.java | 4 +-
.../hive/metastore/TestHiveMetaStore.java | 3 +
.../metastore/hbase/HBaseIntegrationTests.java | 117 +
.../TestHBaseAggrStatsCacheIntegration.java | 691 +
.../hive/metastore/hbase/TestHBaseImport.java | 650 +
.../metastore/hbase/TestHBaseMetastoreSql.java | 223 +
.../hbase/TestHBaseStoreIntegration.java | 1794 +
.../hbase/TestStorageDescriptorSharing.java | 191 +
itests/qtest/pom.xml | 10 +-
itests/util/pom.xml | 32 +
.../metastore/hbase/HBaseStoreTestUtil.java | 45 +
.../org/apache/hadoop/hive/ql/QTestUtil.java | 53 +-
.../hive/jdbc/ZooKeeperHiveClientHelper.java | 32 +-
metastore/if/hive_metastore.thrift | 54 +
metastore/pom.xml | 82 +
.../metastore/hbase/HbaseMetastoreProto.java | 34901 +++++++++++++++++
.../gen/thrift/gen-cpp/ThriftHiveMetastore.cpp | 6919 ++--
.../gen/thrift/gen-cpp/ThriftHiveMetastore.h | 664 +
.../ThriftHiveMetastore_server.skeleton.cpp | 25 +
.../gen/thrift/gen-cpp/hive_metastore_types.cpp | 1294 +-
.../gen/thrift/gen-cpp/hive_metastore_types.h | 371 +
.../hive/metastore/api/AbortTxnRequest.java | 2 +-
.../metastore/api/AddDynamicPartitions.java | 2 +-
.../metastore/api/AddPartitionsRequest.java | 2 +-
.../hive/metastore/api/AddPartitionsResult.java | 2 +-
.../hadoop/hive/metastore/api/AggrStats.java | 2 +-
.../metastore/api/AlreadyExistsException.java | 2 +-
.../metastore/api/BinaryColumnStatsData.java | 2 +-
.../metastore/api/BooleanColumnStatsData.java | 2 +-
.../hive/metastore/api/CheckLockRequest.java | 2 +-
.../metastore/api/ClearFileMetadataRequest.java | 438 +
.../metastore/api/ClearFileMetadataResult.java | 283 +
.../hive/metastore/api/ColumnStatistics.java | 2 +-
.../metastore/api/ColumnStatisticsDesc.java | 2 +-
.../hive/metastore/api/ColumnStatisticsObj.java | 2 +-
.../hive/metastore/api/CommitTxnRequest.java | 2 +-
.../hive/metastore/api/CompactionRequest.java | 2 +-
.../api/ConfigValSecurityException.java | 2 +-
.../api/CurrentNotificationEventId.java | 2 +-
.../hadoop/hive/metastore/api/Database.java | 2 +-
.../apache/hadoop/hive/metastore/api/Date.java | 2 +-
.../hive/metastore/api/DateColumnStatsData.java | 2 +-
.../hadoop/hive/metastore/api/Decimal.java | 2 +-
.../metastore/api/DecimalColumnStatsData.java | 2 +-
.../metastore/api/DoubleColumnStatsData.java | 2 +-
.../hive/metastore/api/DropPartitionsExpr.java | 2 +-
.../metastore/api/DropPartitionsRequest.java | 2 +-
.../metastore/api/DropPartitionsResult.java | 2 +-
.../hive/metastore/api/EnvironmentContext.java | 2 +-
.../hadoop/hive/metastore/api/FieldSchema.java | 2 +-
.../hive/metastore/api/FireEventRequest.java | 2 +-
.../hive/metastore/api/FireEventResponse.java | 2 +-
.../hadoop/hive/metastore/api/Function.java | 2 +-
.../metastore/api/GetAllFunctionsResponse.java | 38 +-
.../api/GetFileMetadataByExprRequest.java | 548 +
.../api/GetFileMetadataByExprResult.java | 703 +
.../metastore/api/GetFileMetadataRequest.java | 438 +
.../metastore/api/GetFileMetadataResult.java | 540 +
.../metastore/api/GetOpenTxnsInfoResponse.java | 2 +-
.../hive/metastore/api/GetOpenTxnsResponse.java | 2 +-
.../api/GetPrincipalsInRoleRequest.java | 2 +-
.../api/GetPrincipalsInRoleResponse.java | 2 +-
.../api/GetRoleGrantsForPrincipalRequest.java | 2 +-
.../api/GetRoleGrantsForPrincipalResponse.java | 2 +-
.../api/GrantRevokePrivilegeRequest.java | 2 +-
.../api/GrantRevokePrivilegeResponse.java | 2 +-
.../metastore/api/GrantRevokeRoleRequest.java | 2 +-
.../metastore/api/GrantRevokeRoleResponse.java | 2 +-
.../hive/metastore/api/HeartbeatRequest.java | 2 +-
.../metastore/api/HeartbeatTxnRangeRequest.java | 2 +-
.../api/HeartbeatTxnRangeResponse.java | 2 +-
.../hive/metastore/api/HiveObjectPrivilege.java | 2 +-
.../hive/metastore/api/HiveObjectRef.java | 2 +-
.../apache/hadoop/hive/metastore/api/Index.java | 2 +-
.../api/IndexAlreadyExistsException.java | 2 +-
.../metastore/api/InsertEventRequestData.java | 2 +-
.../metastore/api/InvalidInputException.java | 2 +-
.../metastore/api/InvalidObjectException.java | 2 +-
.../api/InvalidOperationException.java | 2 +-
.../api/InvalidPartitionException.java | 2 +-
.../hive/metastore/api/LockComponent.java | 2 +-
.../hadoop/hive/metastore/api/LockRequest.java | 2 +-
.../hadoop/hive/metastore/api/LockResponse.java | 2 +-
.../hive/metastore/api/LongColumnStatsData.java | 2 +-
.../hive/metastore/api/MetaException.java | 2 +-
.../hive/metastore/api/MetadataPpdResult.java | 508 +
.../hive/metastore/api/NoSuchLockException.java | 2 +-
.../metastore/api/NoSuchObjectException.java | 2 +-
.../hive/metastore/api/NoSuchTxnException.java | 2 +-
.../hive/metastore/api/NotificationEvent.java | 2 +-
.../metastore/api/NotificationEventRequest.java | 2 +-
.../api/NotificationEventResponse.java | 2 +-
.../hive/metastore/api/OpenTxnRequest.java | 2 +-
.../hive/metastore/api/OpenTxnsResponse.java | 2 +-
.../apache/hadoop/hive/metastore/api/Order.java | 2 +-
.../hadoop/hive/metastore/api/Partition.java | 2 +-
.../api/PartitionListComposingSpec.java | 2 +-
.../hive/metastore/api/PartitionSpec.java | 2 +-
.../api/PartitionSpecWithSharedSD.java | 2 +-
.../hive/metastore/api/PartitionWithoutSD.java | 2 +-
.../metastore/api/PartitionsByExprRequest.java | 2 +-
.../metastore/api/PartitionsByExprResult.java | 2 +-
.../metastore/api/PartitionsStatsRequest.java | 2 +-
.../metastore/api/PartitionsStatsResult.java | 2 +-
.../metastore/api/PrincipalPrivilegeSet.java | 2 +-
.../hadoop/hive/metastore/api/PrivilegeBag.java | 2 +-
.../hive/metastore/api/PrivilegeGrantInfo.java | 2 +-
.../metastore/api/PutFileMetadataRequest.java | 588 +
.../metastore/api/PutFileMetadataResult.java | 283 +
.../hadoop/hive/metastore/api/ResourceUri.java | 2 +-
.../apache/hadoop/hive/metastore/api/Role.java | 2 +-
.../hive/metastore/api/RolePrincipalGrant.java | 2 +-
.../hadoop/hive/metastore/api/Schema.java | 2 +-
.../hadoop/hive/metastore/api/SerDeInfo.java | 2 +-
.../api/SetPartitionsStatsRequest.java | 2 +-
.../hive/metastore/api/ShowCompactRequest.java | 2 +-
.../hive/metastore/api/ShowCompactResponse.java | 2 +-
.../api/ShowCompactResponseElement.java | 2 +-
.../hive/metastore/api/ShowLocksRequest.java | 2 +-
.../hive/metastore/api/ShowLocksResponse.java | 2 +-
.../metastore/api/ShowLocksResponseElement.java | 2 +-
.../hadoop/hive/metastore/api/SkewedInfo.java | 2 +-
.../hive/metastore/api/StorageDescriptor.java | 2 +-
.../metastore/api/StringColumnStatsData.java | 2 +-
.../apache/hadoop/hive/metastore/api/Table.java | 2 +-
.../hive/metastore/api/TableStatsRequest.java | 2 +-
.../hive/metastore/api/TableStatsResult.java | 2 +-
.../hive/metastore/api/ThriftHiveMetastore.java | 8422 ++--
.../hive/metastore/api/TxnAbortedException.java | 2 +-
.../hadoop/hive/metastore/api/TxnInfo.java | 2 +-
.../hive/metastore/api/TxnOpenException.java | 2 +-
.../apache/hadoop/hive/metastore/api/Type.java | 2 +-
.../hive/metastore/api/UnknownDBException.java | 2 +-
.../api/UnknownPartitionException.java | 2 +-
.../metastore/api/UnknownTableException.java | 2 +-
.../hive/metastore/api/UnlockRequest.java | 2 +-
.../hadoop/hive/metastore/api/Version.java | 2 +-
.../gen-php/metastore/ThriftHiveMetastore.php | 2810 +-
.../src/gen/thrift/gen-php/metastore/Types.php | 1009 +-
.../hive_metastore/ThriftHiveMetastore-remote | 49 +-
.../hive_metastore/ThriftHiveMetastore.py | 1563 +-
.../gen/thrift/gen-py/hive_metastore/ttypes.py | 734 +-
.../gen/thrift/gen-rb/hive_metastore_types.rb | 167 +
.../gen/thrift/gen-rb/thrift_hive_metastore.rb | 267 +
.../hadoop/hive/metastore/HiveAlterHandler.java | 38 +-
.../hadoop/hive/metastore/HiveMetaStore.java | 266 +-
.../hive/metastore/HiveMetaStoreClient.java | 112 +-
.../hadoop/hive/metastore/IMetaStoreClient.java | 6 +
.../hadoop/hive/metastore/ObjectStore.java | 418 +-
.../hive/metastore/PartFilterExprUtil.java | 149 +
.../apache/hadoop/hive/metastore/RawStore.java | 66 +-
.../hadoop/hive/metastore/RawStoreProxy.java | 5 +-
.../hbase/AggrStatsInvalidatorFilter.java | 121 +
.../hadoop/hive/metastore/hbase/Counter.java | 53 +
.../hive/metastore/hbase/HBaseConnection.java | 96 +
.../metastore/hbase/HBaseFilterPlanUtil.java | 612 +
.../hive/metastore/hbase/HBaseImport.java | 535 +
.../hive/metastore/hbase/HBaseReadWrite.java | 2106 +
.../hive/metastore/hbase/HBaseSchemaTool.java | 239 +
.../hadoop/hive/metastore/hbase/HBaseStore.java | 2387 ++
.../hadoop/hive/metastore/hbase/HBaseUtils.java | 1340 +
.../hive/metastore/hbase/ObjectCache.java | 81 +
.../hive/metastore/hbase/PartitionCache.java | 168 +
.../metastore/hbase/PartitionKeyComparator.java | 292 +
.../hbase/SharedStorageDescriptor.java | 251 +
.../hadoop/hive/metastore/hbase/StatsCache.java | 326 +
.../metastore/hbase/TephraHBaseConnection.java | 127 +
.../metastore/hbase/VanillaHBaseConnection.java | 137 +
.../stats/BinaryColumnStatsAggregator.java | 35 +
.../stats/BooleanColumnStatsAggregator.java | 35 +
.../hbase/stats/ColumnStatsAggregator.java | 26 +
.../stats/ColumnStatsAggregatorFactory.java | 94 +
.../stats/DecimalColumnStatsAggregator.java | 43 +
.../stats/DoubleColumnStatsAggregator.java | 36 +
.../hbase/stats/LongColumnStatsAggregator.java | 36 +
.../stats/StringColumnStatsAggregator.java | 36 +
.../hive/metastore/parser/ExpressionTree.java | 9 +-
.../metastore/hbase/hbase_metastore_proto.proto | 282 +
.../DummyRawStoreControlledCommit.java | 56 +-
.../DummyRawStoreForJdoConnection.java | 50 +-
.../hadoop/hive/metastore/TestObjectStore.java | 43 +-
.../hadoop/hive/metastore/hbase/MockUtils.java | 199 +
.../hbase/TestHBaseAggregateStatsCache.java | 316 +
.../hbase/TestHBaseFilterPlanUtil.java | 483 +
.../hive/metastore/hbase/TestHBaseStore.java | 1307 +
.../metastore/hbase/TestHBaseStoreCached.java | 378 +
.../hbase/TestSharedStorageDescriptor.java | 153 +
pom.xml | 3 +-
.../hadoop/hive/ql/plan/api/Adjacency.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Graph.java | 2 +-
.../hadoop/hive/ql/plan/api/Operator.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Query.java | 2 +-
.../hadoop/hive/ql/plan/api/QueryPlan.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Stage.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Task.java | 2 +-
.../java/org/apache/hadoop/hive/ql/Driver.java | 6 +
.../org/apache/hadoop/hive/ql/exec/DDLTask.java | 64 +-
.../apache/hadoop/hive/ql/metadata/Hive.java | 9 +-
.../hadoop/hive/ql/metadata/Partition.java | 29 +-
.../AuthorizationPreEventListener.java | 2 +-
.../apache/hadoop/hive/ql/stats/StatsUtils.java | 3 +
.../hive/metastore/TestMetastoreExpr.java | 2 +-
.../dynpart_sort_opt_vectorization.q | 2 +
.../clientpositive/dynpart_sort_optimization.q | 2 +
.../dynpart_sort_opt_vectorization.q.out | 12 +-
.../dynpart_sort_optimization.q.out | 12 +-
.../tez/dynpart_sort_opt_vectorization.q.out | 12 +-
.../tez/dynpart_sort_optimization.q.out | 12 +-
ql/src/test/templates/TestCliDriver.vm | 4 +-
.../hadoop/hive/serde/test/InnerStruct.java | 2 +-
.../hadoop/hive/serde/test/ThriftTestObj.java | 2 +-
.../hadoop/hive/serde2/thrift/test/Complex.java | 2 +-
.../hive/serde2/thrift/test/IntString.java | 2 +-
.../hive/serde2/thrift/test/MegaStruct.java | 2 +-
.../hive/serde2/thrift/test/MiniStruct.java | 2 +-
.../hive/serde2/thrift/test/SetIntString.java | 2 +-
.../BinarySortableSerDeWithEndPrefix.java | 41 +
.../hadoop/hive/service/HiveClusterStatus.java | 2 +-
.../hive/service/HiveServerException.java | 2 +-
.../apache/hadoop/hive/service/ThriftHive.java | 2 +-
.../service/cli/thrift/TArrayTypeEntry.java | 2 +-
.../hive/service/cli/thrift/TBinaryColumn.java | 2 +-
.../hive/service/cli/thrift/TBoolColumn.java | 2 +-
.../hive/service/cli/thrift/TBoolValue.java | 2 +-
.../hive/service/cli/thrift/TByteColumn.java | 2 +-
.../hive/service/cli/thrift/TByteValue.java | 2 +-
.../hive/service/cli/thrift/TCLIService.java | 2 +-
.../cli/thrift/TCancelDelegationTokenReq.java | 2 +-
.../cli/thrift/TCancelDelegationTokenResp.java | 2 +-
.../service/cli/thrift/TCancelOperationReq.java | 2 +-
.../cli/thrift/TCancelOperationResp.java | 2 +-
.../service/cli/thrift/TCloseOperationReq.java | 2 +-
.../service/cli/thrift/TCloseOperationResp.java | 2 +-
.../service/cli/thrift/TCloseSessionReq.java | 2 +-
.../service/cli/thrift/TCloseSessionResp.java | 2 +-
.../hive/service/cli/thrift/TColumnDesc.java | 2 +-
.../hive/service/cli/thrift/TDoubleColumn.java | 2 +-
.../hive/service/cli/thrift/TDoubleValue.java | 2 +-
.../cli/thrift/TExecuteStatementReq.java | 2 +-
.../cli/thrift/TExecuteStatementResp.java | 2 +-
.../service/cli/thrift/TFetchResultsReq.java | 2 +-
.../service/cli/thrift/TFetchResultsResp.java | 2 +-
.../service/cli/thrift/TGetCatalogsReq.java | 2 +-
.../service/cli/thrift/TGetCatalogsResp.java | 2 +-
.../hive/service/cli/thrift/TGetColumnsReq.java | 2 +-
.../service/cli/thrift/TGetColumnsResp.java | 2 +-
.../cli/thrift/TGetDelegationTokenReq.java | 2 +-
.../cli/thrift/TGetDelegationTokenResp.java | 2 +-
.../service/cli/thrift/TGetFunctionsReq.java | 2 +-
.../service/cli/thrift/TGetFunctionsResp.java | 2 +-
.../hive/service/cli/thrift/TGetInfoReq.java | 2 +-
.../hive/service/cli/thrift/TGetInfoResp.java | 2 +-
.../cli/thrift/TGetOperationStatusReq.java | 2 +-
.../cli/thrift/TGetOperationStatusResp.java | 2 +-
.../cli/thrift/TGetResultSetMetadataReq.java | 2 +-
.../cli/thrift/TGetResultSetMetadataResp.java | 2 +-
.../hive/service/cli/thrift/TGetSchemasReq.java | 2 +-
.../service/cli/thrift/TGetSchemasResp.java | 2 +-
.../service/cli/thrift/TGetTableTypesReq.java | 2 +-
.../service/cli/thrift/TGetTableTypesResp.java | 2 +-
.../hive/service/cli/thrift/TGetTablesReq.java | 2 +-
.../hive/service/cli/thrift/TGetTablesResp.java | 2 +-
.../service/cli/thrift/TGetTypeInfoReq.java | 2 +-
.../service/cli/thrift/TGetTypeInfoResp.java | 2 +-
.../service/cli/thrift/THandleIdentifier.java | 2 +-
.../hive/service/cli/thrift/TI16Column.java | 2 +-
.../hive/service/cli/thrift/TI16Value.java | 2 +-
.../hive/service/cli/thrift/TI32Column.java | 2 +-
.../hive/service/cli/thrift/TI32Value.java | 2 +-
.../hive/service/cli/thrift/TI64Column.java | 2 +-
.../hive/service/cli/thrift/TI64Value.java | 2 +-
.../hive/service/cli/thrift/TMapTypeEntry.java | 2 +-
.../service/cli/thrift/TOpenSessionReq.java | 2 +-
.../service/cli/thrift/TOpenSessionResp.java | 2 +-
.../service/cli/thrift/TOperationHandle.java | 2 +-
.../service/cli/thrift/TPrimitiveTypeEntry.java | 2 +-
.../cli/thrift/TRenewDelegationTokenReq.java | 2 +-
.../cli/thrift/TRenewDelegationTokenResp.java | 2 +-
.../apache/hive/service/cli/thrift/TRow.java | 2 +-
.../apache/hive/service/cli/thrift/TRowSet.java | 2 +-
.../hive/service/cli/thrift/TSessionHandle.java | 2 +-
.../apache/hive/service/cli/thrift/TStatus.java | 2 +-
.../hive/service/cli/thrift/TStringColumn.java | 2 +-
.../hive/service/cli/thrift/TStringValue.java | 2 +-
.../service/cli/thrift/TStructTypeEntry.java | 2 +-
.../hive/service/cli/thrift/TTableSchema.java | 2 +-
.../hive/service/cli/thrift/TTypeDesc.java | 2 +-
.../service/cli/thrift/TTypeQualifiers.java | 2 +-
.../service/cli/thrift/TUnionTypeEntry.java | 2 +-
.../cli/thrift/TUserDefinedTypeEntry.java | 2 +-
.../gen-py/hive_service/ThriftHive-remote | 49 +-
347 files changed, 76375 insertions(+), 7194 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/ant/src/org/apache/hadoop/hive/ant/QTestGenTask.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/data/conf/tez/hive-site.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/itests/qtest/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
----------------------------------------------------------------------
diff --cc itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
index 0f17b1b,aab3aab..4cab161
--- a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
+++ b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
@@@ -53,11 -54,15 +53,12 @@@ import java.util.concurrent.TimeUnit
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-import junit.framework.Assert;
-import junit.framework.TestSuite;
-
-import org.apache.commons.lang.StringUtils;
import org.apache.commons.io.FileUtils;
import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
+ import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
@@@ -347,18 -350,43 +353,44 @@@ public class QTestUtil
return "jceks://file" + new Path(keyDir, "test.jks").toUri();
}
+ private void startMiniHBaseCluster() throws Exception {
+ Configuration hbaseConf = HBaseConfiguration.create();
+ hbaseConf.setInt("hbase.master.info.port", -1);
+ utility = new HBaseTestingUtility(hbaseConf);
+ utility.startMiniCluster();
+ conf = new HiveConf(utility.getConfiguration(), Driver.class);
+ HBaseAdmin admin = utility.getHBaseAdmin();
+ // Need to use reflection here to make compilation pass since HBaseIntegrationTests
+ // is not compiled in hadoop-1. All HBaseMetastore tests run under hadoop-2, so this
+ // guarantee HBaseIntegrationTests exist when we hitting this code path
+ java.lang.reflect.Method initHBaseMetastoreMethod = Class.forName(
+ "org.apache.hadoop.hive.metastore.hbase.HBaseStoreTestUtil")
+ .getMethod("initHBaseMetastore", HBaseAdmin.class, HiveConf.class);
+ initHBaseMetastoreMethod.invoke(null, admin, conf);
+ }
+
public QTestUtil(String outDir, String logDir, MiniClusterType clusterType,
- String confDir, String hadoopVer, String initScript,
- String cleanupScript) throws Exception {
+ String confDir, String hadoopVer, String initScript, String cleanupScript)
+ throws Exception {
+ this(outDir, logDir, clusterType, confDir, hadoopVer, initScript, cleanupScript, false);
+ }
+
+ public QTestUtil(String outDir, String logDir, MiniClusterType clusterType,
+ String confDir, String hadoopVer, String initScript, String cleanupScript, boolean useHBaseMetastore)
+ throws Exception {
this.outDir = outDir;
this.logDir = logDir;
+ this.useHBaseMetastore = useHBaseMetastore;
+ Logger hadoopLog = Logger.getLogger("org.apache.hadoop");
+ hadoopLog.setLevel(Level.INFO);
if (confDir != null && !confDir.isEmpty()) {
- HiveConf.setHiveSiteLocation(new URL("file://"
- + new File(confDir).toURI().getPath() + "/hive-site.xml"));
+ HiveConf.setHiveSiteLocation(new URL("file://"+ new File(confDir).toURI().getPath() + "/hive-site.xml"));
System.out.println("Setting hive-site: "+HiveConf.getHiveSiteLocation());
}
+ if (useHBaseMetastore) {
+ startMiniHBaseCluster();
+ }
conf = new HiveConf(Driver.class);
this.hadoopVer = getHadoopMainVersion(hadoopVer);
qMap = new TreeMap<String, String>();
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f84eefa5/ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out
----------------------------------------------------------------------
[23/51] [abbrv] hive git commit: HIVE-11979 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-11979 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/bf8acfb3
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/bf8acfb3
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/bf8acfb3
Branch: refs/heads/master
Commit: bf8acfb3bbe997a9b8c474990889c5cbcb2fbb0f
Parents: 434b7d4 1cf7e25
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Sep 28 12:11:04 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Sep 28 12:11:04 2015 -0700
----------------------------------------------------------------------
.../java/org/apache/hive/beeline/BeeLine.java | 300 +++++++++---
.../org/apache/hive/beeline/BeeLineOpts.java | 43 +-
.../hive/beeline/ClientCommandHookFactory.java | 85 ++++
.../org/apache/hive/beeline/ClientHook.java | 33 ++
.../java/org/apache/hive/beeline/Commands.java | 486 ++++++++++++++-----
.../org/apache/hive/beeline/HiveSchemaTool.java | 22 +-
.../hive/beeline/cli/CliOptionsProcessor.java | 104 ++++
.../org/apache/hive/beeline/cli/HiveCli.java | 37 ++
.../beeline/TestClientCommandHookFactory.java | 32 ++
.../apache/hive/beeline/cli/TestHiveCli.java | 289 +++++++++++
beeline/src/test/resources/hive-site.xml | 37 ++
bin/beeline | 5 +
bin/ext/cli.cmd | 39 +-
bin/ext/cli.sh | 35 +-
bin/ext/util/execHiveCmd.sh | 6 +-
.../org/apache/hadoop/hive/cli/CliDriver.java | 24 +-
.../hadoop/hive/conf/HiveVariableSource.java | 24 +
.../hadoop/hive/conf/VariableSubstitution.java | 70 +++
.../hive/conf/TestVariableSubstitution.java | 63 +++
.../listener/TestNotificationListener.java | 4 +-
.../hcatalog/templeton/tool/LaunchMapper.java | 7 +-
.../org/apache/hive/jdbc/HiveConnection.java | 63 ++-
.../java/org/apache/hadoop/hive/ql/Driver.java | 10 +-
.../apache/hadoop/hive/ql/metadata/Hive.java | 25 +-
.../formatting/MetaDataPrettyFormatUtils.java | 24 +-
.../correlation/ReduceSinkDeDuplication.java | 15 +-
.../ql/parse/ColumnStatsSemanticAnalyzer.java | 14 +-
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 2 +-
.../hive/ql/parse/VariableSubstitution.java | 60 ---
.../ql/processors/AddResourceProcessor.java | 11 +-
.../hive/ql/processors/CompileProcessor.java | 11 +-
.../ql/processors/DeleteResourceProcessor.java | 11 +-
.../hadoop/hive/ql/processors/DfsProcessor.java | 11 +-
.../hadoop/hive/ql/processors/SetProcessor.java | 32 +-
.../hadoop/hive/ql/session/SessionState.java | 21 +
.../clientpositive/insertvalues_espchars.q | 5 +
.../queries/clientpositive/join_grp_diff_keys.q | 21 +
.../clientpositive/describe_pretty.q.out | 366 +++++++-------
.../clientpositive/insertvalues_espchars.q.out | 30 ++
.../clientpositive/join_grp_diff_keys.q.out | 190 ++++++++
.../hive/serde2/avro/AvroDeserializer.java | 2 +-
.../hadoop/hive/serde2/avro/AvroSerDe.java | 13 +-
.../hadoop/hive/serde2/avro/AvroSerdeUtils.java | 7 +-
.../hadoop/hive/serde2/avro/AvroSerializer.java | 2 -
.../cli/operation/HiveCommandOperation.java | 18 +-
.../service/cli/operation/SQLOperation.java | 14 +-
46 files changed, 2178 insertions(+), 545 deletions(-)
----------------------------------------------------------------------
[04/51] [abbrv] hive git commit: HIVE-11853: LLAP: Fix tests related
to stats difference (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/c453c0ed/ql/src/test/results/clientpositive/llap/vector_multi_insert.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_multi_insert.q.out b/ql/src/test/results/clientpositive/llap/vector_multi_insert.q.out
index 3aa3b60..b2e24e8 100644
--- a/ql/src/test/results/clientpositive/llap/vector_multi_insert.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_multi_insert.q.out
@@ -56,6 +56,14 @@ POSTHOOK: query: create table orc_rn3 (rn int)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@orc_rn3
+PREHOOK: query: analyze table orc1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@orc1
+PREHOOK: Output: default@orc1
+POSTHOOK: query: analyze table orc1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@orc1
+POSTHOOK: Output: default@orc1
PREHOOK: query: explain from orc1 a
insert overwrite table orc_rn1 select a.* where a.rn < 100
insert overwrite table orc_rn2 select a.* where a.rn >= 100 and a.rn < 1000
@@ -85,17 +93,17 @@ STAGE PLANS:
Map Operator Tree:
TableScan
alias: a
- Statistics: Num rows: 145 Data size: 580 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (rn < 100) (type: boolean)
- Statistics: Num rows: 48 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: rn (type: int)
outputColumnNames: _col0
- Statistics: Num rows: 48 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 48 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -103,14 +111,14 @@ STAGE PLANS:
name: default.orc_rn1
Filter Operator
predicate: ((rn >= 100) and (rn < 1000)) (type: boolean)
- Statistics: Num rows: 16 Data size: 64 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: rn (type: int)
outputColumnNames: _col0
- Statistics: Num rows: 16 Data size: 64 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 16 Data size: 64 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
@@ -118,14 +126,14 @@ STAGE PLANS:
name: default.orc_rn2
Filter Operator
predicate: (rn >= 1000) (type: boolean)
- Statistics: Num rows: 48 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: rn (type: int)
outputColumnNames: _col0
- Statistics: Num rows: 48 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 48 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
http://git-wip-us.apache.org/repos/asf/hive/blob/c453c0ed/ql/src/test/results/clientpositive/tez/vector_leftsemi_mapjoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/vector_leftsemi_mapjoin.q.out b/ql/src/test/results/clientpositive/tez/vector_leftsemi_mapjoin.q.out
index a851100..7c5f083 100644
--- a/ql/src/test/results/clientpositive/tez/vector_leftsemi_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_leftsemi_mapjoin.q.out
@@ -102,6 +102,14 @@ POSTHOOK: Input: default@t3
8 val_4
8 val_8
9 val_9
+PREHOOK: query: analyze table t3 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t3
+PREHOOK: Output: default@t3
+POSTHOOK: query: analyze table t3 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t3
+POSTHOOK: Output: default@t3
PREHOOK: query: create table t4 (key int, value string) stored as orc
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
http://git-wip-us.apache.org/repos/asf/hive/blob/c453c0ed/ql/src/test/results/clientpositive/tez/vector_multi_insert.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/vector_multi_insert.q.out b/ql/src/test/results/clientpositive/tez/vector_multi_insert.q.out
index 1080bf4..f72949c 100644
--- a/ql/src/test/results/clientpositive/tez/vector_multi_insert.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_multi_insert.q.out
@@ -56,6 +56,14 @@ POSTHOOK: query: create table orc_rn3 (rn int)
POSTHOOK: type: CREATETABLE
POSTHOOK: Output: database:default
POSTHOOK: Output: default@orc_rn3
+PREHOOK: query: analyze table orc1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@orc1
+PREHOOK: Output: default@orc1
+POSTHOOK: query: analyze table orc1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@orc1
+POSTHOOK: Output: default@orc1
PREHOOK: query: explain from orc1 a
insert overwrite table orc_rn1 select a.* where a.rn < 100
insert overwrite table orc_rn2 select a.* where a.rn >= 100 and a.rn < 1000
[47/51] [abbrv] hive git commit: HIVE-12096 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12096 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/4e53bfd1
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/4e53bfd1
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/4e53bfd1
Branch: refs/heads/master
Commit: 4e53bfd10e1d6bd2a24d112d2af452ac49d8b16f
Parents: 1d9574a 09f5e84
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 12 10:47:39 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 12 10:47:39 2015 -0700
----------------------------------------------------------------------
.../apache/hive/hcatalog/data/JsonSerDe.java | 8 +-
.../hive/hcatalog/data/TestJsonSerDe.java | 36 ++++
.../hive/hcatalog/streaming/HiveEndPoint.java | 21 ++
.../hive/hcatalog/streaming/InvalidTable.java | 8 +
.../mutate/worker/BucketIdResolverImpl.java | 16 +-
.../hive/hcatalog/streaming/TestStreaming.java | 68 +++++-
.../mutate/worker/TestBucketIdResolverImpl.java | 2 +-
.../hive/ql/txn/compactor/TestCompactor.java | 13 +-
.../test/resources/testconfiguration.properties | 1 +
.../hadoop/hive/metastore/txn/TxnHandler.java | 4 +
.../hadoop/hive/ql/exec/FileSinkOperator.java | 9 +-
.../hadoop/hive/ql/exec/ReduceSinkOperator.java | 23 +-
.../ql/exec/spark/RemoteHiveSparkClient.java | 22 ++
.../hive/ql/io/DefaultHivePartitioner.java | 3 +-
.../hadoop/hive/ql/lockmgr/DbTxnManager.java | 34 ++-
.../ql/optimizer/calcite/HiveConfigContext.java | 37 ----
.../calcite/HiveHepPlannerContext.java | 37 ++++
.../calcite/HiveVolcanoPlannerContext.java | 37 ++++
.../calcite/cost/HiveVolcanoPlanner.java | 6 +-
.../calcite/rules/HivePreFilteringRule.java | 100 ++++++---
.../calcite/rules/HiveRulesRegistry.java | 44 ++++
.../hadoop/hive/ql/parse/CalcitePlanner.java | 11 +-
.../hive/ql/txn/compactor/CompactorMR.java | 8 +-
.../hive/ql/udf/generic/GenericUDFHash.java | 11 +-
.../hive/ql/lockmgr/TestDbTxnManager.java | 8 +-
.../calcite/TestCBORuleFiredOnlyOnce.java | 168 +++++++++++++++
.../clientpositive/filter_cond_pushdown.q | 5 +
.../test/queries/clientpositive/json_serde1.q | 36 ++++
.../clientpositive/skewjoin_onesideskew.q | 22 ++
.../clientpositive/filter_cond_pushdown.q.out | 80 +++++++
.../results/clientpositive/json_serde1.q.out | 113 ++++++++++
.../clientpositive/skewjoin_onesideskew.q.out | 212 +++++++++++++++++++
.../objectinspector/ObjectInspectorUtils.java | 13 +-
.../TestObjectInspectorUtils.java | 25 +++
.../apache/hive/spark/client/SparkClient.java | 5 +
.../hive/spark/client/SparkClientImpl.java | 5 +
.../org/apache/hive/spark/client/rpc/Rpc.java | 4 +
.../hive/ptest/execution/JIRAService.java | 96 +++++----
.../hive/ptest/execution/TestJIRAService.java | 89 +++++++-
...RAService.testErrorWithMessages.approved.txt | 20 ++
...ervice.testErrorWithoutMessages.approved.txt | 14 ++
.../TestJIRAService.testFailAdd.approved.txt | 21 ++
.../TestJIRAService.testFailNoAdd.approved.txt | 21 ++
.../TestJIRAService.testSuccessAdd.approved.txt | 16 ++
...estJIRAService.testSuccessNoAdd.approved.txt | 16 ++
.../resources/test-configuration.properties | 2 +
46 files changed, 1358 insertions(+), 192 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/4e53bfd1/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/4e53bfd1/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/4e53bfd1/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
----------------------------------------------------------------------
[05/51] [abbrv] hive git commit: HIVE-11853: LLAP: Fix tests related
to stats difference (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/c453c0ed/ql/src/test/results/clientpositive/llap/vector_leftsemi_mapjoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_leftsemi_mapjoin.q.out b/ql/src/test/results/clientpositive/llap/vector_leftsemi_mapjoin.q.out
new file mode 100644
index 0000000..be68b28
--- /dev/null
+++ b/ql/src/test/results/clientpositive/llap/vector_leftsemi_mapjoin.q.out
@@ -0,0 +1,13973 @@
+PREHOOK: query: -- SORT_QUERY_RESULTS
+
+create table t1 stored as orc as select cast(key as int) key, value from src where key <= 10
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@src
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t1
+POSTHOOK: query: -- SORT_QUERY_RESULTS
+
+create table t1 stored as orc as select cast(key as int) key, value from src where key <= 10
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@src
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t1
+PREHOOK: query: select * from t1 sort by key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t1 sort by key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+10 val_10
+2 val_2
+4 val_4
+5 val_5
+5 val_5
+5 val_5
+8 val_8
+9 val_9
+PREHOOK: query: create table t2 stored as orc as select cast(2*key as int) key, value from t1
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@t1
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t2
+POSTHOOK: query: create table t2 stored as orc as select cast(2*key as int) key, value from t1
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@t1
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t2
+PREHOOK: query: select * from t2 sort by key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t2 sort by key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+10 val_5
+10 val_5
+10 val_5
+16 val_8
+18 val_9
+20 val_10
+4 val_2
+8 val_4
+PREHOOK: query: create table t3 stored as orc as select * from (select * from t1 union all select * from t2) b
+PREHOOK: type: CREATETABLE_AS_SELECT
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t3
+POSTHOOK: query: create table t3 stored as orc as select * from (select * from t1 union all select * from t2) b
+POSTHOOK: type: CREATETABLE_AS_SELECT
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t3
+PREHOOK: query: select * from t3 sort by key, value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t3 sort by key, value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+10 val_10
+10 val_5
+10 val_5
+10 val_5
+16 val_8
+18 val_9
+2 val_2
+20 val_10
+4 val_2
+4 val_4
+5 val_5
+5 val_5
+5 val_5
+8 val_4
+8 val_8
+9 val_9
+PREHOOK: query: analyze table t3 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t3
+PREHOOK: Output: default@t3
+POSTHOOK: query: analyze table t3 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t3
+POSTHOOK: Output: default@t3
+PREHOOK: query: create table t4 (key int, value string) stored as orc
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@t4
+POSTHOOK: query: create table t4 (key int, value string) stored as orc
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@t4
+PREHOOK: query: select * from t4
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t4
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t4
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t4
+#### A masked pattern was here ####
+PREHOOK: query: explain select * from t1 a left semi join t2 b on a.key=b.key sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t1 a left semi join t2 b on a.key=b.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t1 a left semi join t2 b on a.key=b.key sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t1 a left semi join t2 b on a.key=b.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+10 val_10
+4 val_4
+8 val_8
+PREHOOK: query: explain select * from t2 a left semi join t1 b on b.key=a.key sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t2 a left semi join t1 b on b.key=a.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t2 a left semi join t1 b on b.key=a.key sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t2 a left semi join t1 b on b.key=a.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+10 val_5
+10 val_5
+10 val_5
+4 val_2
+8 val_4
+PREHOOK: query: explain select * from t1 a left semi join t4 b on b.key=a.key sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t1 a left semi join t4 b on b.key=a.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t1 a left semi join t4 b on b.key=a.key sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t4
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t1 a left semi join t4 b on b.key=a.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t4
+#### A masked pattern was here ####
+PREHOOK: query: explain select a.value from t1 a left semi join t3 b on (b.key = a.key and b.key < '15') sort by a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.value from t1 a left semi join t3 b on (b.key = a.key and b.key < '15') sort by a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col1 (type: int)
+ outputColumnNames: _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key < 15) (type: boolean)
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int), key (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col1 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col1 (type: int)
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.value from t1 a left semi join t3 b on (b.key = a.key and b.key < '15') sort by a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select a.value from t1 a left semi join t3 b on (b.key = a.key and b.key < '15') sort by a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+val_0
+val_0
+val_0
+val_10
+val_2
+val_4
+val_5
+val_5
+val_5
+val_8
+val_9
+PREHOOK: query: explain select * from t1 a left semi join t2 b on a.key = b.key and b.value < "val_10" sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t1 a left semi join t2 b on a.key = b.key and b.value < "val_10" sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((value < 'val_10') and key is not null) (type: boolean)
+ Statistics: Num rows: 2 Data size: 186 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int), value (type: string)
+ outputColumnNames: key, value
+ Statistics: Num rows: 2 Data size: 186 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 186 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 2 Data size: 186 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t1 a left semi join t2 b on a.key = b.key and b.value < "val_10" sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t1 a left semi join t2 b on a.key = b.key and b.value < "val_10" sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+PREHOOK: query: explain select a.value from t1 a left semi join (select key from t3 where key > 5) b on a.key = b.key sort by a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.value from t1 a left semi join (select key from t3 where key > 5) b on a.key = b.key sort by a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 2 <- Map 1 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: t3
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key > 5) (type: boolean)
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 7 Data size: 651 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 2
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col1
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 3
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 7 Data size: 716 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.value from t1 a left semi join (select key from t3 where key > 5) b on a.key = b.key sort by a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select a.value from t1 a left semi join (select key from t3 where key > 5) b on a.key = b.key sort by a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+val_10
+val_8
+val_9
+PREHOOK: query: explain select a.value from t1 a left semi join (select key , value from t2 where key > 5) b on a.key = b.key and b.value <= 'val_20' sort by a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.value from t1 a left semi join (select key , value from t2 where key > 5) b on a.key = b.key and b.value <= 'val_20' sort by a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 2 <- Map 1 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: t2
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (((key > 5) and (value <= 'val_20')) and key is not null) (type: boolean)
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: int), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 2
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col1
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 3
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.value from t1 a left semi join (select key , value from t2 where key > 5) b on a.key = b.key and b.value <= 'val_20' sort by a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select a.value from t1 a left semi join (select key , value from t2 where key > 5) b on a.key = b.key and b.value <= 'val_20' sort by a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+PREHOOK: query: explain select * from t2 a left semi join (select key , value from t1 where key > 2) b on a.key = b.key sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t2 a left semi join (select key , value from t1 where key > 2) b on a.key = b.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 2 <- Map 1 (BROADCAST_EDGE)
+ Reducer 3 <- Map 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: t1
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key > 2) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 2
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 3
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t2 a left semi join (select key , value from t1 where key > 2) b on a.key = b.key sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t2 a left semi join (select key , value from t1 where key > 2) b on a.key = b.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+10 val_5
+10 val_5
+10 val_5
+4 val_2
+8 val_4
+PREHOOK: query: explain select /*+ mapjoin(b) */ a.key from t3 a left semi join t1 b on a.key = b.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select /*+ mapjoin(b) */ a.key from t3 a left semi join t1 b on a.key = b.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 12 Data size: 1125 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 12 Data size: 1125 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 12 Data size: 1125 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 12 Data size: 1125 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select /*+ mapjoin(b) */ a.key from t3 a left semi join t1 b on a.key = b.key sort by a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select /*+ mapjoin(b) */ a.key from t3 a left semi join t1 b on a.key = b.key sort by a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0
+0
+0
+0
+0
+0
+10
+10
+10
+10
+2
+4
+4
+5
+5
+5
+8
+8
+9
+PREHOOK: query: explain select * from t1 a left semi join t2 b on a.key = 2*b.key sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t1 a left semi join t2 b on a.key = 2*b.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int)
+ 1 (2 * _col0) (type: int)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (2 * key) is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: (2 * _col0) (type: int)
+ sort order: +
+ Map-reduce partition columns: (2 * _col0) (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t1 a left semi join t2 b on a.key = 2*b.key sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t1 a left semi join t2 b on a.key = 2*b.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+8 val_8
+PREHOOK: query: explain select * from t1 a join t2 b on a.key = b.key left semi join t3 c on b.key = c.key sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t1 a join t2 b on a.key = b.key left semi join t3 c on b.key = c.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ Left Semi Join 1 to 2
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ 2 _col0 (type: int)
+ outputColumnNames: _col0, _col1, _col5, _col6
+ input vertices:
+ 1 Map 3
+ 2 Map 4
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: int), _col3 (type: string)
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ value expressions: value (type: string)
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string), VALUE._col0 (type: int), VALUE._col1 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t1 a join t2 b on a.key = b.key left semi join t3 c on b.key = c.key sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t1 a join t2 b on a.key = b.key left semi join t3 c on b.key = c.key sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+0 val_0 0 val_0
+10 val_10 10 val_5
+10 val_10 10 val_5
+10 val_10 10 val_5
+4 val_4 4 val_2
+8 val_8 8 val_4
+PREHOOK: query: explain select * from t3 a left semi join t1 b on a.key = b.key and a.value=b.value sort by a.key, a.value
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select * from t3 a left semi join t1 b on a.key = b.key and a.value=b.value sort by a.key, a.value
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 key (type: int), value (type: string)
+ 1 _col0 (type: int), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 3
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int), value (type: string)
+ outputColumnNames: key, value
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
+ Statistics: Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 6 Data size: 613 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select * from t3 a left semi join t1 b on a.key = b.key and a.value=b.value sort by a.key, a.value
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select * from t3 a left semi join t1 b on a.key = b.key and a.value=b.value sort by a.key, a.value
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+0 val_0
+10 val_10
+2 val_2
+4 val_4
+5 val_5
+5 val_5
+5 val_5
+8 val_8
+9 val_9
+PREHOOK: query: explain select /*+ mapjoin(b, c) */ a.key from t3 a left semi join t1 b on a.key = b.key left semi join t2 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select /*+ mapjoin(b, c) */ a.key from t3 a left semi join t1 b on a.key = b.key left semi join t2 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ Left Semi Join 0 to 2
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ 2 _col0 (type: int)
+ outputColumnNames: _col0
+ input vertices:
+ 1 Map 3
+ 2 Map 4
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: key is not null (type: boolean)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 558 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 24 Data size: 2250 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select /*+ mapjoin(b, c) */ a.key from t3 a left semi join t1 b on a.key = b.key left semi join t2 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select /*+ mapjoin(b, c) */ a.key from t3 a left semi join t1 b on a.key = b.key left semi join t2 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0
+0
+0
+0
+0
+0
+10
+10
+10
+10
+4
+4
+8
+8
+PREHOOK: query: explain select a.key from t3 a left outer join t1 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.key from t3 a left outer join t1 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Map 1 <- Map 3 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Outer Join0 to 1
+ Left Semi Join 1 to 2
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ 2 _col0 (type: int)
+ outputColumnNames: _col0
+ input vertices:
+ 1 Map 3
+ 2 Map 4
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 3
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.key from t3 a left outer join t1 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select a.key from t3 a left outer join t1 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+10
+10
+10
+10
+4
+4
+8
+8
+PREHOOK: query: explain select a.key from t1 a full outer join t3 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.key from t1 a full outer join t3 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Outer Join 0 to 1
+ Left Semi Join 1 to 2
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ 2 _col0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reducer 3
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.key from t1 a full outer join t3 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select a.key from t1 a full outer join t3 b on a.key = b.key left semi join t2 c on b.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+10
+10
+10
+10
+4
+4
+8
+8
+NULL
+NULL
+NULL
+PREHOOK: query: explain select a.key from t3 a left semi join t2 b on a.key = b.key left outer join t1 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.key from t3 a left semi join t2 b on a.key = b.key left outer join t1 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ Left Outer Join0 to 2
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ 2 key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reducer 3
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.key from t3 a left semi join t2 b on a.key = b.key left outer join t1 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select a.key from t3 a left semi join t2 b on a.key = b.key left outer join t1 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+10
+10
+10
+10
+16
+18
+20
+4
+4
+8
+8
+PREHOOK: query: explain select a.key from t3 a left semi join t2 b on a.key = b.key right outer join t1 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.key from t3 a left semi join t2 b on a.key = b.key right outer join t1 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ Right Outer Join0 to 2
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ 2 key (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ Reducer 3
+ Execution mode: uber
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 48 Data size: 4501 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+PREHOOK: query: select a.key from t3 a left semi join t2 b on a.key = b.key right outer join t1 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+PREHOOK: Input: default@t1
+PREHOOK: Input: default@t2
+PREHOOK: Input: default@t3
+#### A masked pattern was here ####
+POSTHOOK: query: select a.key from t3 a left semi join t2 b on a.key = b.key right outer join t1 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@t1
+POSTHOOK: Input: default@t2
+POSTHOOK: Input: default@t3
+#### A masked pattern was here ####
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+0
+10
+10
+10
+10
+4
+4
+8
+8
+NULL
+NULL
+NULL
+NULL
+NULL
+PREHOOK: query: explain select a.key from t3 a left semi join t1 b on a.key = b.key full outer join t2 c on a.key = c.key sort by a.key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select a.key from t3 a left semi join t1 b on a.key = b.key full outer join t2 c on a.key = c.key sort by a.key
+POSTHOOK: type: QUERY
+STAGE DEPENDENCIES:
+ Stage-1 is a root stage
+ Stage-0 depends on stages: Stage-1
+
+STAGE PLANS:
+ Stage: Stage-1
+ Tez
+ Edges:
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 22 Data size: 2046 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: int)
+ outputColumnNames: key
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: int)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: c
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: key (type: int)
+ sort order: +
+ Map-reduce partition columns: key (type: int)
+ Statistics: Num rows: 11 Data size: 1023 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: llap
+ Reducer 2
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ Outer Join 0 to 2
+ keys:
+ 0 key (type: int)
+ 1 _col0 (type: int)
+ 2 key (type: int)
+ outputCol
<TRUNCATED>
[40/51] [abbrv] hive git commit: HIVE-12071 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12071 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/be00a031
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/be00a031
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/be00a031
Branch: refs/heads/master
Commit: be00a031da2d55ce8a6bf7280e2a797eb6741cd2
Parents: 7a23913 aded0d3
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 8 12:38:27 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 8 12:38:27 2015 -0700
----------------------------------------------------------------------
.../java/org/apache/hive/beeline/BeeLine.java | 4 +
.../org/apache/hive/beeline/BeeLineOpts.java | 9 +
.../hive/beeline/TestBeelineArgParsing.java | 1 +
.../org/apache/hadoop/hive/cli/CliDriver.java | 9 +-
.../org/apache/hadoop/hive/conf/HiveConf.java | 3 +
.../apache/hadoop/hive/ql/log/PerfLogger.java | 2 +-
.../apache/hadoop/hive/ql/exec/ColumnInfo.java | 2 +-
.../hive/ql/exec/tez/TezSessionState.java | 203 +++++++++++++++----
.../apache/hadoop/hive/ql/exec/tez/TezTask.java | 6 +-
.../ql/io/avro/AvroGenericRecordReader.java | 22 +-
.../ql/metadata/SessionHiveMetaStoreClient.java | 2 +-
.../hadoop/hive/ql/parse/CalcitePlanner.java | 3 +-
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 34 +++-
.../hadoop/hive/ql/ppd/OpProcFactory.java | 10 +-
.../hadoop/hive/ql/session/SessionState.java | 60 ++++--
.../hive/ql/udf/generic/GenericUDFUtils.java | 3 +-
.../hadoop/hive/ql/exec/tez/TestTezTask.java | 2 +
.../ql/io/avro/TestAvroGenericRecordReader.java | 59 ++++++
ql/src/test/queries/clientpositive/join44.q | 12 ++
ql/src/test/queries/clientpositive/temp_table.q | 26 +++
ql/src/test/queries/clientpositive/union36.q | 10 +
.../clientpositive/dynamic_rdd_cache.q.out | 28 +--
ql/src/test/results/clientpositive/join44.q.out | 88 ++++++++
.../join_cond_pushdown_unqual1.q.out | 12 +-
.../join_cond_pushdown_unqual3.q.out | 18 +-
.../join_cond_pushdown_unqual4.q.out | 2 +-
.../results/clientpositive/pointlookup2.q.out | 12 +-
.../results/clientpositive/pointlookup3.q.out | 12 +-
.../spark/dynamic_rdd_cache.q.out | 28 +--
.../spark/join_cond_pushdown_unqual1.q.out | 12 +-
.../spark/join_cond_pushdown_unqual3.q.out | 18 +-
.../spark/join_cond_pushdown_unqual4.q.out | 2 +-
.../clientpositive/spark/temp_table.q.out | 107 ++++++++++
.../results/clientpositive/temp_table.q.out | 107 ++++++++++
.../tez/dynamic_partition_pruning.q.out | 4 +-
.../tez/dynamic_partition_pruning_2.q.out | 54 ++---
.../clientpositive/tez/explainuser_1.q.out | 96 ++++-----
.../results/clientpositive/tez/temp_table.q.out | 107 ++++++++++
.../tez/vector_mr_diff_schema_alias.q.out | 2 +-
.../vectorized_dynamic_partition_pruning.q.out | 4 +-
.../test/results/clientpositive/union36.q.out | 28 +++
.../vector_mr_diff_schema_alias.q.out | 2 +-
42 files changed, 986 insertions(+), 239 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/be00a031/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/be00a031/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
index 41a742c,6ed6421..e034c71
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java
@@@ -201,9 -211,7 +251,9 @@@ public class TezSessionState
// and finally we're ready to create and start the session
// generate basic tez config
- TezConfiguration tezConfig = new TezConfiguration(conf);
+ final TezConfiguration tezConfig = new TezConfiguration(conf);
+
+ // set up the staging directory to use
tezConfig.set(TezConfiguration.TEZ_AM_STAGING_DIR, tezScratchDir.toUri().toString());
Utilities.stripHivePasswordDetails(tezConfig);
@@@ -235,9 -223,8 +285,9 @@@
tezConfig.setInt(TezConfiguration.TEZ_AM_SESSION_MIN_HELD_CONTAINERS, n);
}
- session = TezClient.newBuilder("HIVE-" + sessionId, tezConfig).setIsSession(true)
- .setLocalResources(commonLocalResources)
- final TezClient session = TezClient.create("HIVE-" + sessionId, tezConfig, true,
- commonLocalResources, null);
++ final TezClient session = TezClient.newBuilder("HIVE-" + sessionId, tezConfig)
++ .setIsSession(true).setLocalResources(commonLocalResources)
+ .setServicePluginDescriptor(servicePluginsDescriptor).build();
LOG.info("Opening new Tez Session (id: " + sessionId
+ ", scratch dir: " + tezScratchDir + ")");
http://git-wip-us.apache.org/repos/asf/hive/blob/be00a031/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/be00a031/ql/src/test/results/clientpositive/tez/vector_mr_diff_schema_alias.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/be00a031/ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
----------------------------------------------------------------------
[21/51] [abbrv] hive git commit: HIVE-11967: LLAP: Merge master to
branch (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-11967: LLAP: Merge master to branch (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/3c5b4ced
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/3c5b4ced
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/3c5b4ced
Branch: refs/heads/master
Commit: 3c5b4cedaa76cd40e177d60a47414d2db150ed0e
Parents: 3b64bd6 6c2d71c
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Fri Sep 25 13:38:23 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Fri Sep 25 13:38:23 2015 -0500
----------------------------------------------------------------------
NOTICE | 3 +
.../apache/hadoop/hive/ant/GenVectorCode.java | 31 -
bin/hive | 2 +-
.../org/apache/hadoop/hive/conf/HiveConf.java | 2 +-
.../vectorization/VectorizationBench.java | 32 +-
.../org/apache/hive/jdbc/TestJdbcDriver2.java | 80 +-
.../test/resources/testconfiguration.properties | 1 +
.../apache/hive/jdbc/HivePreparedStatement.java | 4 +-
.../hive/jdbc/ZooKeeperHiveClientHelper.java | 2 +-
.../oracle/hive-schema-0.13.0.oracle.sql | 10 +-
.../oracle/hive-schema-0.14.0.oracle.sql | 10 +-
.../oracle/hive-txn-schema-0.13.0.oracle.sql | 10 +-
.../oracle/hive-txn-schema-0.14.0.oracle.sql | 10 +-
.../hadoop/hive/metastore/ObjectStore.java | 16 +-
.../hadoop/hive/metastore/txn/TxnDbUtil.java | 4 +-
.../hadoop/hive/metastore/txn/TxnHandler.java | 5 +-
packaging/src/main/assembly/bin.xml | 1 +
.../ExpressionTemplates/IfExprColumnColumn.txt | 186 --
.../org/apache/hadoop/hive/ql/ErrorMsg.java | 2 +-
.../hadoop/hive/ql/exec/FunctionRegistry.java | 1 +
.../ql/exec/vector/VectorGroupByOperator.java | 5 +-
.../exec/vector/VectorSMBMapJoinOperator.java | 15 +-
.../ql/exec/vector/VectorizationContext.java | 203 +-
.../expressions/FilterStringColumnInList.java | 13 +-
.../expressions/FilterStructColumnInList.java | 178 ++
.../exec/vector/expressions/IStructInExpr.java | 36 +
.../IfExprDoubleColumnDoubleColumn.java | 167 ++
.../expressions/IfExprLongColumnLongColumn.java | 166 ++
.../vector/expressions/StringColumnInList.java | 4 +
.../vector/expressions/StructColumnInList.java | 174 ++
.../hadoop/hive/ql/io/orc/OrcInputFormat.java | 6 +-
.../functions/HiveSqlCountAggFunction.java | 72 +
.../functions/HiveSqlMinMaxAggFunction.java | 49 +
.../functions/HiveSqlSumAggFunction.java | 125 ++
.../rules/HiveAggregateJoinTransposeRule.java | 372 ++++
.../translator/SqlFunctionConverter.java | 40 +-
.../hive/ql/optimizer/physical/Vectorizer.java | 86 +-
.../ql/optimizer/physical/Vectorizer.java.orig | 1744 +++++++++++++++++
.../ql/optimizer/physical/Vectorizer.java.rej | 86 +
.../hive/ql/optimizer/ppr/PartitionPruner.java | 26 +-
.../hadoop/hive/ql/parse/CalcitePlanner.java | 5 +
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 8 +-
.../hadoop/hive/ql/plan/VectorGroupByDesc.java | 10 +
.../hadoop/hive/ql/session/SessionState.java | 3 +
.../apache/hadoop/hive/ql/stats/StatsUtils.java | 182 +-
.../hive/ql/stats/jdbc/JDBCStatsPublisher.java | 13 +-
.../ql/stats/jdbc/JDBCStatsSetupConstants.java | 4 +-
.../hive/ql/udf/generic/GenericUDAFSum.java | 2 +-
.../udf/generic/GenericUDAFSumEmptyIsZero.java | 63 +
.../hive/ql/udf/generic/GenericUDFIf.java | 4 +-
.../exec/vector/TestVectorizationContext.java | 4 +-
.../TestVectorConditionalExpressions.java | 3 +-
.../hive/ql/io/orc/TestInputOutputFormat.java | 19 +
.../TestNegativePartitionPrunerCompactExpr.java | 27 +
.../TestPositivePartitionPrunerCompactExpr.java | 115 ++
.../test/queries/clientnegative/ctasnullcol.q | 2 +
.../queries/clientpositive/avrocountemptytbl.q | 8 +
.../clientpositive/groupby_join_pushdown.q | 55 +
.../clientpositive/vector_auto_smb_mapjoin_14.q | 297 +++
.../clientpositive/vector_groupby_reduce.q | 62 +-
.../queries/clientpositive/vector_struct_in.q | 247 +++
.../clientpositive/vectorization_limit.q | 4 +-
.../results/clientnegative/ctasnullcol.q.out | 5 +
.../clientpositive/avrocountemptytbl.q.out | 58 +
.../clientpositive/groupby_join_pushdown.q.out | 1522 +++++++++++++++
.../results/clientpositive/show_functions.q.out | 1 +
.../tez/vector_auto_smb_mapjoin_14.q.out | 1576 +++++++++++++++
.../tez/vector_groupby_reduce.q.out | 1452 +++++++++++++-
.../tez/vectorization_limit.q.out | 8 +-
.../vector_auto_smb_mapjoin_14.q.out | 1792 ++++++++++++++++++
.../clientpositive/vector_groupby_reduce.q.out | 1466 +++++++++++++-
.../clientpositive/vector_struct_in.q.out | 825 ++++++++
.../clientpositive/vectorization_limit.q.out | 8 +-
.../hadoop/hive/serde2/avro/AvroSerdeUtils.java | 24 +-
shims/0.23/pom.xml | 1 -
.../apache/hadoop/hive/shims/Hadoop23Shims.java | 23 +-
76 files changed, 13371 insertions(+), 506 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/packaging/src/main/assembly/bin.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorGroupByOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/ql/src/test/results/clientpositive/tez/vectorization_limit.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/3c5b4ced/shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
----------------------------------------------------------------------
[07/51] [abbrv] hive git commit: HIVE-11888: LLAP: Merge master into
branch (for HIVE-11860) (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-11888: LLAP: Merge master into branch (for HIVE-11860) (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/1373ce71
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/1373ce71
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/1373ce71
Branch: refs/heads/master
Commit: 1373ce714292b1efc37a820eb24827c2f9095b53
Parents: c453c0e ae588db
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Fri Sep 18 17:25:15 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Fri Sep 18 17:25:15 2015 -0500
----------------------------------------------------------------------
bin/beeline | 5 +
.../test/resources/testconfiguration.properties | 1 +
.../hadoop/hive/metastore/txn/TxnHandler.java | 61 ++-
.../hadoop/hive/ql/exec/StatsNoJobTask.java | 25 +-
.../apache/hadoop/hive/ql/exec/tez/TezTask.java | 14 +-
.../queries/clientpositive/union_fast_stats.q | 68 +++
.../clientpositive/tez/union_fast_stats.q.out | 526 +++++++++++++++++++
.../clientpositive/union_fast_stats.q.out | 526 +++++++++++++++++++
8 files changed, 1181 insertions(+), 45 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/1373ce71/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/1373ce71/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java
----------------------------------------------------------------------
[16/51] [abbrv] hive git commit: HIVE-11905: LLAP: Update golden
files for orc_llap and llapdecider tests (Prasanth Jayachandran)
Posted by se...@apache.org.
HIVE-11905: LLAP: Update golden files for orc_llap and llapdecider tests (Prasanth Jayachandran)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/683fb689
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/683fb689
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/683fb689
Branch: refs/heads/master
Commit: 683fb689c03b6627b2e1a63b6426aeffd27c644a
Parents: b6ad7a2
Author: Prasanth Jayachandran <j....@gmail.com>
Authored: Mon Sep 21 14:48:37 2015 -0500
Committer: Prasanth Jayachandran <j....@gmail.com>
Committed: Mon Sep 21 14:48:37 2015 -0500
----------------------------------------------------------------------
.../clientpositive/llap/llapdecider.q.out | 24 ++++++++++----------
.../results/clientpositive/llap/orc_llap.q.out | 8 +++----
.../test/results/clientpositive/orc_llap.q.out | 8 +++----
.../clientpositive/tez/llapdecider.q.out | 24 ++++++++++----------
4 files changed, 32 insertions(+), 32 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/683fb689/ql/src/test/results/clientpositive/llap/llapdecider.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/llapdecider.q.out b/ql/src/test/results/clientpositive/llap/llapdecider.q.out
index 6aa5513..676a0e4 100644
--- a/ql/src/test/results/clientpositive/llap/llapdecider.q.out
+++ b/ql/src/test/results/clientpositive/llap/llapdecider.q.out
@@ -23,11 +23,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 205 Data size: 19475 Basic stats: COMPLETE Column stats: COMPLETE
@@ -103,11 +103,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
@@ -173,11 +173,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
@@ -254,11 +254,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 205 Data size: 19475 Basic stats: COMPLETE Column stats: COMPLETE
http://git-wip-us.apache.org/repos/asf/hive/blob/683fb689/ql/src/test/results/clientpositive/llap/orc_llap.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/orc_llap.q.out b/ql/src/test/results/clientpositive/llap/orc_llap.q.out
index 870c6e6..e62fd92 100644
--- a/ql/src/test/results/clientpositive/llap/orc_llap.q.out
+++ b/ql/src/test/results/clientpositive/llap/orc_llap.q.out
@@ -425,11 +425,11 @@ STAGE PLANS:
Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cstring1 (type: string), cstring2 (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: cstring1, cstring2
Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: string), _col1 (type: string)
+ keys: cstring1 (type: string), cstring2 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
@@ -816,11 +816,11 @@ STAGE PLANS:
Statistics: Num rows: 8014 Data size: 1602939 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cstring1 (type: string), cstring2 (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: cstring1, cstring2
Statistics: Num rows: 8014 Data size: 1602939 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: string), _col1 (type: string)
+ keys: cstring1 (type: string), cstring2 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 8014 Data size: 1602939 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/683fb689/ql/src/test/results/clientpositive/orc_llap.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/orc_llap.q.out b/ql/src/test/results/clientpositive/orc_llap.q.out
index eb83b41..b93771f 100644
--- a/ql/src/test/results/clientpositive/orc_llap.q.out
+++ b/ql/src/test/results/clientpositive/orc_llap.q.out
@@ -469,11 +469,11 @@ STAGE PLANS:
Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cstring1 (type: string), cstring2 (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: cstring1, cstring2
Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: string), _col1 (type: string)
+ keys: cstring1 (type: string), cstring2 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 122880 Data size: 29079940 Basic stats: COMPLETE Column stats: NONE
@@ -902,11 +902,11 @@ STAGE PLANS:
Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cstring1 (type: string), cstring2 (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: cstring1, cstring2
Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
- keys: _col0 (type: string), _col1 (type: string)
+ keys: cstring1 (type: string), cstring2 (type: string)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 7966 Data size: 1593293 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/683fb689/ql/src/test/results/clientpositive/tez/llapdecider.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/llapdecider.q.out b/ql/src/test/results/clientpositive/tez/llapdecider.q.out
index 6aa5513..676a0e4 100644
--- a/ql/src/test/results/clientpositive/tez/llapdecider.q.out
+++ b/ql/src/test/results/clientpositive/tez/llapdecider.q.out
@@ -23,11 +23,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 205 Data size: 19475 Basic stats: COMPLETE Column stats: COMPLETE
@@ -103,11 +103,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
@@ -173,11 +173,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: NONE
@@ -254,11 +254,11 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ outputColumnNames: key, value
Statistics: Num rows: 500 Data size: 88000 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: count(_col1)
- keys: _col0 (type: string)
+ aggregations: count(value)
+ keys: key (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 205 Data size: 19475 Basic stats: COMPLETE Column stats: COMPLETE
[39/51] [abbrv] hive git commit: HIVE-12042 : LLAP: update some out
files (Sergey Shelukhin) ADDENDUM
Posted by se...@apache.org.
HIVE-12042 : LLAP: update some out files (Sergey Shelukhin) ADDENDUM
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/7a239137
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/7a239137
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/7a239137
Branch: refs/heads/master
Commit: 7a2391376e8f8aeefa78a2e0a1d0293809c80df5
Parents: b861b9e
Author: Sergey Shelukhin <se...@apache.org>
Authored: Wed Oct 7 16:35:13 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Wed Oct 7 16:35:13 2015 -0700
----------------------------------------------------------------------
.../clientpositive/llap/explainuser_1.q.out | 76 ++++++++++----------
1 file changed, 38 insertions(+), 38 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/7a239137/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 789180a..3ca2f50 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -1986,7 +1986,7 @@ Stage-0
Statistics:Num rows: 4 Data size: 728 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col1 (type: int), _col2 (type: float), _col3 (type: string), _col4 (type: int)
Filter Operator [FIL_27]
- predicate:((((_col1 + _col4) = 2) and ((_col4 + 1) = 2)) and _col0 is not null) (type: boolean)
+ predicate:((((_col1 + _col4) = 2) and _col0 is not null) and ((_col4 + 1) = 2)) (type: boolean)
Statistics:Num rows: 4 Data size: 728 Basic stats: COMPLETE Column stats: COMPLETE
Merge Join Operator [MERGEJOIN_31]
| condition map:[{"":"Outer Join 0 to 1"}]
@@ -2797,7 +2797,7 @@ Stage-0
| outputColumnNames:["_col0","_col1","_col2","_col3"]
| Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
| Filter Operator [FIL_61]
- | predicate:(((UDFToDouble(_col0) >= 1.0) or (_col3 >= 1)) and ((UDFToDouble(_col0) + UDFToDouble(_col3)) >= 0.0)) (type: boolean)
+ | predicate:(((UDFToDouble(_col0) + UDFToDouble(_col3)) >= 0.0) and ((UDFToDouble(_col0) >= 1.0) or (_col3 >= 1))) (type: boolean)
| Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
| Select Operator [SEL_65]
| outputColumnNames:["_col0","_col1","_col3"]
@@ -4240,7 +4240,7 @@ Stage-0
Select Operator [SEL_13]
Statistics:Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator [FIL_45]
- predicate:(((_col2 - _col1) > 600.0) and (_col0 is null or _col1 is null)) (type: boolean)
+ predicate:((_col0 is null or _col1 is null) and ((_col2 - _col1) > 600.0)) (type: boolean)
Statistics:Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator [GBY_11]
| aggregations:["min(VALUE._col0)","max(VALUE._col1)"]
@@ -6031,36 +6031,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key AND a.value <=> b.value
PREHOOK: type: QUERY
@@ -6078,34 +6078,34 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int), value (type: int)","1":"key (type: int), value (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int), value (type: int)
| Map-reduce partition columns:key (type: int), value (type: int)
| sort order:++
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int), value (type: int)
Map-reduce partition columns:key (type: int), value (type: int)
sort order:++
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a RIGHT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6123,36 +6123,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Right Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6170,36 +6170,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a LEFT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6217,36 +6217,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Left Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: drop table sales
PREHOOK: type: DROPTABLE
[03/51] [abbrv] hive git commit: HIVE-11886 : LLAP: merge master into
branch (Sergey Shelukhin) ADDENDUM SPARK TESTS
Posted by se...@apache.org.
HIVE-11886 : LLAP: merge master into branch (Sergey Shelukhin) ADDENDUM SPARK TESTS
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/4fa4e40e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/4fa4e40e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/4fa4e40e
Branch: refs/heads/master
Commit: 4fa4e40e5d3d76633b2a1da1435cb6a15cbc4721
Parents: f324305
Author: Sergey Shelukhin <se...@apache.org>
Authored: Fri Sep 18 15:06:51 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Fri Sep 18 15:06:51 2015 -0700
----------------------------------------------------------------------
.../results/clientpositive/spark/vector_distinct_2.q.out | 8 --------
.../results/clientpositive/spark/vector_groupby_3.q.out | 8 --------
.../clientpositive/spark/vectorized_nested_mapjoin.q.out | 9 +--------
3 files changed, 1 insertion(+), 24 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/4fa4e40e/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out b/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
index 563213a..e2143ad 100644
--- a/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
+++ b/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
@@ -154,12 +154,6 @@ STAGE PLANS:
expressions: _col1 (type: string), _col0 (type: tinyint)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
-<<<<<<< HEAD
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
-=======
File Output Operator
compressed: false
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
@@ -167,8 +161,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
->>>>>>> master
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/4fa4e40e/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out b/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
index b2402db..47a3846 100644
--- a/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
+++ b/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
@@ -157,12 +157,6 @@ STAGE PLANS:
expressions: _col1 (type: string), _col0 (type: tinyint), _col2 (type: bigint)
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
-<<<<<<< HEAD
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
-=======
File Output Operator
compressed: false
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
@@ -170,8 +164,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.TextInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
->>>>>>> master
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/4fa4e40e/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out b/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
index bf23ae4..6aabab6 100644
--- a/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
@@ -105,15 +105,8 @@ STAGE PLANS:
Reduce Output Operator
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
-<<<<<<< HEAD
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: double)
- Execution mode: vectorized
-=======
value expressions: _col0 (type: double)
->>>>>>> master
+ Execution mode: vectorized
Local Work:
Map Reduce Local Work
Reducer 3
[13/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 36f1099..7d9d99e 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -398,28 +398,28 @@ Stage-0
compressed:false
Statistics:Num rows: 10 Data size: 885 Basic stats: COMPLETE Column stats: COMPLETE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
- Select Operator [SEL_6]
+ Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 10 Data size: 885 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_5]
+ Group By Operator [GBY_4]
| aggregations:["sum(VALUE._col0)"]
- | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
| outputColumnNames:["_col0","_col1","_col2","_col3"]
| Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
|<-Map 1 [SIMPLE_EDGE]
- Reduce Output Operator [RS_4]
- key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
+ Reduce Output Operator [RS_3]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
sort order:+++
Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col3 (type: bigint)
- Group By Operator [GBY_3]
- aggregations:["sum(_col1)"]
- keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
+ Group By Operator [GBY_2]
+ aggregations:["sum(c_int)"]
+ keys:key (type: string), c_int (type: int), c_float (type: float)
outputColumnNames:["_col0","_col1","_col2","_col3"]
Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator [SEL_1]
- outputColumnNames:["_col0","_col1","_col2"]
+ outputColumnNames:["key","c_int","c_float"]
Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_0]
alias:cbo_t1
@@ -464,28 +464,28 @@ Stage-0
keys:_col0 (type: bigint), _col1 (type: float)
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 5 Data size: 100 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_6]
+ Select Operator [SEL_5]
outputColumnNames:["_col0","_col1"]
Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_5]
+ Group By Operator [GBY_4]
| aggregations:["sum(VALUE._col0)"]
- | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
| outputColumnNames:["_col0","_col1","_col2","_col3"]
| Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
|<-Map 1 [SIMPLE_EDGE]
- Reduce Output Operator [RS_4]
- key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
+ Reduce Output Operator [RS_3]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
sort order:+++
Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col3 (type: bigint)
- Group By Operator [GBY_3]
- aggregations:["sum(_col1)"]
- keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
+ Group By Operator [GBY_2]
+ aggregations:["sum(c_int)"]
+ keys:key (type: string), c_int (type: int), c_float (type: float)
outputColumnNames:["_col0","_col1","_col2","_col3"]
Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator [SEL_1]
- outputColumnNames:["_col0","_col1","_col2"]
+ outputColumnNames:["key","c_int","c_float"]
Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
TableScan [TS_0]
alias:cbo_t1
@@ -525,152 +525,149 @@ Stage-0
sort order:-+
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col0 (type: int), _col2 (type: bigint)
- Group By Operator [GBY_41]
- | aggregations:["count(VALUE._col0)"]
- | keys:KEY._col0 (type: int), KEY._col1 (type: bigint)
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 5 [SIMPLE_EDGE]
- Reduce Output Operator [RS_40]
- key expressions:_col0 (type: int), _col1 (type: bigint)
- Map-reduce partition columns:_col0 (type: int), _col1 (type: bigint)
- sort order:++
- Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col2 (type: bigint)
- Group By Operator [GBY_39]
- aggregations:["count()"]
- keys:_col0 (type: int), _col1 (type: bigint)
- outputColumnNames:["_col0","_col1","_col2"]
+ Select Operator [SEL_41]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_40]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 5 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_39]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_37]
- outputColumnNames:["_col0","_col1"]
- Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_36]
- predicate:((_col1 > 0) or (_col6 >= 0)) (type: boolean)
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_38]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_37]
+ outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_55]
- | condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
- | outputColumnNames:["_col1","_col2","_col6"]
- | Statistics:Num rows: 3 Data size: 48 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Map 11 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_34]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int)
- | Select Operator [SEL_30]
- | outputColumnNames:["_col0","_col1"]
- | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_53]
- | predicate:key is not null (type: boolean)
- | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
- | TableScan [TS_29]
- | alias:cbo_t3
- | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 4 [SIMPLE_EDGE]
- Reduce Output Operator [RS_32]
- key expressions:_col0 (type: string)
- Map-reduce partition columns:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col1 (type: int), _col2 (type: bigint)
- Select Operator [SEL_28]
- outputColumnNames:["_col0","_col1","_col2"]
+ Filter Operator [FIL_36]
+ predicate:((_col1 > 0) or (_col6 >= 0)) (type: boolean)
+ Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_55]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col6"]
+ | Statistics:Num rows: 3 Data size: 48 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 11 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_34]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_30]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_53]
+ | predicate:key is not null (type: boolean)
+ | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_29]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_32]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_27]
- predicate:((_col1 + _col4) >= 0) (type: boolean)
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_28]
+ outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_54]
- | condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
- | outputColumnNames:["_col0","_col1","_col2","_col4"]
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 10 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_25]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int)
- | Select Operator [SEL_20]
- | | outputColumnNames:["_col0","_col1"]
- | | Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
- | |<-Reducer 9 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_19]
- | key expressions:_col3 (type: double), _col2 (type: bigint)
- | sort order:-+
- | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col0 (type: string), _col1 (type: int)
- | Select Operator [SEL_18]
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
- | Group By Operator [GBY_17]
- | | aggregations:["sum(VALUE._col0)"]
- | | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
- | | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | |<-Map 8 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_16]
- | key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | sort order:+++
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col3 (type: bigint)
- | Group By Operator [GBY_15]
- | aggregations:["sum(_col1)"]
- | keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | Select Operator [SEL_13]
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_52]
- | predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
- | Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
- | TableScan [TS_11]
- | alias:cbo_t2
- | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 3 [SIMPLE_EDGE]
- Reduce Output Operator [RS_23]
- key expressions:_col0 (type: string)
- Map-reduce partition columns:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col1 (type: int), _col2 (type: bigint)
- Select Operator [SEL_9]
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 2 [SIMPLE_EDGE]
- Reduce Output Operator [RS_8]
- key expressions:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col1 (type: int), _col2 (type: bigint)
- Select Operator [SEL_7]
- outputColumnNames:["_col0","_col1","_col2"]
+ Filter Operator [FIL_27]
+ predicate:((_col1 + _col4) >= 0) (type: boolean)
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_54]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col0","_col1","_col2","_col4"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 10 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_25]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_20]
+ | | outputColumnNames:["_col0","_col1"]
+ | | Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 9 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_19]
+ | key expressions:_col3 (type: double), _col2 (type: bigint)
+ | sort order:-+
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col0 (type: string), _col1 (type: int)
+ | Select Operator [SEL_17]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_16]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Map 8 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_15]
+ | key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | sort order:+++
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col3 (type: bigint)
+ | Group By Operator [GBY_14]
+ | aggregations:["sum(c_int)"]
+ | keys:key (type: string), c_int (type: int), c_float (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_52]
+ | predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
+ | Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_11]
+ | alias:cbo_t2
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_23]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_9]
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 2 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_8]
+ key expressions:_col0 (type: string)
+ sort order:+
Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_6]
- | aggregations:["sum(VALUE._col0)"]
- | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Map 1 [SIMPLE_EDGE]
- Reduce Output Operator [RS_5]
- key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- sort order:+++
- Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col3 (type: bigint)
- Group By Operator [GBY_4]
- aggregations:["sum(_col1)"]
- keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- outputColumnNames:["_col0","_col1","_col2","_col3"]
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_6]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_5]
+ | aggregations:["sum(VALUE._col0)"]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_4]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_2]
- outputColumnNames:["_col0","_col1","_col2"]
- Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col3 (type: bigint)
+ Group By Operator [GBY_3]
+ aggregations:["sum(c_int)"]
+ keys:key (type: string), c_int (type: int), c_float (type: float)
+ outputColumnNames:["_col0","_col1","_col2","_col3"]
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator [FIL_51]
predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
@@ -711,143 +708,146 @@ Stage-0
sort order:+-
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col1 (type: bigint), _col2 (type: bigint)
- Group By Operator [GBY_40]
- | aggregations:["count(VALUE._col0)"]
- | keys:KEY._col0 (type: int), KEY._col1 (type: bigint)
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 5 [SIMPLE_EDGE]
- Reduce Output Operator [RS_39]
- key expressions:_col0 (type: int), _col1 (type: bigint)
- Map-reduce partition columns:_col0 (type: int), _col1 (type: bigint)
- sort order:++
- Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col2 (type: bigint)
- Group By Operator [GBY_38]
- aggregations:["count()"]
- keys:_col0 (type: int), _col1 (type: bigint)
- outputColumnNames:["_col0","_col1","_col2"]
+ Select Operator [SEL_40]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_39]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 5 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_38]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_34]
- outputColumnNames:["_col0","_col1"]
- Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_48]
- predicate:((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_col6) + _col2) >= 0)) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_37]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_34]
+ outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_53]
- | condition map:[{"":"Left Outer Join0 to 1"}]
- | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
- | outputColumnNames:["_col1","_col2","_col6"]
- | Statistics:Num rows: 4 Data size: 64 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Map 10 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_31]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int)
- | Select Operator [SEL_29]
- | outputColumnNames:["_col0","_col1"]
- | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
- | TableScan [TS_28]
- | alias:cbo_t3
- | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 4 [SIMPLE_EDGE]
- Reduce Output Operator [RS_30]
- key expressions:_col0 (type: string)
- Map-reduce partition columns:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col1 (type: int), _col2 (type: bigint)
- Select Operator [SEL_27]
- outputColumnNames:["_col0","_col1","_col2"]
+ Filter Operator [FIL_48]
+ predicate:((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_col6) + _col2) >= 0)) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_53]
+ | condition map:[{"":"Left Outer Join0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col6"]
+ | Statistics:Num rows: 4 Data size: 64 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 10 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_31]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_29]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_28]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_30]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_26]
- predicate:((_col1 + _col4) >= 0) (type: boolean)
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_27]
+ outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_52]
- | condition map:[{"":"Left Outer Join0 to 1"}]
- | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
- | outputColumnNames:["_col0","_col1","_col2","_col4"]
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 3 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_23]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int), _col2 (type: bigint)
- | Select Operator [SEL_11]
- | | outputColumnNames:["_col0","_col1","_col2"]
- | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- | |<-Reducer 2 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_10]
- | key expressions:_col3 (type: bigint), _col1 (type: int)
- | sort order:+-
- | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col0 (type: string), _col2 (type: bigint)
- | Select Operator [SEL_9]
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
- | Group By Operator [GBY_8]
- | | aggregations:["sum(VALUE._col0)"]
- | | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
- | | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | |<-Map 1 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_7]
- | key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | sort order:+++
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col3 (type: bigint)
- | Group By Operator [GBY_6]
- | aggregations:["sum(_col1)"]
- | keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | Select Operator [SEL_2]
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_49]
- | predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
- | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | TableScan [TS_0]
- | alias:cbo_t1
- | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 9 [SIMPLE_EDGE]
- Reduce Output Operator [RS_24]
- key expressions:_col0 (type: string)
- Map-reduce partition columns:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col1 (type: int)
- Select Operator [SEL_22]
- outputColumnNames:["_col0","_col1"]
+ Filter Operator [FIL_26]
+ predicate:((_col1 + _col4) >= 0) (type: boolean)
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_52]
+ | condition map:[{"":"Left Outer Join0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col0","_col1","_col2","_col4"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_23]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_11]
+ | | outputColumnNames:["_col0","_col1","_col2"]
+ | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 2 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_10]
+ | key expressions:_col3 (type: bigint), _col1 (type: int)
+ | sort order:+-
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col0 (type: string), _col2 (type: bigint)
+ | Select Operator [SEL_8]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_7]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Map 1 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_6]
+ | key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | sort order:+++
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col3 (type: bigint)
+ | Group By Operator [GBY_5]
+ | aggregations:["sum(_col2)"]
+ | keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | Select Operator [SEL_2]
+ | outputColumnNames:["_col0","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_49]
+ | predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_0]
+ | alias:cbo_t1
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 9 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_24]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_21]
- | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Map 8 [SIMPLE_EDGE]
- Reduce Output Operator [RS_20]
- key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- sort order:+++
- Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_19]
- keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- outputColumnNames:["_col0","_col1","_col2"]
+ value expressions:_col1 (type: int)
+ Select Operator [SEL_21]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_20]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 8 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_19]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_15]
+ Group By Operator [GBY_18]
+ keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_50]
- predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ Select Operator [SEL_15]
+ outputColumnNames:["_col0","_col2","_col3"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- TableScan [TS_13]
- alias:cbo_t2
- Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_50]
+ predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_13]
+ alias:cbo_t2
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
PREHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by b+c, a desc) cbo_t1 right outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0) cbo_t2 on cbo_t1.a=p right outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 2) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c
PREHOOK: type: QUERY
@@ -867,129 +867,132 @@ Stage-0
limit:-1
Stage-1
Reducer 5
- File Output Operator [FS_36]
+ File Output Operator [FS_35]
compressed:false
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
- Group By Operator [GBY_34]
- | aggregations:["count(VALUE._col0)"]
- | keys:KEY._col0 (type: int), KEY._col1 (type: bigint)
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 4 [SIMPLE_EDGE]
- Reduce Output Operator [RS_33]
- key expressions:_col0 (type: int), _col1 (type: bigint)
- Map-reduce partition columns:_col0 (type: int), _col1 (type: bigint)
- sort order:++
- Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col2 (type: bigint)
- Group By Operator [GBY_32]
- aggregations:["count()"]
- keys:_col0 (type: int), _col1 (type: bigint)
- outputColumnNames:["_col0","_col1","_col2"]
+ Select Operator [SEL_34]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_33]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_32]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_30]
- outputColumnNames:["_col0","_col1"]
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_31]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_29]
- predicate:(((_col1 + _col4) >= 2) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ Select Operator [SEL_30]
+ outputColumnNames:["_col2","_col6"]
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Merge Join Operator [MERGEJOIN_41]
- | condition map:[{"":"Right Outer Join0 to 1"},{"":"Right Outer Join0 to 2"}]
- | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
- | outputColumnNames:["_col1","_col2","_col4","_col6"]
- | Statistics:Num rows: 4 Data size: 80 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Map 8 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_27]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int)
- | Select Operator [SEL_24]
- | outputColumnNames:["_col0","_col1"]
- | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
- | TableScan [TS_23]
- | alias:cbo_t3
- | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 3 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_25]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int), _col2 (type: bigint)
- | Select Operator [SEL_11]
- | | outputColumnNames:["_col0","_col1","_col2"]
- | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
- | |<-Reducer 2 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_10]
- | key expressions:_col3 (type: bigint), _col0 (type: string)
- | sort order:+-
- | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col1 (type: int), _col2 (type: bigint)
- | Select Operator [SEL_9]
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
- | Group By Operator [GBY_8]
- | | aggregations:["sum(VALUE._col0)"]
- | | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
- | | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | |<-Map 1 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_7]
- | key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | sort order:+++
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | value expressions:_col3 (type: bigint)
- | Group By Operator [GBY_6]
- | aggregations:["sum(_col1)"]
- | keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
- | Select Operator [SEL_2]
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | Filter Operator [FIL_39]
- | predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
- | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- | TableScan [TS_0]
- | alias:cbo_t1
- | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 7 [SIMPLE_EDGE]
- Reduce Output Operator [RS_26]
- key expressions:_col0 (type: string)
- Map-reduce partition columns:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col1 (type: int)
- Select Operator [SEL_22]
- outputColumnNames:["_col0","_col1"]
+ Filter Operator [FIL_29]
+ predicate:(((_col1 + _col4) >= 2) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_40]
+ | condition map:[{"":"Right Outer Join0 to 1"},{"":"Right Outer Join0 to 2"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col4","_col6"]
+ | Statistics:Num rows: 4 Data size: 80 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 8 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_27]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_24]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_23]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_25]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_11]
+ | | outputColumnNames:["_col0","_col1","_col2"]
+ | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 2 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_10]
+ | key expressions:_col3 (type: bigint), _col0 (type: string)
+ | sort order:+-
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_8]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_7]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Map 1 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_6]
+ | key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | sort order:+++
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col3 (type: bigint)
+ | Group By Operator [GBY_5]
+ | aggregations:["sum(_col2)"]
+ | keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | Select Operator [SEL_2]
+ | outputColumnNames:["_col0","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_38]
+ | predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_0]
+ | alias:cbo_t1
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 7 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_26]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_21]
- | keys:KEY._col0 (type: float), KEY._col1 (type: int), KEY._col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Map 6 [SIMPLE_EDGE]
- Reduce Output Operator [RS_20]
- key expressions:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- sort order:+++
- Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator [GBY_19]
- keys:_col0 (type: float), _col1 (type: int), _col2 (type: string)
- outputColumnNames:["_col0","_col1","_col2"]
+ value expressions:_col1 (type: int)
+ Select Operator [SEL_21]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_20]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 6 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_19]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_15]
+ Group By Operator [GBY_18]
+ keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_40]
- predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ Select Operator [SEL_15]
+ outputColumnNames:["_col0","_col2","_col3"]
Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
- TableScan [TS_13]
- alias:cbo_t2
- Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_39]
+ predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_13]
+ alias:cbo_t2
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
PREHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by c+a desc) cbo_t1 full outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by p+q desc, r asc) cbo_t2 on cbo_t1.a=p full outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c having cbo_t3.c_int > 0 and (c_int >=1 or c >= 1) and (c_int + c) >= 0 order by cbo_t3.c_int
PREHOOK: type: QUERY
@@ -1011,150 +1014,153 @@ Stage-0
limit:-1
Stage-1
Reducer 6
- File Output Operator [FS_43]
+ File Output Operator [FS_42]
compressed:false
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
- Select Operator [SEL_42]
+ Select Operator [SEL_41]
| outputColumnNames:["_col0","_col1","_col2"]
| Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
|<-Reducer 5 [SIMPLE_EDGE]
- Reduce Output Operator [RS_41]
+ Reduce Output Operator [RS_40]
key expressions:_col0 (type: int)
sort order:+
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
value expressions:_col1 (type: bigint), _col2 (type: bigint)
- Group By Operator [GBY_39]
- | aggregations:["count(VALUE._col0)"]
- | keys:KEY._col0 (type: int), KEY._col1 (type: bigint)
- | outputColumnNames:["_col0","_col1","_col2"]
- | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- |<-Reducer 4 [SIMPLE_EDGE]
- Reduce Output Operator [RS_38]
- key expressions:_col0 (type: int), _col1 (type: bigint)
- Map-reduce partition columns:_col0 (type: int), _col1 (type: bigint)
- sort order:++
- Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions:_col2 (type: bigint)
- Group By Operator [GBY_37]
- aggregations:["count()"]
- keys:_col0 (type: int), _col1 (type: bigint)
- outputColumnNames:["_col0","_col1","_col2"]
+ Select Operator [SEL_39]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_38]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_37]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator [SEL_33]
- outputColumnNames:["_col0","_col1"]
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_36]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator [FIL_45]
- predicate:(((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_
<TRUNCATED>
[30/51] [abbrv] hive git commit: HIVE-12015 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12015 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/c5ccf669
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/c5ccf669
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/c5ccf669
Branch: refs/heads/master
Commit: c5ccf6694490a63329b3f4d9040dd976abd9d790
Parents: a1bc2ef 5074423
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 1 19:38:52 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 1 19:38:52 2015 -0700
----------------------------------------------------------------------
.../common/metrics/common/MetricsConstant.java | 5 +
.../hcatalog/pig/TestHCatLoaderEncryption.java | 3 +
.../hive/ql/security/FolderPermissionBase.java | 53 +-
.../upgrade/derby/021-HIVE-11970.derby.sql | 6 +
.../upgrade/derby/hive-schema-1.3.0.derby.sql | 12 +-
.../upgrade/derby/hive-schema-2.0.0.derby.sql | 12 +-
.../derby/upgrade-1.2.0-to-1.3.0.derby.sql | 1 +
.../derby/upgrade-1.2.0-to-2.0.0.derby.sql | 3 +-
.../upgrade/mssql/007-HIVE-11970.mssql.sql | 6 +
.../upgrade/mssql/hive-schema-1.3.0.mssql.sql | 12 +-
.../upgrade/mssql/hive-schema-2.0.0.mssql.sql | 12 +-
.../mssql/upgrade-1.2.0-to-1.3.0.mssql.sql | 1 +
.../mssql/upgrade-1.2.0-to-2.0.0.mssql.sql | 7 +-
.../upgrade/mysql/022-HIVE-11970.mysql.sql | 6 +
.../upgrade/mysql/hive-schema-1.3.0.mysql.sql | 12 +-
.../upgrade/mysql/hive-schema-2.0.0.mysql.sql | 12 +-
.../mysql/upgrade-1.2.0-to-1.3.0.mysql.sql | 1 +
.../mysql/upgrade-1.2.0-to-2.0.0.mysql.sql | 2 +
.../upgrade/oracle/022-HIVE-11970.oracle.sql | 23 +
.../upgrade/oracle/hive-schema-1.3.0.oracle.sql | 12 +-
.../upgrade/oracle/hive-schema-2.0.0.oracle.sql | 12 +-
.../oracle/upgrade-1.2.0-to-1.3.0.oracle.sql | 2 +
.../oracle/upgrade-1.2.0-to-2.0.0.oracle.sql | 2 +
.../postgres/021-HIVE-11970.postgres.sql | 6 +
.../postgres/hive-schema-1.3.0.postgres.sql | 12 +-
.../postgres/hive-schema-2.0.0.postgres.sql | 12 +-
.../upgrade-1.2.0-to-1.3.0.postgres.sql | 1 +
.../upgrade-1.2.0-to-2.0.0.postgres.sql | 1 +
.../hive/metastore/MetaStoreDirectSql.java | 34 +-
.../hadoop/hive/metastore/txn/TxnHandler.java | 159 +++---
.../hadoop/hive/ql/exec/FileSinkOperator.java | 19 +-
.../apache/hadoop/hive/ql/exec/Utilities.java | 16 +
.../hadoop/hive/ql/io/HiveFileFormatUtils.java | 95 +++-
.../hadoop/hive/ql/io/InputFormatChecker.java | 5 +-
.../hadoop/hive/ql/io/RCFileInputFormat.java | 3 +-
.../ql/io/SequenceFileInputFormatChecker.java | 3 +-
.../hive/ql/io/VectorizedRCFileInputFormat.java | 3 +-
.../hive/ql/io/orc/ColumnStatisticsImpl.java | 55 +-
.../apache/hadoop/hive/ql/io/orc/InStream.java | 25 +
.../hive/ql/io/orc/MetadataReaderImpl.java | 2 +-
.../apache/hadoop/hive/ql/io/orc/OrcFile.java | 33 +-
.../hadoop/hive/ql/io/orc/OrcInputFormat.java | 4 +-
.../hadoop/hive/ql/io/orc/OrcOutputFormat.java | 145 ++++-
.../apache/hadoop/hive/ql/io/orc/OrcUtils.java | 177 +-----
.../hadoop/hive/ql/io/orc/ReaderImpl.java | 45 +-
.../hadoop/hive/ql/io/orc/TypeDescription.java | 466 ++++++++++++++++
.../ql/io/orc/VectorizedOrcInputFormat.java | 2 +-
.../apache/hadoop/hive/ql/io/orc/Writer.java | 9 +
.../hadoop/hive/ql/io/orc/WriterImpl.java | 550 +++++++++----------
.../hadoop/hive/ql/lockmgr/DbLockManager.java | 21 +
.../zookeeper/ZooKeeperHiveLockManager.java | 41 ++
.../apache/hadoop/hive/ql/metadata/Hive.java | 108 +---
.../hive/ql/optimizer/ColumnPrunerProcCtx.java | 2 +-
.../optimizer/ConstantPropagateProcFactory.java | 11 +-
.../hive/ql/optimizer/GenMapRedUtils.java | 10 +-
.../calcite/translator/HiveGBOpConvUtil.java | 43 +-
.../hadoop/hive/ql/parse/FromClauseParser.g | 30 +-
.../apache/hadoop/hive/ql/parse/HiveParser.g | 7 +-
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 33 +-
.../hive/ql/plan/DynamicPartitionCtx.java | 27 -
.../hadoop/hive/ql/io/orc/orc_proto.proto | 1 +
.../hive/ql/exec/TestFileSinkOperator.java | 386 ++++++++-----
.../hive/ql/io/orc/TestColumnStatistics.java | 43 +-
.../hive/ql/io/orc/TestInputOutputFormat.java | 15 +-
.../hadoop/hive/ql/io/orc/TestOrcFile.java | 41 +-
.../hive/ql/io/orc/TestOrcRawRecordMerger.java | 2 +-
.../hadoop/hive/ql/io/orc/TestOrcWideTable.java | 224 +-------
.../hive/ql/io/orc/TestTypeDescription.java | 67 +++
.../zookeeper/TestZookeeperLockManager.java | 50 ++
.../hive/ql/txn/compactor/CompactorTest.java | 2 +-
.../cbo_rp_gby2_map_multi_distinct.q | 38 ++
ql/src/test/queries/clientpositive/join_parse.q | 20 +
.../queries/clientpositive/update_all_types.q | 2 +-
.../clientpositive/windowing_windowspec2.q | 16 +-
.../resources/orc-file-dump-bloomfilter.out | 2 +-
.../resources/orc-file-dump-bloomfilter2.out | 2 +-
.../orc-file-dump-dictionary-threshold.out | 2 +-
ql/src/test/resources/orc-file-dump.json | 2 +-
ql/src/test/resources/orc-file-dump.out | 2 +-
ql/src/test/resources/orc-file-has-null.out | 2 +-
.../clientnegative/cte_with_in_subquery.q.out | 2 +-
.../clientpositive/annotate_stats_part.q.out | 6 +-
.../clientpositive/annotate_stats_table.q.out | 4 +-
.../cbo_rp_gby2_map_multi_distinct.q.out | 236 ++++++++
.../dynpart_sort_opt_vectorization.q.out | 16 +-
.../dynpart_sort_optimization2.q.out | 8 +-
.../extrapolate_part_stats_full.q.out | 24 +-
.../extrapolate_part_stats_partial.q.out | 76 +--
.../extrapolate_part_stats_partial_ndv.q.out | 38 +-
.../results/clientpositive/join_parse.q.out | 516 +++++++++++++++++
.../results/clientpositive/orc_analyze.q.out | 46 +-
.../results/clientpositive/orc_file_dump.q.out | 18 +-
.../clientpositive/orc_int_type_promotion.q.out | 6 +-
.../clientpositive/spark/vectorized_ptf.q.out | 108 ++--
.../tez/dynpart_sort_opt_vectorization.q.out | 16 +-
.../tez/dynpart_sort_optimization2.q.out | 8 +-
.../clientpositive/tez/orc_analyze.q.out | 46 +-
.../clientpositive/tez/union_fast_stats.q.out | 16 +-
.../clientpositive/tez/update_all_types.q.out | 4 +-
.../clientpositive/tez/vector_outer_join1.q.out | 48 +-
.../clientpositive/tez/vector_outer_join4.q.out | 48 +-
.../clientpositive/tez/vectorized_ptf.q.out | 108 ++--
.../clientpositive/union_fast_stats.q.out | 16 +-
.../clientpositive/update_all_types.q.out | 4 +-
.../results/clientpositive/vectorized_ptf.q.out | 104 ++--
.../clientpositive/windowing_windowspec2.q.out | 198 +++----
106 files changed, 3256 insertions(+), 1815 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
index 1c350db,5b21af9..1d79aff
--- a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
@@@ -3920,38 -3918,19 +3921,53 @@@ public final class Utilities
}
}
+ /**
+ * Returns the full path to the Jar containing the class. It always return a JAR.
+ *
+ * @param klass
+ * class.
+ *
+ * @return path to the Jar containing the class.
+ */
+ @SuppressWarnings("rawtypes")
+ public static String jarFinderGetJar(Class klass) {
+ Preconditions.checkNotNull(klass, "klass");
+ ClassLoader loader = klass.getClassLoader();
+ if (loader != null) {
+ String class_file = klass.getName().replaceAll("\\.", "/") + ".class";
+ try {
+ for (Enumeration itr = loader.getResources(class_file); itr.hasMoreElements();) {
+ URL url = (URL) itr.nextElement();
+ String path = url.getPath();
+ if (path.startsWith("file:")) {
+ path = path.substring("file:".length());
+ }
+ path = URLDecoder.decode(path, "UTF-8");
+ if ("jar".equals(url.getProtocol())) {
+ path = URLDecoder.decode(path, "UTF-8");
+ return path.replaceAll("!.*$", "");
+ }
+ }
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ return null;
+ }
+
+ public static int getDPColOffset(FileSinkDesc conf) {
+
+ if (conf.getWriteType() == AcidUtils.Operation.DELETE) {
+ // For deletes, there is only ROW__ID in non-partitioning, non-bucketing columns.
+ //See : UpdateDeleteSemanticAnalyzer::reparseAndSuperAnalyze() for details.
+ return 1;
+ } else if (conf.getWriteType() == AcidUtils.Operation.UPDATE) {
+ // For updates, ROW__ID is an extra column at index 0.
+ //See : UpdateDeleteSemanticAnalyzer::reparseAndSuperAnalyze() for details.
+ return getColumnNames(conf.getTableInfo().getProperties()).size() + 1;
+ } else {
+ return getColumnNames(conf.getTableInfo().getProperties()).size();
+ }
+
+ }
}
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java
index 8f0824b,6fec8b7..3dde0c4
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/InStream.java
@@@ -34,13 -35,12 +35,14 @@@ import com.google.protobuf.CodedInputSt
public abstract class InStream extends InputStream {
private static final Log LOG = LogFactory.getLog(InStream.class);
+ private static final int PROTOBUF_MESSAGE_MAX_LIMIT = 1024 << 20; // 1GB
+ protected final Long fileId;
protected final String name;
- protected final long length;
+ protected long length;
- public InStream(String name, long length) {
+ public InStream(Long fileId, String name, long length) {
+ this.fileId = fileId;
this.name = name;
this.length = length;
}
@@@ -468,9 -444,31 +470,32 @@@
CompressionCodec codec,
int bufferSize) throws IOException {
if (codec == null) {
- return new UncompressedStream(name, input, length);
+ return new UncompressedStream(fileId, name, input, length);
} else {
- return new CompressedStream(name, input, length, codec, bufferSize);
+ return new CompressedStream(fileId, name, input, length, codec, bufferSize);
}
}
+
+ /**
+ * Creates coded input stream (used for protobuf message parsing) with higher message size limit.
+ *
+ * @param name the name of the stream
+ * @param input the list of ranges of bytes for the stream; from disk or cache
+ * @param length the length in bytes of the stream
+ * @param codec the compression codec
+ * @param bufferSize the compression buffer size
+ * @return coded input stream
+ * @throws IOException
+ */
- public static CodedInputStream createCodedInputStream(String name,
++ public static CodedInputStream createCodedInputStream(Long fileId,
++ String name,
+ List<DiskRange> input,
+ long length,
+ CompressionCodec codec,
+ int bufferSize) throws IOException {
- InStream inStream = create(name, input, length, codec, bufferSize);
++ InStream inStream = create(fileId, name, input, length, codec, bufferSize);
+ CodedInputStream codedInputStream = CodedInputStream.newInstance(inStream);
+ codedInputStream.setSizeLimit(PROTOBUF_MESSAGE_MAX_LIMIT);
+ return codedInputStream;
+ }
}
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
index 1456df3,0000000..5afba51
mode 100644,000000..100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
@@@ -1,123 -1,0 +1,123 @@@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.io.orc;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.BufferChunk;
+
+import com.google.common.collect.Lists;
+
+public class MetadataReaderImpl implements MetadataReader {
+ private final FSDataInputStream file;
+ private final CompressionCodec codec;
+ private final int bufferSize;
+ private final int typeCount;
+
+ public MetadataReaderImpl(FileSystem fileSystem, Path path,
+ CompressionCodec codec, int bufferSize, int typeCount) throws IOException {
+ this(fileSystem.open(path), codec, bufferSize, typeCount);
+ }
+
+ public MetadataReaderImpl(FSDataInputStream file,
+ CompressionCodec codec, int bufferSize, int typeCount) {
+ this.file = file;
+ this.codec = codec;
+ this.bufferSize = bufferSize;
+ this.typeCount = typeCount;
+ }
+
+ @Override
+ public RecordReaderImpl.Index readRowIndex(StripeInformation stripe,
+ OrcProto.StripeFooter footer, boolean[] included, OrcProto.RowIndex[] indexes,
+ boolean[] sargColumns, OrcProto.BloomFilterIndex[] bloomFilterIndices) throws IOException {
+ if (footer == null) {
+ footer = readStripeFooter(stripe);
+ }
+ if (indexes == null) {
+ indexes = new OrcProto.RowIndex[typeCount];
+ }
+ if (bloomFilterIndices == null) {
+ bloomFilterIndices = new OrcProto.BloomFilterIndex[typeCount];
+ }
+ long offset = stripe.getOffset();
+ List<OrcProto.Stream> streams = footer.getStreamsList();
+ for (int i = 0; i < streams.size(); i++) {
+ OrcProto.Stream stream = streams.get(i);
+ OrcProto.Stream nextStream = null;
+ if (i < streams.size() - 1) {
+ nextStream = streams.get(i+1);
+ }
+ int col = stream.getColumn();
+ int len = (int) stream.getLength();
+ // row index stream and bloom filter are interlaced, check if the sarg column contains bloom
+ // filter and combine the io to read row index and bloom filters for that column together
+ if (stream.hasKind() && (stream.getKind() == OrcProto.Stream.Kind.ROW_INDEX)) {
+ boolean readBloomFilter = false;
+ if (sargColumns != null && sargColumns[col] &&
+ nextStream.getKind() == OrcProto.Stream.Kind.BLOOM_FILTER) {
+ len += nextStream.getLength();
+ i += 1;
+ readBloomFilter = true;
+ }
+ if ((included == null || included[col]) && indexes[col] == null) {
+ byte[] buffer = new byte[len];
+ file.readFully(offset, buffer, 0, buffer.length);
+ ByteBuffer bb = ByteBuffer.wrap(buffer);
+ indexes[col] = OrcProto.RowIndex.parseFrom(InStream.create(null, "index",
+ Lists.<DiskRange>newArrayList(new BufferChunk(bb, 0)), stream.getLength(),
+ codec, bufferSize));
+ if (readBloomFilter) {
+ bb.position((int) stream.getLength());
+ bloomFilterIndices[col] = OrcProto.BloomFilterIndex.parseFrom(InStream.create(
+ null, "bloom_filter", Lists.<DiskRange>newArrayList(new BufferChunk(bb, 0)),
+ nextStream.getLength(), codec, bufferSize));
+ }
+ }
+ }
+ offset += len;
+ }
+
+ RecordReaderImpl.Index index = new RecordReaderImpl.Index(indexes, bloomFilterIndices);
+ return index;
+ }
+
+ @Override
+ public OrcProto.StripeFooter readStripeFooter(StripeInformation stripe) throws IOException {
+ long offset = stripe.getOffset() + stripe.getIndexLength() + stripe.getDataLength();
+ int tailLength = (int) stripe.getFooterLength();
+
+ // read the footer
+ ByteBuffer tailBuf = ByteBuffer.allocate(tailLength);
+ file.readFully(offset, tailBuf.array(), tailBuf.arrayOffset(), tailLength);
- return OrcProto.StripeFooter.parseFrom(InStream.create(null, "footer",
++ return OrcProto.StripeFooter.parseFrom(InStream.createCodedInputStream(null, "footer",
+ Lists.<DiskRange>newArrayList(new BufferChunk(tailBuf, 0)),
+ tailLength, codec, bufferSize));
+ }
+
+ @Override
+ public void close() throws IOException {
+ file.close();
+ }
+}
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
index 04654bc,57bde3e..8941db1
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
@@@ -107,8 -106,8 +107,8 @@@ import com.google.common.util.concurren
* that added this event. Insert and update events include the entire row, while
* delete events have null for row.
*/
- public class OrcInputFormat implements InputFormat<NullWritable, OrcStruct>,
+ public class OrcInputFormat implements InputFormat<NullWritable, OrcStruct>,
- InputFormatChecker, VectorizedInputFormatInterface,
+ InputFormatChecker, VectorizedInputFormatInterface, LlapWrappableInputFormatInterface,
AcidInputFormat<NullWritable, OrcStruct>, CombineHiveInputFormat.AvoidSplitCombination {
static enum SplitStrategyKind{
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
index ce630bb,3bac48a..f3689fe
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
@@@ -423,47 -383,16 +419,16 @@@ public class ReaderImpl implements Read
int footerSize, CompressionCodec codec, int bufferSize) throws IOException {
bb.position(footerAbsPos);
bb.limit(footerAbsPos + footerSize);
- InputStream instream = InStream.create(null, "footer", Lists.<DiskRange>newArrayList(
- new BufferChunk(bb, 0)), footerSize, codec, bufferSize);
- return OrcProto.Footer.parseFrom(instream);
- return OrcProto.Footer.parseFrom(InStream.createCodedInputStream("footer",
++ return OrcProto.Footer.parseFrom(InStream.createCodedInputStream(null, "footer",
+ Lists.<DiskRange>newArrayList(new BufferChunk(bb, 0)), footerSize, codec, bufferSize));
}
private static OrcProto.Metadata extractMetadata(ByteBuffer bb, int metadataAbsPos,
int metadataSize, CompressionCodec codec, int bufferSize) throws IOException {
bb.position(metadataAbsPos);
bb.limit(metadataAbsPos + metadataSize);
- InputStream instream = InStream.create(null, "metadata", Lists.<DiskRange>newArrayList(
- new BufferChunk(bb, 0)), metadataSize, codec, bufferSize);
- CodedInputStream in = CodedInputStream.newInstance(instream);
- int msgLimit = DEFAULT_PROTOBUF_MESSAGE_LIMIT;
- OrcProto.Metadata meta = null;
- do {
- try {
- in.setSizeLimit(msgLimit);
- meta = OrcProto.Metadata.parseFrom(in);
- } catch (InvalidProtocolBufferException e) {
- if (e.getMessage().contains("Protocol message was too large")) {
- LOG.warn("Metadata section is larger than " + msgLimit + " bytes. Increasing the max" +
- " size of the coded input stream." );
-
- msgLimit = msgLimit << 1;
- if (msgLimit > PROTOBUF_MESSAGE_MAX_LIMIT) {
- LOG.error("Metadata section exceeds max protobuf message size of " +
- PROTOBUF_MESSAGE_MAX_LIMIT + " bytes.");
- throw e;
- }
-
- // we must have failed in the middle of reading instream and instream doesn't support
- // resetting the stream
- instream = InStream.create(null, "metadata", Lists.<DiskRange>newArrayList(
- new BufferChunk(bb, 0)), metadataSize, codec, bufferSize);
- in = CodedInputStream.newInstance(instream);
- } else {
- throw e;
- }
- }
- } while (meta == null);
- return meta;
- return OrcProto.Metadata.parseFrom(InStream.createCodedInputStream("metadata",
++ return OrcProto.Metadata.parseFrom(InStream.createCodedInputStream(null, "metadata",
+ Lists.<DiskRange>newArrayList(new BufferChunk(bb, 0)), metadataSize, codec, bufferSize));
}
private static OrcProto.PostScript extractPostScript(ByteBuffer bb, Path path,
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestColumnStatistics.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
----------------------------------------------------------------------
diff --cc ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
index 7e81615,06e3362..e78f7aa
--- a/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
+++ b/ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
@@@ -656,17 -657,18 +657,18 @@@ public class TestOrcFile
Reader reader = OrcFile.createReader(testFilePath,
OrcFile.readerOptions(conf).filesystem(fs));
- assertEquals(3, OrcUtils.getFlattenedColumnsCount(inspector));
+ TypeDescription schema = writer.getSchema();
+ assertEquals(2, schema.getMaximumId());
boolean[] expected = new boolean[] {false, true, false};
- boolean[] included = OrcUtils.includeColumns("int1", "int1,string1", inspector);
+ boolean[] included = OrcUtils.includeColumns("int1", schema);
assertEquals(true, Arrays.equals(expected, included));
- Metadata metadata = reader.getMetadata();
- int numStripes = metadata.getStripeStatistics().size();
+ List<StripeStatistics> stats = reader.getStripeStatistics();
+ int numStripes = stats.size();
assertEquals(3, numStripes);
- StripeStatistics ss1 = metadata.getStripeStatistics().get(0);
- StripeStatistics ss2 = metadata.getStripeStatistics().get(1);
- StripeStatistics ss3 = metadata.getStripeStatistics().get(2);
+ StripeStatistics ss1 = stats.get(0);
+ StripeStatistics ss2 = stats.get(1);
+ StripeStatistics ss3 = stats.get(2);
assertEquals(5000, ss1.getColumnStatistics()[0].getNumberOfValues());
assertEquals(5000, ss2.getColumnStatistics()[0].getNumberOfValues());
@@@ -777,9 -777,11 +777,9 @@@
true, true, true, true};
included = OrcUtils.includeColumns(
"boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map",
- "boolean1,byte1,short1,int1,long1,float1,double1,bytes1,string1,middle,list,map", inspector);
+ schema);
assertEquals(true, Arrays.equals(expected, included));
- Metadata metadata = reader.getMetadata();
-
// check the stats
ColumnStatistics[] stats = reader.getStatistics();
assertEquals(2, stats[1].getNumberOfValues());
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/tez/vector_outer_join1.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/vector_outer_join1.q.out
index 46a25e7,92e7163..c94c3f2
--- a/ql/src/test/results/clientpositive/tez/vector_outer_join1.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_outer_join1.q.out
@@@ -562,10 -562,9 +562,10 @@@ STAGE PLANS
key expressions: _col0 (type: tinyint)
sort order: +
Map-reduce partition columns: _col0 (type: tinyint)
- Statistics: Num rows: 15 Data size: 3483 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 3651 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), sum(VALUE._col1)
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/tez/vector_outer_join4.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/vector_outer_join4.q.out
index f0a9185,aa201ad..f66d5a4
--- a/ql/src/test/results/clientpositive/tez/vector_outer_join4.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_outer_join4.q.out
@@@ -931,10 -931,9 +931,10 @@@ STAGE PLANS
key expressions: _col0 (type: tinyint)
sort order: +
Map-reduce partition columns: _col0 (type: tinyint)
- Statistics: Num rows: 30 Data size: 6838 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 30 Data size: 7006 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/c5ccf669/ql/src/test/results/clientpositive/vectorized_ptf.q.out
----------------------------------------------------------------------
[44/51] [abbrv] hive git commit: HIVE-12073 : LLAP: disable session
reuse for MiniTez cluster (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12073 : LLAP: disable session reuse for MiniTez cluster (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/4cc2089e
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/4cc2089e
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/4cc2089e
Branch: refs/heads/master
Commit: 4cc2089e0a9d2e0aa92218e3010b1d06dbc42bed
Parents: 3cd915b
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 8 13:21:01 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 8 13:21:01 2015 -0700
----------------------------------------------------------------------
.../util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/4cc2089e/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
----------------------------------------------------------------------
diff --git a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
index 477a08d..174a988 100644
--- a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
+++ b/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
@@ -955,7 +955,7 @@ public class QTestUtil {
ss.setIsSilent(true);
SessionState oldSs = SessionState.get();
- if (oldSs != null && (clusterType == MiniClusterType.tez || clusterType == MiniClusterType.llap
+ if (oldSs != null && (clusterType == MiniClusterType.llap
|| clusterType == MiniClusterType.spark || clusterType == MiniClusterType.miniSparkOnYarn)) {
sparkSession = oldSs.getSparkSession();
ss.setSparkSession(sparkSession);
@@ -1023,7 +1023,7 @@ public class QTestUtil {
ss.err = System.out;
SessionState oldSs = SessionState.get();
- if (oldSs != null && (clusterType == MiniClusterType.tez || clusterType == MiniClusterType.llap
+ if (oldSs != null && (clusterType == MiniClusterType.llap
|| clusterType == MiniClusterType.miniSparkOnYarn || clusterType == MiniClusterType.miniSparkOnYarn)) {
sparkSession = oldSs.getSparkSession();
ss.setSparkSession(sparkSession);
[36/51] [abbrv] hive git commit: HIVE-12047 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/b87f63ca/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 7161e01,0000000..789180a
mode 100644,000000..100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@@ -1,8929 -1,0 +1,8929 @@@
+PREHOOK: query: explain create table src_orc_merge_test_part(key int, value string) partitioned by (ds string, ts string) stored as orc
+PREHOOK: type: CREATETABLE
+POSTHOOK: query: explain create table src_orc_merge_test_part(key int, value string) partitioned by (ds string, ts string) stored as orc
+POSTHOOK: type: CREATETABLE
+Stage-0
+ Create Table Operator:
+ columns:["key int","value string"]
+ input format:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat
+ name:default.src_orc_merge_test_part
+ output format:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat
+ partition columns:["ds string","ts string"]
+
+PREHOOK: query: create table src_orc_merge_test_part(key int, value string) partitioned by (ds string, ts string) stored as orc
+PREHOOK: type: CREATETABLE
+PREHOOK: Output: database:default
+PREHOOK: Output: default@src_orc_merge_test_part
+POSTHOOK: query: create table src_orc_merge_test_part(key int, value string) partitioned by (ds string, ts string) stored as orc
+POSTHOOK: type: CREATETABLE
+POSTHOOK: Output: database:default
+POSTHOOK: Output: default@src_orc_merge_test_part
+PREHOOK: query: alter table src_orc_merge_test_part add partition (ds='2012-01-03', ts='2012-01-03+14:46:31')
+PREHOOK: type: ALTERTABLE_ADDPARTS
+PREHOOK: Output: default@src_orc_merge_test_part
+POSTHOOK: query: alter table src_orc_merge_test_part add partition (ds='2012-01-03', ts='2012-01-03+14:46:31')
+POSTHOOK: type: ALTERTABLE_ADDPARTS
+POSTHOOK: Output: default@src_orc_merge_test_part
+POSTHOOK: Output: default@src_orc_merge_test_part@ds=2012-01-03/ts=2012-01-03+14%3A46%3A31
+PREHOOK: query: desc extended src_orc_merge_test_part partition (ds='2012-01-03', ts='2012-01-03+14:46:31')
+PREHOOK: type: DESCTABLE
+PREHOOK: Input: default@src_orc_merge_test_part
+POSTHOOK: query: desc extended src_orc_merge_test_part partition (ds='2012-01-03', ts='2012-01-03+14:46:31')
+POSTHOOK: type: DESCTABLE
+POSTHOOK: Input: default@src_orc_merge_test_part
+key int
+value string
+ds string
+ts string
+
+# Partition Information
+# col_name data_type comment
+
+ds string
+ts string
+
+#### A masked pattern was here ####
+PREHOOK: query: explain insert overwrite table src_orc_merge_test_part partition(ds='2012-01-03', ts='2012-01-03+14:46:31') select * from src
+PREHOOK: type: QUERY
+POSTHOOK: query: explain insert overwrite table src_orc_merge_test_part partition(ds='2012-01-03', ts='2012-01-03+14:46:31') select * from src
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Stage-3
+ Stats-Aggr Operator
+ Stage-0
+ Move Operator
+ partition:{"ds":"2012-01-03","ts":"2012-01-03+14:46:31"}
+ table:{"input format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde","name:":"default.src_orc_merge_test_part"}
+ Stage-2
+ Dependency Collection{}
+ Stage-1
+ Map 1
+ File Output Operator [FS_3]
+ compressed:false
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde","name:":"default.src_orc_merge_test_part"}
+ Select Operator [SEL_1]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_0]
+ alias:src
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+
+PREHOOK: query: insert overwrite table src_orc_merge_test_part partition(ds='2012-01-03', ts='2012-01-03+14:46:31') select * from src
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src
+PREHOOK: Output: default@src_orc_merge_test_part@ds=2012-01-03/ts=2012-01-03+14%3A46%3A31
+POSTHOOK: query: insert overwrite table src_orc_merge_test_part partition(ds='2012-01-03', ts='2012-01-03+14:46:31') select * from src
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src
+POSTHOOK: Output: default@src_orc_merge_test_part@ds=2012-01-03/ts=2012-01-03+14%3A46%3A31
+POSTHOOK: Lineage: src_orc_merge_test_part PARTITION(ds=2012-01-03,ts=2012-01-03+14:46:31).key EXPRESSION [(src)src.FieldSchema(name:key, type:string, comment:default), ]
+POSTHOOK: Lineage: src_orc_merge_test_part PARTITION(ds=2012-01-03,ts=2012-01-03+14:46:31).value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
+PREHOOK: query: explain insert into table src_orc_merge_test_part partition(ds='2012-01-03', ts='2012-01-03+14:46:31') select * from src limit 100
+PREHOOK: type: QUERY
+POSTHOOK: query: explain insert into table src_orc_merge_test_part partition(ds='2012-01-03', ts='2012-01-03+14:46:31') select * from src limit 100
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-3
+ Stats-Aggr Operator
+ Stage-0
+ Move Operator
+ partition:{"ds":"2012-01-03","ts":"2012-01-03+14:46:31"}
+ table:{"input format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde","name:":"default.src_orc_merge_test_part"}
+ Stage-2
+ Dependency Collection{}
+ Stage-1
+ Reducer 2
+ File Output Operator [FS_7]
+ compressed:false
+ Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.hive.ql.io.orc.OrcInputFormat","output format:":"org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat","serde:":"org.apache.hadoop.hive.ql.io.orc.OrcSerde","name:":"default.src_orc_merge_test_part"}
+ Select Operator [SEL_6]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
+ Limit [LIM_5]
+ Number of rows:100
+ Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_4]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_3]
+ sort order:
+ Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col0 (type: string), _col1 (type: string)
+ Limit [LIM_2]
+ Number of rows:100
+ Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_1]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_0]
+ alias:src
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+
+PREHOOK: query: explain select count(1) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select count(1) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 2
+ File Output Operator [FS_8]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Group By Operator [GBY_6]
+ | aggregations:["count(VALUE._col0)"]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_5]
+ sort order:
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col0 (type: bigint)
+ Group By Operator [GBY_4]
+ aggregations:["count(1)"]
+ outputColumnNames:["_col0"]
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_2]
+ Statistics:Num rows: 500 Data size: 47000 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_0]
+ alias:src_orc_merge_test_part
+ Statistics:Num rows: 500 Data size: 47000 Basic stats: COMPLETE Column stats: NONE
+
+PREHOOK: query: explain select sum(hash(key)), sum(hash(value)) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select sum(hash(key)), sum(hash(value)) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 2
+ File Output Operator [FS_8]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Group By Operator [GBY_6]
+ | aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_5]
+ sort order:
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col0 (type: bigint), _col1 (type: bigint)
+ Group By Operator [GBY_4]
+ aggregations:["sum(_col0)","sum(_col1)"]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_2]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 500 Data size: 47000 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_0]
+ alias:src_orc_merge_test_part
+ Statistics:Num rows: 500 Data size: 47000 Basic stats: COMPLETE Column stats: NONE
+
+PREHOOK: query: alter table src_orc_merge_test_part partition (ds='2012-01-03', ts='2012-01-03+14:46:31') concatenate
+PREHOOK: type: ALTER_PARTITION_MERGE
+PREHOOK: Input: default@src_orc_merge_test_part
+PREHOOK: Output: default@src_orc_merge_test_part@ds=2012-01-03/ts=2012-01-03+14%3A46%3A31
+POSTHOOK: query: alter table src_orc_merge_test_part partition (ds='2012-01-03', ts='2012-01-03+14:46:31') concatenate
+POSTHOOK: type: ALTER_PARTITION_MERGE
+POSTHOOK: Input: default@src_orc_merge_test_part
+POSTHOOK: Output: default@src_orc_merge_test_part@ds=2012-01-03/ts=2012-01-03+14%3A46%3A31
+PREHOOK: query: explain select count(1) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select count(1) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 2
+ File Output Operator [FS_8]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Group By Operator [GBY_6]
+ | aggregations:["count(VALUE._col0)"]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_5]
+ sort order:
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col0 (type: bigint)
+ Group By Operator [GBY_4]
+ aggregations:["count(1)"]
+ outputColumnNames:["_col0"]
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_2]
+ Statistics:Num rows: 1 Data size: 2515 Basic stats: PARTIAL Column stats: NONE
+ TableScan [TS_0]
+ alias:src_orc_merge_test_part
+ Statistics:Num rows: 1 Data size: 2515 Basic stats: PARTIAL Column stats: NONE
+
+PREHOOK: query: explain select sum(hash(key)), sum(hash(value)) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select sum(hash(key)), sum(hash(value)) from src_orc_merge_test_part where ds='2012-01-03' and ts='2012-01-03+14:46:31'
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 2
+ File Output Operator [FS_8]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Group By Operator [GBY_6]
+ | aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_5]
+ sort order:
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col0 (type: bigint), _col1 (type: bigint)
+ Group By Operator [GBY_4]
+ aggregations:["sum(_col0)","sum(_col1)"]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_2]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 24 Data size: 2515 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_0]
+ alias:src_orc_merge_test_part
+ Statistics:Num rows: 24 Data size: 2515 Basic stats: COMPLETE Column stats: NONE
+
+PREHOOK: query: drop table src_orc_merge_test_part
+PREHOOK: type: DROPTABLE
+PREHOOK: Input: default@src_orc_merge_test_part
+PREHOOK: Output: default@src_orc_merge_test_part
+POSTHOOK: query: drop table src_orc_merge_test_part
+POSTHOOK: type: DROPTABLE
+POSTHOOK: Input: default@src_orc_merge_test_part
+POSTHOOK: Output: default@src_orc_merge_test_part
+Warning: Map Join MAPJOIN[20][bigTable=?] in task 'Map 1' is a cross product
+PREHOOK: query: explain select sum(hash(a.k1,a.v1,a.k2, a.v2))
+from (
+select src1.key as k1, src1.value as v1,
+ src2.key as k2, src2.value as v2 FROM
+ (select * FROM src WHERE src.key < 10) src1
+ JOIN
+ (select * FROM src WHERE src.key < 10) src2
+ SORT BY k1, v1, k2, v2
+) a
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select sum(hash(a.k1,a.v1,a.k2, a.v2))
+from (
+select src1.key as k1, src1.value as v1,
+ src2.key as k2, src2.value as v2 FROM
+ (select * FROM src WHERE src.key < 10) src1
+ JOIN
+ (select * FROM src WHERE src.key < 10) src2
+ SORT BY k1, v1, k2, v2
+) a
+POSTHOOK: type: QUERY
+Plan not optimized by CBO.
+
+Vertex dependency in root stage
+Map 1 <- Map 4 (BROADCAST_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 3
+ File Output Operator [FS_17]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Group By Operator [GBY_15]
+ | aggregations:["sum(VALUE._col0)"]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ |<-Reducer 2 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_14]
+ sort order:
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col0 (type: bigint)
+ Group By Operator [GBY_13]
+ aggregations:["sum(hash(_col0,_col1,_col2,_col3))"]
+ outputColumnNames:["_col0"]
+ Statistics:Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator [SEL_11]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_10]
+ key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string)
+ sort order:++++
+ Statistics:Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator [MAPJOIN_20]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{}
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 4 [BROADCAST_EDGE]
+ | Reduce Output Operator [RS_7]
+ | sort order:
+ | Statistics:Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col0 (type: string), _col1 (type: string)
+ | Select Operator [SEL_5]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ | Filter Operator [FIL_19]
+ | predicate:(key < 10) (type: boolean)
+ | Statistics:Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_3]
+ | alias:src
+ | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ |<-Select Operator [SEL_2]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator [FIL_18]
+ predicate:(key < 10) (type: boolean)
+ Statistics:Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_0]
+ alias:src
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+
+PREHOOK: query: explain select key, (c_int+1)+2 as x, sum(c_int) from cbo_t1 group by c_float, cbo_t1.c_int, key
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select key, (c_int+1)+2 as x, sum(c_int) from cbo_t1 group by c_float, cbo_t1.c_int, key
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 2
+ File Output Operator [FS_7]
+ compressed:false
+ Statistics:Num rows: 10 Data size: 885 Basic stats: COMPLETE Column stats: COMPLETE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Select Operator [SEL_5]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 10 Data size: 885 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_4]
+ | aggregations:["sum(VALUE._col0)"]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_3]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
+ Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col3 (type: bigint)
+ Group By Operator [GBY_2]
+ aggregations:["sum(c_int)"]
+ keys:key (type: string), c_int (type: int), c_float (type: float)
+ outputColumnNames:["_col0","_col1","_col2","_col3"]
+ Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_1]
+ outputColumnNames:["key","c_int","c_float"]
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_0]
+ alias:cbo_t1
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+
+PREHOOK: query: explain select x, y, count(*) from (select key, (c_int+c_float+1+2) as x, sum(c_int) as y from cbo_t1 group by c_float, cbo_t1.c_int, key) R group by y, x
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select x, y, count(*) from (select key, (c_int+c_float+1+2) as x, sum(c_int) as y from cbo_t1 group by c_float, cbo_t1.c_int, key) R group by y, x
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 3
+ File Output Operator [FS_12]
+ compressed:false
+ Statistics:Num rows: 5 Data size: 100 Basic stats: COMPLETE Column stats: COMPLETE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Select Operator [SEL_11]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 5 Data size: 100 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_10]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 5 Data size: 100 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 2 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_9]
+ key expressions:_col0 (type: bigint), _col1 (type: float)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: float)
+ sort order:++
+ Statistics:Num rows: 5 Data size: 100 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_8]
+ aggregations:["count()"]
+ keys:_col0 (type: bigint), _col1 (type: float)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 5 Data size: 100 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_5]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_4]
+ | aggregations:["sum(VALUE._col0)"]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_3]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
+ Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col3 (type: bigint)
+ Group By Operator [GBY_2]
+ aggregations:["sum(c_int)"]
+ keys:key (type: string), c_int (type: int), c_float (type: float)
+ outputColumnNames:["_col0","_col1","_col2","_col3"]
+ Statistics:Num rows: 10 Data size: 917 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_1]
+ outputColumnNames:["key","c_int","c_float"]
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_0]
+ alias:cbo_t1
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+
+PREHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key order by a) cbo_t1 join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key order by q/10 desc, r asc) cbo_t2 on cbo_t1.a=p join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c order by cbo_t3.c_int+c desc, c
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key order by a) cbo_t1 join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key order by q/10 desc, r asc) cbo_t2 on cbo_t1.a=p join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c order by cbo_t3.c_int+c desc, c
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Map 11 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+Reducer 9 <- Map 8 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 7
+ File Output Operator [FS_45]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Select Operator [SEL_44]
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 6 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_43]
+ key expressions:(UDFToLong(_col0) + _col1) (type: bigint), _col1 (type: bigint)
+ sort order:-+
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col0 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_41]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_40]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 5 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_39]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_38]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_37]
+ outputColumnNames:["_col2","_col6"]
+ Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_36]
+ predicate:((_col1 > 0) or (_col6 >= 0)) (type: boolean)
+ Statistics:Num rows: 2 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_55]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col6"]
+ | Statistics:Num rows: 3 Data size: 48 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 11 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_34]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_30]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_53]
+ | predicate:key is not null (type: boolean)
+ | Statistics:Num rows: 18 Data size: 1424 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_29]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_32]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_28]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_27]
+ predicate:((_col1 + _col4) >= 0) (type: boolean)
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_54]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col0","_col1","_col2","_col4"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 10 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_25]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_20]
+ | | outputColumnNames:["_col0","_col1"]
+ | | Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 9 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_19]
+ | key expressions:_col3 (type: double), _col2 (type: bigint)
+ | sort order:-+
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col0 (type: string), _col1 (type: int)
+ | Select Operator [SEL_17]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_16]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Map 8 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_15]
+ | key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | sort order:+++
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col3 (type: bigint)
+ | Group By Operator [GBY_14]
+ | aggregations:["sum(c_int)"]
+ | keys:key (type: string), c_int (type: int), c_float (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_52]
+ | predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
+ | Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_11]
+ | alias:cbo_t2
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_23]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_9]
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 2 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_8]
+ key expressions:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_6]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_5]
+ | aggregations:["sum(VALUE._col0)"]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 1 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_4]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col3 (type: bigint)
+ Group By Operator [GBY_3]
+ aggregations:["sum(c_int)"]
+ keys:key (type: string), c_int (type: int), c_float (type: float)
+ outputColumnNames:["_col0","_col1","_col2","_col3"]
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_51]
+ predicate:((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and key is not null) (type: boolean)
+ Statistics:Num rows: 3 Data size: 279 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_0]
+ alias:cbo_t1
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+
+PREHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by b % c asc, b desc) cbo_t1 left outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0) cbo_t2 on cbo_t1.a=p left outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c having cbo_t3.c_int > 0 and (c_int >=1 or c >= 1) and (c_int + c) >= 0 order by cbo_t3.c_int % c asc, cbo_t3.c_int desc
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by b % c asc, b desc) cbo_t1 left outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0) cbo_t2 on cbo_t1.a=p left outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c having cbo_t3.c_int > 0 and (c_int >=1 or c >= 1) and (c_int + c) >= 0 order by cbo_t3.c_int % c asc, cbo_t3.c_int desc
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 5 <- Map 10 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+Reducer 9 <- Map 8 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 7
+ File Output Operator [FS_44]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Select Operator [SEL_43]
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 6 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_42]
+ key expressions:(UDFToLong(_col0) % _col1) (type: bigint), _col0 (type: int)
+ sort order:+-
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: bigint), _col2 (type: bigint)
+ Select Operator [SEL_40]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_39]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 5 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_38]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_37]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_34]
+ outputColumnNames:["_col2","_col6"]
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_48]
+ predicate:((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_col6) + _col2) >= 0)) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ Statistics:Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_53]
+ | condition map:[{"":"Left Outer Join0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col6"]
+ | Statistics:Num rows: 4 Data size: 64 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 10 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_31]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_29]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_28]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_30]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int), _col2 (type: bigint)
+ Select Operator [SEL_27]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_26]
+ predicate:((_col1 + _col4) >= 0) (type: boolean)
+ Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_52]
+ | condition map:[{"":"Left Outer Join0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col0","_col1","_col2","_col4"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_23]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_11]
+ | | outputColumnNames:["_col0","_col1","_col2"]
+ | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 2 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_10]
+ | key expressions:_col3 (type: bigint), _col1 (type: int)
+ | sort order:+-
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col0 (type: string), _col2 (type: bigint)
+ | Select Operator [SEL_8]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_7]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Map 1 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_6]
+ | key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | sort order:+++
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col3 (type: bigint)
+ | Group By Operator [GBY_5]
+ | aggregations:["sum(_col2)"]
+ | keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | Select Operator [SEL_2]
+ | outputColumnNames:["_col0","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_49]
+ | predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_0]
+ | alias:cbo_t1
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 9 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_24]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int)
+ Select Operator [SEL_21]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_20]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 8 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_19]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_18]
+ keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_15]
+ outputColumnNames:["_col0","_col2","_col3"]
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_50]
+ predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_13]
+ alias:cbo_t2
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+
+PREHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by b+c, a desc) cbo_t1 right outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0) cbo_t2 on cbo_t1.a=p right outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 2) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by b+c, a desc) cbo_t1 right outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0) cbo_t2 on cbo_t1.a=p right outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 2) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 8 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
+Reducer 7 <- Map 6 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 5
+ File Output Operator [FS_35]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Select Operator [SEL_34]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_33]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_32]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_31]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_30]
+ outputColumnNames:["_col2","_col6"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_29]
+ predicate:(((_col1 + _col4) >= 2) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_40]
+ | condition map:[{"":"Right Outer Join0 to 1"},{"":"Right Outer Join0 to 2"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col4","_col6"]
+ | Statistics:Num rows: 4 Data size: 80 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 8 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_27]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_24]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_23]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_25]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_11]
+ | | outputColumnNames:["_col0","_col1","_col2"]
+ | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 2 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_10]
+ | key expressions:_col3 (type: bigint), _col0 (type: string)
+ | sort order:+-
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_8]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_7]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Map 1 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_6]
+ | key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ | sort order:+++
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col3 (type: bigint)
+ | Group By Operator [GBY_5]
+ | aggregations:["sum(_col2)"]
+ | keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+ | Select Operator [SEL_2]
+ | outputColumnNames:["_col0","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | Filter Operator [FIL_38]
+ | predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_0]
+ | alias:cbo_t1
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 7 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_26]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: int)
+ Select Operator [SEL_21]
+ outputColumnNames:["_col0","_col1"]
+ Statistics:Num rows: 1 Data size: 89 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_20]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 6 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_19]
+ key expressions:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: int), _col2 (type: float)
+ sort order:+++
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_18]
+ keys:_col0 (type: string), _col2 (type: int), _col3 (type: float)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_15]
+ outputColumnNames:["_col0","_col2","_col3"]
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_39]
+ predicate:((((((c_int + 1) >= 0) and ((c_int > 0) or (c_float >= 0.0))) and (c_float > 0.0)) and ((c_int >= 1) or (c_float >= 1.0))) and ((UDFToFloat(c_int) + c_float) >= 0.0)) (type: boolean)
+ Statistics:Num rows: 1 Data size: 93 Basic stats: COMPLETE Column stats: COMPLETE
+ TableScan [TS_13]
+ alias:cbo_t2
+ Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+
+PREHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by c+a desc) cbo_t1 full outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by p+q desc, r asc) cbo_t2 on cbo_t1.a=p full outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c having cbo_t3.c_int > 0 and (c_int >=1 or c >= 1) and (c_int + c) >= 0 order by cbo_t3.c_int
+PREHOOK: type: QUERY
+POSTHOOK: query: explain select cbo_t3.c_int, c, count(*) from (select key as a, c_int+1 as b, sum(c_int) as c from cbo_t1 where (cbo_t1.c_int + 1 >= 0) and (cbo_t1.c_int > 0 or cbo_t1.c_float >= 0) group by c_float, cbo_t1.c_int, key having cbo_t1.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by c+a desc) cbo_t1 full outer join (select key as p, c_int+1 as q, sum(c_int) as r from cbo_t2 where (cbo_t2.c_int + 1 >= 0) and (cbo_t2.c_int > 0 or cbo_t2.c_float >= 0) group by c_float, cbo_t2.c_int, key having cbo_t2.c_float > 0 and (c_int >=1 or c_float >= 1) and (c_int + c_float) >= 0 order by p+q desc, r asc) cbo_t2 on cbo_t1.a=p full outer join cbo_t3 on cbo_t1.a=key where (b + cbo_t2.q >= 0) and (b > 0 or c_int >= 0) group by cbo_t3.c_int, c having cbo_t3.c_int > 0 and (c_int >=1 or c >= 1) and (c_int + c) >= 0 order by cbo_t3.c_int
+POSTHOOK: type: QUERY
+Plan optimized by CBO.
+
+Vertex dependency in root stage
+Reducer 2 <- Map 1 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+Reducer 8 <- Map 7 (SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
+
+Stage-0
+ Fetch Operator
+ limit:-1
+ Stage-1
+ Reducer 6
+ File Output Operator [FS_42]
+ compressed:false
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
+ Select Operator [SEL_41]
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 5 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_40]
+ key expressions:_col0 (type: int)
+ sort order:+
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col1 (type: bigint), _col2 (type: bigint)
+ Select Operator [SEL_39]
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator [GBY_38]
+ | aggregations:["count(VALUE._col0)"]
+ | keys:KEY._col0 (type: bigint), KEY._col1 (type: int)
+ | outputColumnNames:["_col0","_col1","_col2"]
+ | Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 4 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_37]
+ key expressions:_col0 (type: bigint), _col1 (type: int)
+ Map-reduce partition columns:_col0 (type: bigint), _col1 (type: int)
+ sort order:++
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions:_col2 (type: bigint)
+ Group By Operator [GBY_36]
+ aggregations:["count()"]
+ keys:_col2 (type: bigint), _col6 (type: int)
+ outputColumnNames:["_col0","_col1","_col2"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator [SEL_33]
+ outputColumnNames:["_col2","_col6"]
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator [FIL_44]
+ predicate:(((((_col6 > 0) and ((_col6 >= 1) or (_col2 >= 1))) and ((UDFToLong(_col6) + _col2) >= 0)) and ((_col1 + _col4) >= 0)) and ((_col1 > 0) or (_col6 >= 0))) (type: boolean)
+ Statistics:Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator [MERGEJOIN_47]
+ | condition map:[{"":"Outer Join 0 to 1"},{"":"Outer Join 0 to 2"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)","2":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col4","_col6"]
+ | Statistics:Num rows: 4 Data size: 80 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Map 10 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_30]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int)
+ | Select Operator [SEL_27]
+ | outputColumnNames:["_col0","_col1"]
+ | Statistics:Num rows: 20 Data size: 1602 Basic stats: COMPLETE Column stats: COMPLETE
+ | TableScan [TS_26]
+ | alias:cbo_t3
+ | Statistics:Num rows: 20 Data size: 262 Basic stats: COMPLETE Column stats: COMPLETE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_28]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_11]
+ | | outputColumnNames:["_col0","_col1","_col2"]
+ | | Statistics:Num rows: 1 Data size: 97 Basic stats: COMPLETE Column stats: COMPLETE
+ | |<-Reducer 2 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_10]
+ | key expressions:_col3 (type: double)
+ | sort order:-
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | value expressions:_col0 (type: string), _col1 (type: int), _col2 (type: bigint)
+ | Select Operator [SEL_8]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 1 Data size: 105 Basic stats: COMPLETE Column stats: COMPLETE
+ | Group By Operator [GBY_7]
+ | | aggregations:["sum(VALUE._col0)"]
+ | | keys:KEY._col0 (type: string), KEY._col1 (type: int), KEY._col2 (type: float)
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 1 Data size: 101 Basic stats: COMPLETE Column stats: COMPLETE
+
<TRUNCATED>
[12/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/explainuser_2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_2.q.out b/ql/src/test/results/clientpositive/llap/explainuser_2.q.out
index 8156789..57fcc3c 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_2.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_2.q.out
@@ -335,245 +335,248 @@ Stage-0
limit:100
Stage-1
Reducer 5
- File Output Operator [FS_69]
+ File Output Operator [FS_68]
compressed:false
Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
- Limit [LIM_68]
+ Limit [LIM_67]
Number of rows:100
Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
- Select Operator [SEL_67]
+ Select Operator [SEL_66]
| outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
| Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
|<-Reducer 4 [SIMPLE_EDGE]
- Reduce Output Operator [RS_66]
+ Reduce Output Operator [RS_65]
key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string)
sort order:+++
Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
value expressions:_col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint)
- Group By Operator [GBY_64]
- | aggregations:["count(VALUE._col0)","count(VALUE._col1)","count(VALUE._col2)"]
- | keys:KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
- | Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
- |<-Reducer 3 [SIMPLE_EDGE]
- Reduce Output Operator [RS_63]
- key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: string), _col1 (type: string), _col2 (type: string)
- sort order:+++
- Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- value expressions:_col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint)
- Group By Operator [GBY_62]
- aggregations:["count(_col3)","count(_col4)","count(_col5)"]
- keys:_col0 (type: string), _col1 (type: string), _col2 (type: string)
- outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Select Operator [SEL_64]
+ outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator [GBY_63]
+ | aggregations:["count(VALUE._col0)","count(VALUE._col1)","count(VALUE._col2)"]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
+ | outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ | Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
+ |<-Reducer 3 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_62]
+ key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: string), _col2 (type: string)
+ sort order:+++
Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- Select Operator [SEL_60]
+ value expressions:_col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint)
+ Group By Operator [GBY_61]
+ aggregations:["count(_col13)","count(_col21)","count(_col3)"]
+ keys:_col2 (type: string), _col12 (type: string), _col20 (type: string)
outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- Merge Join Operator [MERGEJOIN_111]
- | condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"0":"_col1 (type: string), _col3 (type: string)","1":"_col15 (type: string), _col17 (type: string)"}
- | outputColumnNames:["_col2","_col3","_col12","_col13","_col20","_col21"]
- | Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- |<-Reducer 11 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_58]
- | key expressions:_col15 (type: string), _col17 (type: string)
- | Map-reduce partition columns:_col15 (type: string), _col17 (type: string)
- | sort order:++
- | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
- | value expressions:_col6 (type: string), _col7 (type: string), _col14 (type: string)
- | Select Operator [SEL_49]
- | outputColumnNames:["_col14","_col15","_col17","_col6","_col7"]
- | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
- | Merge Join Operator [MERGEJOIN_110]
- | | condition map:[{"":"Inner Join 0 to 1"}]
- | | keys:{"0":"_col4 (type: string), _col6 (type: string)","1":"_col2 (type: string), _col4 (type: string)"}
- | | outputColumnNames:["_col2","_col3","_col14","_col15","_col17"]
- | | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
- | |<-Reducer 10 [SIMPLE_EDGE]
- | | Reduce Output Operator [RS_45]
- | | key expressions:_col4 (type: string), _col6 (type: string)
- | | Map-reduce partition columns:_col4 (type: string), _col6 (type: string)
- | | sort order:++
- | | Statistics:Num rows: 665 Data size: 7069 Basic stats: COMPLETE Column stats: NONE
- | | value expressions:_col2 (type: string), _col3 (type: string)
- | | Merge Join Operator [MERGEJOIN_108]
- | | | condition map:[{"":"Inner Join 0 to 1"}]
- | | | keys:{"0":"_col3 (type: string)","1":"_col1 (type: string)"}
- | | | outputColumnNames:["_col2","_col3","_col4","_col6"]
- | | | Statistics:Num rows: 665 Data size: 7069 Basic stats: COMPLETE Column stats: NONE
- | | |<-Map 14 [SIMPLE_EDGE]
- | | | Reduce Output Operator [RS_42]
- | | | key expressions:_col1 (type: string)
- | | | Map-reduce partition columns:_col1 (type: string)
- | | | sort order:+
- | | | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
- | | | Select Operator [SEL_16]
- | | | outputColumnNames:["_col1"]
- | | | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
- | | | Filter Operator [FIL_102]
- | | | predicate:((key = 'src1key') and value is not null) (type: boolean)
- | | | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
- | | | TableScan [TS_14]
- | | | alias:src1
- | | | Statistics:Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE
- | | |<-Reducer 9 [SIMPLE_EDGE]
- | | Reduce Output Operator [RS_40]
- | | key expressions:_col3 (type: string)
- | | Map-reduce partition columns:_col3 (type: string)
- | | sort order:+
- | | Statistics:Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
- | | value expressions:_col2 (type: string), _col4 (type: string), _col6 (type: string)
- | | Merge Join Operator [MERGEJOIN_107]
- | | | condition map:[{"":"Inner Join 0 to 1"}]
- | | | keys:{"0":"_col2 (type: string)","1":"_col0 (type: string)"}
- | | | outputColumnNames:["_col2","_col3","_col4","_col6"]
- | | | Statistics:Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
- | | |<-Map 13 [SIMPLE_EDGE]
- | | | Reduce Output Operator [RS_37]
- | | | key expressions:_col0 (type: string)
- | | | Map-reduce partition columns:_col0 (type: string)
- | | | sort order:+
- | | | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | | | Select Operator [SEL_13]
- | | | outputColumnNames:["_col0"]
- | | | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | | | Filter Operator [FIL_101]
- | | | predicate:((value = 'd1value') and key is not null) (type: boolean)
- | | | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | | | TableScan [TS_11]
- | | | alias:d1
- | | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- | | |<-Reducer 8 [SIMPLE_EDGE]
- | | Reduce Output Operator [RS_35]
- | | key expressions:_col2 (type: string)
- | | Map-reduce partition columns:_col2 (type: string)
- | | sort order:+
- | | Statistics:Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- | | value expressions:_col3 (type: string), _col4 (type: string), _col6 (type: string)
- | | Merge Join Operator [MERGEJOIN_106]
- | | | condition map:[{"":"Inner Join 0 to 1"}]
- | | | keys:{"0":"_col1 (type: string)","1":"_col3 (type: string)"}
- | | | outputColumnNames:["_col2","_col3","_col4","_col6"]
- | | | Statistics:Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- | | |<-Map 12 [SIMPLE_EDGE]
- | | | Reduce Output Operator [RS_32]
- | | | key expressions:_col3 (type: string)
- | | | Map-reduce partition columns:_col3 (type: string)
- | | | sort order:+
- | | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | | value expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string), _col4 (type: string)
- | | | Select Operator [SEL_10]
- | | | outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
- | | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | | Filter Operator [FIL_100]
- | | | predicate:((((((v3 = 'ssv3') and v2 is not null) and k1 is not null) and v1 is not null) and k2 is not null) and k3 is not null) (type: boolean)
- | | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | | TableScan [TS_8]
- | | | alias:ss
- | | | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
- | | |<-Map 7 [SIMPLE_EDGE]
- | | Reduce Output Operator [RS_30]
- | | key expressions:_col1 (type: string)
- | | Map-reduce partition columns:_col1 (type: string)
- | | sort order:+
- | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- | | Select Operator [SEL_7]
- | | outputColumnNames:["_col1"]
- | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- | | Filter Operator [FIL_99]
- | | predicate:((key = 'srcpartkey') and value is not null) (type: boolean)
- | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- | | TableScan [TS_5]
- | | alias:srcpart
- | | Statistics:Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- | |<-Reducer 16 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_47]
- | key expressions:_col2 (type: string), _col4 (type: string)
- | Map-reduce partition columns:_col2 (type: string), _col4 (type: string)
- | sort order:++
- | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- | value expressions:_col3 (type: string), _col5 (type: string)
- | Merge Join Operator [MERGEJOIN_109]
- | | condition map:[{"":"Inner Join 0 to 1"}]
- | | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
- | | outputColumnNames:["_col2","_col3","_col4","_col5"]
- | | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- | |<-Map 15 [SIMPLE_EDGE]
- | | Reduce Output Operator [RS_24]
- | | key expressions:_col0 (type: string)
- | | Map-reduce partition columns:_col0 (type: string)
- | | sort order:+
- | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | value expressions:_col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string)
- | | Select Operator [SEL_19]
- | | outputColumnNames:["_col0","_col2","_col3","_col4","_col5"]
- | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | Filter Operator [FIL_103]
- | | predicate:((((((v1 = 'srv1') and k1 is not null) and k2 is not null) and k3 is not null) and v2 is not null) and v3 is not null) (type: boolean)
- | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | TableScan [TS_17]
- | | alias:sr
- | | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
- | |<-Map 17 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_26]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | Select Operator [SEL_22]
- | outputColumnNames:["_col0"]
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | Filter Operator [FIL_104]
- | predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_20]
- | alias:d1
- | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- |<-Reducer 2 [SIMPLE_EDGE]
- Reduce Output Operator [RS_56]
- key expressions:_col1 (type: string), _col3 (type: string)
- Map-reduce partition columns:_col1 (type: string), _col3 (type: string)
- sort order:++
- Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- value expressions:_col2 (type: string)
- Merge Join Operator [MERGEJOIN_105]
- | condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
- | outputColumnNames:["_col1","_col2","_col3"]
- | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- |<-Map 1 [SIMPLE_EDGE]
- | Reduce Output Operator [RS_51]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
- | sort order:+
- | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
- | value expressions:_col1 (type: string), _col2 (type: string), _col3 (type: string)
- | Select Operator [SEL_1]
- | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
- | Filter Operator [FIL_97]
- | predicate:((k1 is not null and v2 is not null) and v3 is not null) (type: boolean)
- | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_0]
- | alias:cs
- | Statistics:Num rows: 170 Data size: 5890 Basic stats: COMPLETE Column stats: NONE
- |<-Map 6 [SIMPLE_EDGE]
- Reduce Output Operator [RS_53]
- key expressions:_col0 (type: string)
- Map-reduce partition columns:_col0 (type: string)
- sort order:+
- Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Select Operator [SEL_4]
- outputColumnNames:["_col0"]
+ Select Operator [SEL_60]
+ outputColumnNames:["_col2","_col12","_col20","_col13","_col21","_col3"]
+ Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
+ Merge Join Operator [MERGEJOIN_110]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"0":"_col1 (type: string), _col3 (type: string)","1":"_col15 (type: string), _col17 (type: string)"}
+ | outputColumnNames:["_col2","_col3","_col12","_col13","_col20","_col21"]
+ | Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
+ |<-Reducer 11 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_58]
+ | key expressions:_col15 (type: string), _col17 (type: string)
+ | Map-reduce partition columns:_col15 (type: string), _col17 (type: string)
+ | sort order:++
+ | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col6 (type: string), _col7 (type: string), _col14 (type: string)
+ | Select Operator [SEL_49]
+ | outputColumnNames:["_col14","_col15","_col17","_col6","_col7"]
+ | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
+ | Merge Join Operator [MERGEJOIN_109]
+ | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | keys:{"0":"_col4 (type: string), _col6 (type: string)","1":"_col2 (type: string), _col4 (type: string)"}
+ | | outputColumnNames:["_col2","_col3","_col14","_col15","_col17"]
+ | | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
+ | |<-Reducer 10 [SIMPLE_EDGE]
+ | | Reduce Output Operator [RS_45]
+ | | key expressions:_col4 (type: string), _col6 (type: string)
+ | | Map-reduce partition columns:_col4 (type: string), _col6 (type: string)
+ | | sort order:++
+ | | Statistics:Num rows: 665 Data size: 7069 Basic stats: COMPLETE Column stats: NONE
+ | | value expressions:_col2 (type: string), _col3 (type: string)
+ | | Merge Join Operator [MERGEJOIN_107]
+ | | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | | keys:{"0":"_col3 (type: string)","1":"_col1 (type: string)"}
+ | | | outputColumnNames:["_col2","_col3","_col4","_col6"]
+ | | | Statistics:Num rows: 665 Data size: 7069 Basic stats: COMPLETE Column stats: NONE
+ | | |<-Map 14 [SIMPLE_EDGE]
+ | | | Reduce Output Operator [RS_42]
+ | | | key expressions:_col1 (type: string)
+ | | | Map-reduce partition columns:_col1 (type: string)
+ | | | sort order:+
+ | | | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
+ | | | Select Operator [SEL_16]
+ | | | outputColumnNames:["_col1"]
+ | | | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
+ | | | Filter Operator [FIL_101]
+ | | | predicate:((key = 'src1key') and value is not null) (type: boolean)
+ | | | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
+ | | | TableScan [TS_14]
+ | | | alias:src1
+ | | | Statistics:Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE
+ | | |<-Reducer 9 [SIMPLE_EDGE]
+ | | Reduce Output Operator [RS_40]
+ | | key expressions:_col3 (type: string)
+ | | Map-reduce partition columns:_col3 (type: string)
+ | | sort order:+
+ | | Statistics:Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ | | value expressions:_col2 (type: string), _col4 (type: string), _col6 (type: string)
+ | | Merge Join Operator [MERGEJOIN_106]
+ | | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | | keys:{"0":"_col2 (type: string)","1":"_col0 (type: string)"}
+ | | | outputColumnNames:["_col2","_col3","_col4","_col6"]
+ | | | Statistics:Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ | | |<-Map 13 [SIMPLE_EDGE]
+ | | | Reduce Output Operator [RS_37]
+ | | | key expressions:_col0 (type: string)
+ | | | Map-reduce partition columns:_col0 (type: string)
+ | | | sort order:+
+ | | | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | | | Select Operator [SEL_13]
+ | | | outputColumnNames:["_col0"]
+ | | | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | | | Filter Operator [FIL_100]
+ | | | predicate:((value = 'd1value') and key is not null) (type: boolean)
+ | | | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | | | TableScan [TS_11]
+ | | | alias:d1
+ | | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ | | |<-Reducer 8 [SIMPLE_EDGE]
+ | | Reduce Output Operator [RS_35]
+ | | key expressions:_col2 (type: string)
+ | | Map-reduce partition columns:_col2 (type: string)
+ | | sort order:+
+ | | Statistics:Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
+ | | value expressions:_col3 (type: string), _col4 (type: string), _col6 (type: string)
+ | | Merge Join Operator [MERGEJOIN_105]
+ | | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | | keys:{"0":"_col1 (type: string)","1":"_col3 (type: string)"}
+ | | | outputColumnNames:["_col2","_col3","_col4","_col6"]
+ | | | Statistics:Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
+ | | |<-Map 12 [SIMPLE_EDGE]
+ | | | Reduce Output Operator [RS_32]
+ | | | key expressions:_col3 (type: string)
+ | | | Map-reduce partition columns:_col3 (type: string)
+ | | | sort order:+
+ | | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | | value expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string), _col4 (type: string)
+ | | | Select Operator [SEL_10]
+ | | | outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
+ | | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | | Filter Operator [FIL_99]
+ | | | predicate:((((((v3 = 'ssv3') and v2 is not null) and k1 is not null) and v1 is not null) and k2 is not null) and k3 is not null) (type: boolean)
+ | | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | | TableScan [TS_8]
+ | | | alias:ss
+ | | | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
+ | | |<-Map 7 [SIMPLE_EDGE]
+ | | Reduce Output Operator [RS_30]
+ | | key expressions:_col1 (type: string)
+ | | Map-reduce partition columns:_col1 (type: string)
+ | | sort order:+
+ | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ | | Select Operator [SEL_7]
+ | | outputColumnNames:["_col1"]
+ | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ | | Filter Operator [FIL_98]
+ | | predicate:((key = 'srcpartkey') and value is not null) (type: boolean)
+ | | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ | | TableScan [TS_5]
+ | | alias:srcpart
+ | | Statistics:Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
+ | |<-Reducer 16 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_47]
+ | key expressions:_col2 (type: string), _col4 (type: string)
+ | Map-reduce partition columns:_col2 (type: string), _col4 (type: string)
+ | sort order:++
+ | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col3 (type: string), _col5 (type: string)
+ | Merge Join Operator [MERGEJOIN_108]
+ | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | | outputColumnNames:["_col2","_col3","_col4","_col5"]
+ | | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ | |<-Map 15 [SIMPLE_EDGE]
+ | | Reduce Output Operator [RS_24]
+ | | key expressions:_col0 (type: string)
+ | | Map-reduce partition columns:_col0 (type: string)
+ | | sort order:+
+ | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | value expressions:_col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string)
+ | | Select Operator [SEL_19]
+ | | outputColumnNames:["_col0","_col2","_col3","_col4","_col5"]
+ | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | Filter Operator [FIL_102]
+ | | predicate:((((((v1 = 'srv1') and k1 is not null) and k2 is not null) and k3 is not null) and v2 is not null) and v3 is not null) (type: boolean)
+ | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | TableScan [TS_17]
+ | | alias:sr
+ | | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
+ | |<-Map 17 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_26]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | Select Operator [SEL_22]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | Filter Operator [FIL_103]
+ | predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_20]
+ | alias:d1
+ | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ |<-Reducer 2 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_56]
+ key expressions:_col1 (type: string), _col3 (type: string)
+ Map-reduce partition columns:_col1 (type: string), _col3 (type: string)
+ sort order:++
+ Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ value expressions:_col2 (type: string)
+ Merge Join Operator [MERGEJOIN_104]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"0":"_col0 (type: string)","1":"_col0 (type: string)"}
+ | outputColumnNames:["_col1","_col2","_col3"]
+ | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 1 [SIMPLE_EDGE]
+ | Reduce Output Operator [RS_51]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col1 (type: string), _col2 (type: string), _col3 (type: string)
+ | Select Operator [SEL_1]
+ | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
+ | Filter Operator [FIL_96]
+ | predicate:((k1 is not null and v2 is not null) and v3 is not null) (type: boolean)
+ | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_0]
+ | alias:cs
+ | Statistics:Num rows: 170 Data size: 5890 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 6 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_53]
+ key expressions:_col0 (type: string)
+ Map-reduce partition columns:_col0 (type: string)
+ sort order:+
Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Filter Operator [FIL_98]
- predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
+ Select Operator [SEL_4]
+ outputColumnNames:["_col0"]
Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- TableScan [TS_2]
- alias:d1
- Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator [FIL_97]
+ predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
+ Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_2]
+ alias:d1
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain
SELECT x.key, z.value, y.value
@@ -1522,199 +1525,202 @@ Stage-0
limit:100
Stage-1
Reducer 5
- File Output Operator [FS_69]
+ File Output Operator [FS_68]
compressed:false
Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
- Limit [LIM_68]
+ Limit [LIM_67]
Number of rows:100
Statistics:Num rows: 100 Data size: 1000 Basic stats: COMPLETE Column stats: NONE
- Select Operator [SEL_67]
+ Select Operator [SEL_66]
| outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
| Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
|<-Reducer 4 [SIMPLE_EDGE]
- Reduce Output Operator [RS_66]
+ Reduce Output Operator [RS_65]
key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string)
sort order:+++
Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
value expressions:_col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint)
- Group By Operator [GBY_64]
- | aggregations:["count(VALUE._col0)","count(VALUE._col1)","count(VALUE._col2)"]
- | keys:KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
- | outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
- | Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
- |<-Map 3 [SIMPLE_EDGE]
- Reduce Output Operator [RS_63]
- key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string)
- Map-reduce partition columns:_col0 (type: string), _col1 (type: string), _col2 (type: string)
- sort order:+++
- Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- value expressions:_col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint)
- Group By Operator [GBY_62]
- aggregations:["count(_col3)","count(_col4)","count(_col5)"]
- keys:_col0 (type: string), _col1 (type: string), _col2 (type: string)
- outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Select Operator [SEL_64]
+ outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator [GBY_63]
+ | aggregations:["count(VALUE._col0)","count(VALUE._col1)","count(VALUE._col2)"]
+ | keys:KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
+ | outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ | Statistics:Num rows: 402 Data size: 4276 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 3 [SIMPLE_EDGE]
+ Reduce Output Operator [RS_62]
+ key expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string)
+ Map-reduce partition columns:_col0 (type: string), _col1 (type: string), _col2 (type: string)
+ sort order:+++
Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- Select Operator [SEL_60]
+ value expressions:_col3 (type: bigint), _col4 (type: bigint), _col5 (type: bigint)
+ Group By Operator [GBY_61]
+ aggregations:["count(_col13)","count(_col21)","count(_col3)"]
+ keys:_col2 (type: string), _col12 (type: string), _col20 (type: string)
outputColumnNames:["_col0","_col1","_col2","_col3","_col4","_col5"]
Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- Map Join Operator [MAPJOIN_111]
- | condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"Map 2":"_col1 (type: string), _col3 (type: string)","Map 3":"_col15 (type: string), _col17 (type: string)"}
- | outputColumnNames:["_col2","_col3","_col12","_col13","_col20","_col21"]
- | Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
- |<-Map 2 [BROADCAST_EDGE]
- | Reduce Output Operator [RS_56]
- | key expressions:_col1 (type: string), _col3 (type: string)
- | Map-reduce partition columns:_col1 (type: string), _col3 (type: string)
- | sort order:++
- | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- | value expressions:_col2 (type: string)
- | Map Join Operator [MAPJOIN_105]
- | | condition map:[{"":"Inner Join 0 to 1"}]
- | | keys:{"Map 1":"_col0 (type: string)","Map 2":"_col0 (type: string)"}
- | | outputColumnNames:["_col1","_col2","_col3"]
- | | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- | |<-Map 1 [BROADCAST_EDGE]
- | | Reduce Output Operator [RS_51]
- | | key expressions:_col0 (type: string)
- | | Map-reduce partition columns:_col0 (type: string)
- | | sort order:+
- | | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
- | | value expressions:_col1 (type: string), _col2 (type: string), _col3 (type: string)
- | | Select Operator [SEL_1]
- | | outputColumnNames:["_col0","_col1","_col2","_col3"]
- | | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
- | | Filter Operator [FIL_97]
- | | predicate:((k1 is not null and v2 is not null) and v3 is not null) (type: boolean)
- | | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
- | | TableScan [TS_0]
- | | alias:cs
- | | Statistics:Num rows: 170 Data size: 5890 Basic stats: COMPLETE Column stats: NONE
- | |<-Select Operator [SEL_4]
- | outputColumnNames:["_col0"]
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | Filter Operator [FIL_98]
- | predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_2]
- | alias:d1
- | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- |<-Select Operator [SEL_49]
- outputColumnNames:["_col14","_col15","_col17","_col6","_col7"]
- Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
- Map Join Operator [MAPJOIN_110]
- | condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"Map 3":"_col4 (type: string), _col6 (type: string)","Map 10":"_col2 (type: string), _col4 (type: string)"}
- | outputColumnNames:["_col2","_col3","_col14","_col15","_col17"]
- | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
- |<-Map 10 [BROADCAST_EDGE]
- | Reduce Output Operator [RS_47]
- | key expressions:_col2 (type: string), _col4 (type: string)
- | Map-reduce partition columns:_col2 (type: string), _col4 (type: string)
- | sort order:++
- | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- | value expressions:_col3 (type: string), _col5 (type: string)
- | Map Join Operator [MAPJOIN_109]
- | | condition map:[{"":"Inner Join 0 to 1"}]
- | | keys:{"Map 9":"_col0 (type: string)","Map 10":"_col0 (type: string)"}
- | | outputColumnNames:["_col2","_col3","_col4","_col5"]
- | | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- | |<-Map 9 [BROADCAST_EDGE]
- | | Reduce Output Operator [RS_24]
- | | key expressions:_col0 (type: string)
- | | Map-reduce partition columns:_col0 (type: string)
- | | sort order:+
- | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | value expressions:_col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string)
- | | Select Operator [SEL_19]
- | | outputColumnNames:["_col0","_col2","_col3","_col4","_col5"]
- | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | Filter Operator [FIL_103]
- | | predicate:((((((v1 = 'srv1') and k1 is not null) and k2 is not null) and k3 is not null) and v2 is not null) and v3 is not null) (type: boolean)
- | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | | TableScan [TS_17]
- | | alias:sr
- | | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
- | |<-Select Operator [SEL_22]
- | outputColumnNames:["_col0"]
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | Filter Operator [FIL_104]
- | predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_20]
- | alias:d1
- | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- |<-Map Join Operator [MAPJOIN_108]
+ Select Operator [SEL_60]
+ outputColumnNames:["_col2","_col12","_col20","_col13","_col21","_col3"]
+ Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator [MAPJOIN_110]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"Map 2":"_col1 (type: string), _col3 (type: string)","Map 3":"_col15 (type: string), _col17 (type: string)"}
+ | outputColumnNames:["_col2","_col3","_col12","_col13","_col20","_col21"]
+ | Statistics:Num rows: 804 Data size: 8552 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 2 [BROADCAST_EDGE]
+ | Reduce Output Operator [RS_56]
+ | key expressions:_col1 (type: string), _col3 (type: string)
+ | Map-reduce partition columns:_col1 (type: string), _col3 (type: string)
+ | sort order:++
+ | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col2 (type: string)
+ | Map Join Operator [MAPJOIN_104]
+ | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | keys:{"Map 1":"_col0 (type: string)","Map 2":"_col0 (type: string)"}
+ | | outputColumnNames:["_col1","_col2","_col3"]
+ | | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ | |<-Map 1 [BROADCAST_EDGE]
+ | | Reduce Output Operator [RS_51]
+ | | key expressions:_col0 (type: string)
+ | | Map-reduce partition columns:_col0 (type: string)
+ | | sort order:+
+ | | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
+ | | value expressions:_col1 (type: string), _col2 (type: string), _col3 (type: string)
+ | | Select Operator [SEL_1]
+ | | outputColumnNames:["_col0","_col1","_col2","_col3"]
+ | | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
+ | | Filter Operator [FIL_96]
+ | | predicate:((k1 is not null and v2 is not null) and v3 is not null) (type: boolean)
+ | | Statistics:Num rows: 22 Data size: 762 Basic stats: COMPLETE Column stats: NONE
+ | | TableScan [TS_0]
+ | | alias:cs
+ | | Statistics:Num rows: 170 Data size: 5890 Basic stats: COMPLETE Column stats: NONE
+ | |<-Select Operator [SEL_4]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | Filter Operator [FIL_97]
+ | predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_2]
+ | alias:d1
+ | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ |<-Select Operator [SEL_49]
+ outputColumnNames:["_col14","_col15","_col17","_col6","_col7"]
+ Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator [MAPJOIN_109]
| condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"Map 3":"_col3 (type: string)","Map 8":"_col1 (type: string)"}
- | outputColumnNames:["_col2","_col3","_col4","_col6"]
- | Statistics:Num rows: 665 Data size: 7069 Basic stats: COMPLETE Column stats: NONE
- |<-Map 8 [BROADCAST_EDGE]
- | Reduce Output Operator [RS_42]
- | key expressions:_col1 (type: string)
- | Map-reduce partition columns:_col1 (type: string)
- | sort order:+
- | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
- | Select Operator [SEL_16]
- | outputColumnNames:["_col1"]
- | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
- | Filter Operator [FIL_102]
- | predicate:((key = 'src1key') and value is not null) (type: boolean)
- | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_14]
- | alias:src1
- | Statistics:Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE
+ | keys:{"Map 3":"_col4 (type: string), _col6 (type: string)","Map 10":"_col2 (type: string), _col4 (type: string)"}
+ | outputColumnNames:["_col2","_col3","_col14","_col15","_col17"]
+ | Statistics:Num rows: 731 Data size: 7775 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 10 [BROADCAST_EDGE]
+ | Reduce Output Operator [RS_47]
+ | key expressions:_col2 (type: string), _col4 (type: string)
+ | Map-reduce partition columns:_col2 (type: string), _col4 (type: string)
+ | sort order:++
+ | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col3 (type: string), _col5 (type: string)
+ | Map Join Operator [MAPJOIN_108]
+ | | condition map:[{"":"Inner Join 0 to 1"}]
+ | | keys:{"Map 9":"_col0 (type: string)","Map 10":"_col0 (type: string)"}
+ | | outputColumnNames:["_col2","_col3","_col4","_col5"]
+ | | Statistics:Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ | |<-Map 9 [BROADCAST_EDGE]
+ | | Reduce Output Operator [RS_24]
+ | | key expressions:_col0 (type: string)
+ | | Map-reduce partition columns:_col0 (type: string)
+ | | sort order:+
+ | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | value expressions:_col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string)
+ | | Select Operator [SEL_19]
+ | | outputColumnNames:["_col0","_col2","_col3","_col4","_col5"]
+ | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | Filter Operator [FIL_102]
+ | | predicate:((((((v1 = 'srv1') and k1 is not null) and k2 is not null) and k3 is not null) and v2 is not null) and v3 is not null) (type: boolean)
+ | | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | | TableScan [TS_17]
+ | | alias:sr
+ | | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
+ | |<-Select Operator [SEL_22]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | Filter Operator [FIL_103]
+ | predicate:((value) IN ('2000Q1', '2000Q2', '2000Q3') and key is not null) (type: boolean)
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_20]
+ | alias:d1
+ | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
|<-Map Join Operator [MAPJOIN_107]
| condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"Map 3":"_col2 (type: string)","Map 7":"_col0 (type: string)"}
+ | keys:{"Map 3":"_col3 (type: string)","Map 8":"_col1 (type: string)"}
| outputColumnNames:["_col2","_col3","_col4","_col6"]
- | Statistics:Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
- |<-Map 7 [BROADCAST_EDGE]
- | Reduce Output Operator [RS_37]
- | key expressions:_col0 (type: string)
- | Map-reduce partition columns:_col0 (type: string)
+ | Statistics:Num rows: 665 Data size: 7069 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 8 [BROADCAST_EDGE]
+ | Reduce Output Operator [RS_42]
+ | key expressions:_col1 (type: string)
+ | Map-reduce partition columns:_col1 (type: string)
| sort order:+
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | Select Operator [SEL_13]
- | outputColumnNames:["_col0"]
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
+ | Select Operator [SEL_16]
+ | outputColumnNames:["_col1"]
+ | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
| Filter Operator [FIL_101]
- | predicate:((value = 'd1value') and key is not null) (type: boolean)
- | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_11]
- | alias:d1
- | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ | predicate:((key = 'src1key') and value is not null) (type: boolean)
+ | Statistics:Num rows: 6 Data size: 45 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_14]
+ | alias:src1
+ | Statistics:Num rows: 25 Data size: 191 Basic stats: COMPLETE Column stats: NONE
|<-Map Join Operator [MAPJOIN_106]
| condition map:[{"":"Inner Join 0 to 1"}]
- | keys:{"Map 3":"_col1 (type: string)","Map 6":"_col3 (type: string)"}
+ | keys:{"Map 3":"_col2 (type: string)","Map 7":"_col0 (type: string)"}
| outputColumnNames:["_col2","_col3","_col4","_col6"]
- | Statistics:Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- |<-Map 6 [BROADCAST_EDGE]
- | Reduce Output Operator [RS_32]
- | key expressions:_col3 (type: string)
- | Map-reduce partition columns:_col3 (type: string)
+ | Statistics:Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 7 [BROADCAST_EDGE]
+ | Reduce Output Operator [RS_37]
+ | key expressions:_col0 (type: string)
+ | Map-reduce partition columns:_col0 (type: string)
| sort order:+
- | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | value expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string), _col4 (type: string)
- | Select Operator [SEL_10]
- | outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
- | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | Select Operator [SEL_13]
+ | outputColumnNames:["_col0"]
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
| Filter Operator [FIL_100]
- | predicate:((((((v3 = 'ssv3') and v2 is not null) and k1 is not null) and v1 is not null) and k2 is not null) and k3 is not null) (type: boolean)
- | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
- | TableScan [TS_8]
- | alias:ss
- | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
- |<-Select Operator [SEL_7]
- outputColumnNames:["_col1"]
- Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator [FIL_99]
- predicate:((key = 'srcpartkey') and value is not null) (type: boolean)
+ | predicate:((value = 'd1value') and key is not null) (type: boolean)
+ | Statistics:Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_11]
+ | alias:d1
+ | Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ |<-Map Join Operator [MAPJOIN_105]
+ | condition map:[{"":"Inner Join 0 to 1"}]
+ | keys:{"Map 3":"_col1 (type: string)","Map 6":"_col3 (type: string)"}
+ | outputColumnNames:["_col2","_col3","_col4","_col6"]
+ | Statistics:Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
+ |<-Map 6 [BROADCAST_EDGE]
+ | Reduce Output Operator [RS_32]
+ | key expressions:_col3 (type: string)
+ | Map-reduce partition columns:_col3 (type: string)
+ | sort order:+
+ | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | value expressions:_col0 (type: string), _col1 (type: string), _col2 (type: string), _col4 (type: string)
+ | Select Operator [SEL_10]
+ | outputColumnNames:["_col0","_col1","_col2","_col3","_col4"]
+ | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | Filter Operator [FIL_99]
+ | predicate:((((((v3 = 'ssv3') and v2 is not null) and k1 is not null) and v1 is not null) and k2 is not null) and k3 is not null) (type: boolean)
+ | Statistics:Num rows: 2 Data size: 69 Basic stats: COMPLETE Column stats: NONE
+ | TableScan [TS_8]
+ | alias:ss
+ | Statistics:Num rows: 85 Data size: 2945 Basic stats: COMPLETE Column stats: NONE
+ |<-Select Operator [SEL_7]
+ outputColumnNames:["_col1"]
Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- TableScan [TS_5]
- alias:srcpart
- Statistics:Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator [FIL_98]
+ predicate:((key = 'srcpartkey') and value is not null) (type: boolean)
+ Statistics:Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ TableScan [TS_5]
+ alias:srcpart
+ Statistics:Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain
SELECT x.key, z.value, y.value
[02/51] [abbrv] hive git commit: HIVE-11886 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-11886 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/f324305a
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/f324305a
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/f324305a
Branch: refs/heads/master
Commit: f324305a71ac31faa568b8a0078b1e9b217a3570
Parents: 79c7031 e9c8d7c
Author: Sergey Shelukhin <se...@apache.org>
Authored: Fri Sep 18 13:35:36 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Fri Sep 18 13:35:36 2015 -0700
----------------------------------------------------------------------
.../benchmark/serde/LazySimpleSerDeBench.java | 453 ++++
.../hive/ql/security/FolderPermissionBase.java | 17 +-
.../test/resources/testconfiguration.properties | 2 +
.../org/apache/hadoop/hive/ql/QTestUtil.java | 9 +-
.../llap/io/encoded/OrcEncodedDataReader.java | 3 +-
.../hadoop/hive/metastore/HiveMetaStore.java | 6 +
pom.xml | 22 +-
.../org/apache/hadoop/hive/ql/exec/DDLTask.java | 269 +-
.../apache/hadoop/hive/ql/exec/MoveTask.java | 19 +-
.../ql/exec/persistence/PTFRowContainer.java | 14 +-
.../hive/ql/exec/persistence/RowContainer.java | 12 +-
.../ql/exec/tez/tools/KeyValuesInputMerger.java | 1 -
.../ql/exec/vector/VectorizationContext.java | 10 +-
.../hadoop/hive/ql/hooks/LineageLogger.java | 95 +-
.../hadoop/hive/ql/io/orc/OrcInputFormat.java | 150 +-
.../apache/hadoop/hive/ql/io/orc/OrcSerde.java | 1 +
.../apache/hadoop/hive/ql/io/orc/OrcStruct.java | 2 +-
.../hadoop/hive/ql/io/orc/RecordReaderImpl.java | 4 +-
.../hive/ql/io/parquet/ProjectionPusher.java | 3 +-
.../hive/ql/io/sarg/ConvertAstToSearchArg.java | 4 +
.../apache/hadoop/hive/ql/lib/RuleRegExp.java | 61 +-
.../ql/optimizer/ColumnPrunerProcFactory.java | 3 +
.../hive/ql/optimizer/ConvertJoinMapJoin.java | 4 +-
.../calcite/reloperators/HiveBetween.java | 75 +
.../optimizer/calcite/reloperators/HiveIn.java | 41 +
.../rules/HiveAggregateProjectMergeRule.java | 151 ++
.../calcite/rules/HivePreFilteringRule.java | 37 +-
.../calcite/rules/HiveRelFieldTrimmer.java | 145 +-
.../translator/PlanModifierForASTConv.java | 4 +-
.../translator/SqlFunctionConverter.java | 16 +-
.../hive/ql/optimizer/lineage/LineageCtx.java | 8 +-
.../hadoop/hive/ql/parse/CalcitePlanner.java | 11 +-
.../hive/ql/parse/DDLSemanticAnalyzer.java | 17 +
.../apache/hadoop/hive/ql/parse/HiveParser.g | 7 +-
.../apache/hadoop/hive/ql/parse/QBSubQuery.java | 7 -
.../hadoop/hive/ql/parse/SemanticAnalyzer.java | 2 +
.../hive/ql/parse/SemanticAnalyzerFactory.java | 2 +
.../hadoop/hive/ql/parse/SubQueryUtils.java | 11 -
.../org/apache/hadoop/hive/ql/plan/DDLWork.java | 21 +
.../hadoop/hive/ql/plan/HiveOperation.java | 1 +
.../hive/ql/plan/ShowCreateDatabaseDesc.java | 94 +
.../authorization/plugin/HiveOperationType.java | 1 +
.../plugin/sqlstd/Operation2Privilege.java | 2 +
.../org/apache/hadoop/hive/ql/udf/UDFJson.java | 2 +
.../hive/ql/udf/generic/GenericUDAFMax.java | 16 +-
.../exec/persistence/TestPTFRowContainer.java | 31 +-
.../hadoop/hive/ql/io/orc/TestOrcStruct.java | 2 +
.../clientpositive/drop_table_with_index.q | 35 +
.../queries/clientpositive/exchgpartition2lel.q | 32 +
ql/src/test/queries/clientpositive/lineage3.q | 26 +
.../test/queries/clientpositive/load_orc_part.q | 5 +
.../clientpositive/show_create_database.q | 3 +
.../queries/clientpositive/subquery_views.q | 22 +-
.../queries/clientpositive/vector_char_cast.q | 9 +
.../queries/clientpositive/windowing_udaf.q | 4 +
.../subquery_exists_implicit_gby.q.out | 8 +-
.../subquery_nested_subquery.q.out | 4 +-
.../subquery_notexists_implicit_gby.q.out | 8 +-
.../subquery_windowing_corr.q.out | 7 +-
.../alter_partition_coltype.q.out | 8 +-
.../clientpositive/annotate_stats_groupby.q.out | 106 +-
.../annotate_stats_groupby2.q.out | 28 +-
.../results/clientpositive/auto_join18.q.out | 12 +-
.../auto_join18_multi_distinct.q.out | 12 +-
.../results/clientpositive/auto_join27.q.out | 18 +-
.../results/clientpositive/auto_join32.q.out | 4 +-
.../clientpositive/binarysortable_1.q.out | Bin 4329 -> 4325 bytes
.../clientpositive/correlationoptimizer2.q.out | 220 +-
.../clientpositive/correlationoptimizer6.q.out | 232 +-
ql/src/test/results/clientpositive/count.q.out | 14 +-
.../results/clientpositive/ctas_colname.q.out | 52 +-
.../test/results/clientpositive/database.q.out | 2 +-
.../clientpositive/decimal_precision.q.out | 4 +-
.../results/clientpositive/decimal_udf.q.out | 30 +-
.../results/clientpositive/distinct_stats.q.out | 14 +-
.../clientpositive/drop_table_with_index.q.out | 152 ++
.../dynpart_sort_opt_vectorization.q.out | 105 +-
.../dynpart_sort_optimization.q.out | 105 +-
...ryption_select_read_only_encrypted_tbl.q.out | 4 +-
.../clientpositive/exchgpartition2lel.q.out | 182 ++
.../clientpositive/explain_logical.q.out | 78 +-
.../clientpositive/fetch_aggregation.q.out | 4 +-
.../test/results/clientpositive/gby_star.q.out | 54 +-
.../test/results/clientpositive/groupby12.q.out | 6 +-
.../results/clientpositive/groupby5_map.q.out | 4 +-
.../clientpositive/groupby5_map_skew.q.out | 4 +-
.../results/clientpositive/groupby_cube1.q.out | 12 +-
.../groupby_distinct_samekey.q.out | 6 +-
.../clientpositive/groupby_grouping_sets2.q.out | 10 +-
.../clientpositive/groupby_grouping_sets3.q.out | 12 +-
.../clientpositive/groupby_grouping_sets5.q.out | 8 +-
.../clientpositive/groupby_grouping_sets6.q.out | 8 +-
.../clientpositive/groupby_position.q.out | 36 +-
.../clientpositive/groupby_resolution.q.out | 60 +-
.../clientpositive/groupby_rollup1.q.out | 12 +-
.../clientpositive/groupby_sort_10.q.out | 8 +-
.../clientpositive/groupby_sort_11.q.out | 10 +-
.../results/clientpositive/groupby_sort_8.q.out | 12 +-
ql/src/test/results/clientpositive/having.q.out | 62 +-
.../test/results/clientpositive/having2.q.out | 12 +-
.../clientpositive/index_auto_mult_tables.q.out | 12 +-
.../clientpositive/index_auto_self_join.q.out | 12 +-
.../clientpositive/index_auto_update.q.out | 6 +-
.../index_bitmap_auto_partitioned.q.out | 6 +-
.../index_bitmap_compression.q.out | 6 +-
.../infer_bucket_sort_dyn_part.q.out | 4 +-
.../infer_bucket_sort_map_operators.q.out | 4 +-
ql/src/test/results/clientpositive/join18.q.out | 12 +-
.../clientpositive/join18_multi_distinct.q.out | 12 +-
ql/src/test/results/clientpositive/join31.q.out | 36 +-
.../limit_partition_metadataonly.q.out | 4 +-
.../results/clientpositive/limit_pushdown.q.out | 36 +-
.../test/results/clientpositive/lineage2.q.out | 2 +-
.../test/results/clientpositive/lineage3.q.out | 72 +-
.../list_bucket_query_multiskew_3.q.out | 2 +-
.../results/clientpositive/load_orc_part.q.out | 26 +
.../clientpositive/mapjoin_mapjoin.q.out | 32 +-
.../clientpositive/metadata_only_queries.q.out | 4 +-
.../results/clientpositive/metadataonly1.q.out | 112 +-
.../results/clientpositive/multiMapJoin2.q.out | 226 +-
.../nonblock_op_deduplicate.q.out | 8 +-
.../results/clientpositive/nonmr_fetch.q.out | 14 +-
.../clientpositive/partition_multilevels.q.out | 8 +-
.../test/results/clientpositive/ppd_gby.q.out | 12 +-
.../test/results/clientpositive/ppd_gby2.q.out | 60 +-
.../clientpositive/ppd_join_filter.q.out | 98 +-
.../ql_rewrite_gbtoidx_cbo_1.q.out | 168 +-
.../ql_rewrite_gbtoidx_cbo_2.q.out | 94 +-
.../reduce_deduplicate_extended.q.out | 32 +-
.../clientpositive/selectDistinctStar.q.out | 44 +-
.../clientpositive/show_create_database.q.out | 19 +
.../clientpositive/spark/auto_join18.q.out | 10 +-
.../spark/auto_join18_multi_distinct.q.out | 12 +-
.../clientpositive/spark/auto_join27.q.out | 18 +-
.../clientpositive/spark/auto_join32.q.out | 53 +-
.../results/clientpositive/spark/count.q.out | 14 +-
.../clientpositive/spark/groupby5_map.q.out | 4 +-
.../spark/groupby5_map_skew.q.out | 4 +-
.../clientpositive/spark/groupby_cube1.q.out | 12 +-
.../clientpositive/spark/groupby_position.q.out | 18 +-
.../spark/groupby_resolution.q.out | 60 +-
.../clientpositive/spark/groupby_rollup1.q.out | 12 +-
.../results/clientpositive/spark/having.q.out | 62 +-
.../spark/infer_bucket_sort_map_operators.q.out | 4 +-
.../results/clientpositive/spark/join18.q.out | 10 +-
.../spark/join18_multi_distinct.q.out | 12 +-
.../results/clientpositive/spark/join31.q.out | 36 +-
.../spark/limit_partition_metadataonly.q.out | 4 +-
.../clientpositive/spark/limit_pushdown.q.out | 34 +-
.../clientpositive/spark/mapjoin_mapjoin.q.out | 24 +-
.../spark/metadata_only_queries.q.out | 4 +-
.../clientpositive/spark/ppd_join_filter.q.out | 90 +-
.../spark/ql_rewrite_gbtoidx_cbo_1.q.out | 168 +-
.../clientpositive/spark/stats_only_null.q.out | 8 +-
.../clientpositive/spark/subquery_in.q.out | 36 +-
.../results/clientpositive/spark/union11.q.out | 42 +-
.../results/clientpositive/spark/union14.q.out | 28 +-
.../results/clientpositive/spark/union15.q.out | 28 +-
.../results/clientpositive/spark/union28.q.out | 4 +-
.../results/clientpositive/spark/union30.q.out | 4 +-
.../results/clientpositive/spark/union33.q.out | 8 +-
.../results/clientpositive/spark/union5.q.out | 34 +-
.../results/clientpositive/spark/union7.q.out | 28 +-
.../clientpositive/spark/union_remove_21.q.out | 4 +-
.../spark/vector_count_distinct.q.out | 4 +-
.../spark/vector_decimal_aggregate.q.out | 12 +-
.../spark/vector_distinct_2.q.out | 28 +-
.../clientpositive/spark/vector_groupby_3.q.out | 30 +-
.../spark/vector_mapjoin_reduce.q.out | 36 +-
.../clientpositive/spark/vector_orderby_5.q.out | 6 +-
.../clientpositive/spark/vectorization_0.q.out | 16 +-
.../clientpositive/spark/vectorization_13.q.out | 32 +-
.../clientpositive/spark/vectorization_15.q.out | 16 +-
.../clientpositive/spark/vectorization_16.q.out | 16 +-
.../clientpositive/spark/vectorization_9.q.out | 16 +-
.../spark/vectorization_pushdown.q.out | 4 +-
.../spark/vectorization_short_regress.q.out | 74 +-
.../spark/vectorized_nested_mapjoin.q.out | 17 +-
.../spark/vectorized_timestamp_funcs.q.out | 12 +-
.../clientpositive/stats_only_null.q.out | 8 +-
.../results/clientpositive/stats_ppr_all.q.out | 16 +-
.../subq_where_serialization.q.out | 18 +-
.../clientpositive/subquery_exists_having.q.out | 48 +-
.../results/clientpositive/subquery_in.q.out | 36 +-
.../clientpositive/subquery_in_having.q.out | 260 +-
.../clientpositive/subquery_notexists.q.out | 18 +-
.../subquery_notexists_having.q.out | 26 +-
.../results/clientpositive/subquery_notin.q.out | 24 +-
.../subquery_notin_having.q.java1.7.out | 50 +-
.../subquery_unqualcolumnrefs.q.out | 74 +-
.../results/clientpositive/subquery_views.q.out | 124 +-
.../test/results/clientpositive/tez/count.q.out | 14 +-
.../tez/dynamic_partition_pruning.q.out | 88 +-
.../tez/dynpart_sort_opt_vectorization.q.out | 90 +-
.../tez/dynpart_sort_optimization.q.out | 89 +-
.../clientpositive/tez/explainuser_1.q.out | 2319 +++++++++---------
.../clientpositive/tez/explainuser_2.q.out | 782 +++---
.../results/clientpositive/tez/having.q.out | 62 +-
.../clientpositive/tez/limit_pushdown.q.out | 34 +-
.../clientpositive/tez/mapjoin_mapjoin.q.out | 24 +-
.../tez/metadata_only_queries.q.out | 4 +-
.../clientpositive/tez/metadataonly1.q.out | 44 +-
.../test/results/clientpositive/tez/mrr.q.out | 94 +-
.../clientpositive/tez/selectDistinctStar.q.out | 44 +-
.../tez/show_create_database.q.out | 19 +
.../clientpositive/tez/stats_only_null.q.out | 8 +-
.../clientpositive/tez/subquery_in.q.out | 36 +-
.../results/clientpositive/tez/tez_dml.q.out | 6 +-
.../results/clientpositive/tez/union5.q.out | 44 +-
.../results/clientpositive/tez/union7.q.out | 28 +-
.../clientpositive/tez/unionDistinct_1.q.out | 8 +-
.../clientpositive/tez/vector_aggregate_9.q.out | 4 +-
.../tez/vector_binary_join_groupby.q.out | 4 +-
.../clientpositive/tez/vector_char_cast.q.out | 35 +
.../tez/vector_count_distinct.q.out | 4 +-
.../tez/vector_decimal_aggregate.q.out | 12 +-
.../tez/vector_decimal_precision.q.out | 4 +-
.../clientpositive/tez/vector_decimal_udf.q.out | 30 +-
.../clientpositive/tez/vector_distinct_2.q.out | 28 +-
.../clientpositive/tez/vector_groupby_3.q.out | 30 +-
.../tez/vector_groupby_reduce.q.out | 8 +-
.../tez/vector_grouping_sets.q.out | 8 +-
.../tez/vector_mapjoin_reduce.q.out | 36 +-
.../clientpositive/tez/vector_orderby_5.q.out | 6 +-
.../clientpositive/tez/vector_outer_join2.q.out | 20 +-
.../tez/vector_partition_diff_num_cols.q.out | 20 +-
.../tez/vector_partitioned_date_time.q.out | 12 +-
.../tez/vector_reduce_groupby_decimal.q.out | 24 +-
.../clientpositive/tez/vectorization_0.q.out | 16 +-
.../clientpositive/tez/vectorization_13.q.out | 32 +-
.../clientpositive/tez/vectorization_15.q.out | 16 +-
.../clientpositive/tez/vectorization_16.q.out | 16 +-
.../clientpositive/tez/vectorization_9.q.out | 16 +-
.../tez/vectorization_limit.q.out | 14 +-
.../tez/vectorization_pushdown.q.out | 4 +-
.../tez/vectorization_short_regress.q.out | 74 +-
.../tez/vectorized_distinct_gby.q.out | 8 +-
.../vectorized_dynamic_partition_pruning.q.out | 88 +-
.../tez/vectorized_nested_mapjoin.q.out | 18 +-
.../clientpositive/tez/vectorized_parquet.q.out | 6 +-
.../tez/vectorized_timestamp_funcs.q.out | 12 +-
ql/src/test/results/clientpositive/udf8.q.out | 4 +-
.../test/results/clientpositive/udf_count.q.out | 16 +-
.../test/results/clientpositive/union11.q.out | 70 +-
.../test/results/clientpositive/union14.q.out | 32 +-
.../test/results/clientpositive/union15.q.out | 38 +-
.../test/results/clientpositive/union28.q.out | 8 +-
.../test/results/clientpositive/union30.q.out | 8 +-
.../test/results/clientpositive/union33.q.out | 8 +-
ql/src/test/results/clientpositive/union5.q.out | 48 +-
ql/src/test/results/clientpositive/union7.q.out | 32 +-
.../clientpositive/unionDistinct_1.q.out | 8 +-
.../clientpositive/union_remove_21.q.out | 8 +-
.../clientpositive/vector_aggregate_9.q.out | 4 +-
.../vector_aggregate_without_gby.q.out | 4 +-
.../vector_binary_join_groupby.q.out | 4 +-
.../clientpositive/vector_char_cast.q.out | 35 +
.../clientpositive/vector_count_distinct.q.out | 6 +-
.../vector_decimal_aggregate.q.out | 12 +-
.../vector_decimal_precision.q.out | 4 +-
.../clientpositive/vector_decimal_udf.q.out | 30 +-
.../clientpositive/vector_distinct_2.q.out | 28 +-
.../clientpositive/vector_groupby_3.q.out | 30 +-
.../clientpositive/vector_groupby_reduce.q.out | 8 +-
.../clientpositive/vector_grouping_sets.q.out | 8 +-
.../clientpositive/vector_left_outer_join.q.out | 8 +-
.../clientpositive/vector_mapjoin_reduce.q.out | 36 +-
.../clientpositive/vector_orderby_5.q.out | 6 +-
.../clientpositive/vector_outer_join1.q.out | 8 +-
.../clientpositive/vector_outer_join2.q.out | 28 +-
.../clientpositive/vector_outer_join3.q.out | 24 +-
.../clientpositive/vector_outer_join4.q.out | 8 +-
.../clientpositive/vector_outer_join5.q.out | 48 +-
.../vector_partition_diff_num_cols.q.out | 20 +-
.../vector_partitioned_date_time.q.out | 12 +-
.../vector_reduce_groupby_decimal.q.out | 24 +-
.../clientpositive/vectorization_0.q.out | 16 +-
.../clientpositive/vectorization_13.q.out | 32 +-
.../clientpositive/vectorization_15.q.out | 16 +-
.../clientpositive/vectorization_16.q.out | 16 +-
.../clientpositive/vectorization_9.q.out | 16 +-
.../clientpositive/vectorization_limit.q.out | 16 +-
.../clientpositive/vectorization_pushdown.q.out | 4 +-
.../vectorization_short_regress.q.out | 74 +-
.../vectorized_distinct_gby.q.out | 12 +-
.../vectorized_nested_mapjoin.q.out | 26 +-
.../clientpositive/vectorized_parquet.q.out | 6 +-
.../vectorized_parquet_types.q.out | 6 +-
.../vectorized_timestamp_funcs.q.out | 12 +-
.../results/clientpositive/windowing_udaf.q.out | 12 +
.../hive/serde2/ColumnProjectionUtils.java | 22 +
.../hadoop/hive/serde2/lazy/LazyByte.java | 4 +
.../hadoop/hive/serde2/lazy/LazyDouble.java | 4 +
.../hadoop/hive/serde2/lazy/LazyFloat.java | 4 +
.../hadoop/hive/serde2/lazy/LazyInteger.java | 4 +
.../hadoop/hive/serde2/lazy/LazyLong.java | 4 +
.../hadoop/hive/serde2/lazy/LazyShort.java | 4 +
.../hadoop/hive/serde2/lazy/LazyUtils.java | 28 +
.../org/apache/hive/service/cli/Column.java | 2 +-
.../org/apache/hive/service/cli/TestColumn.java | 129 +
.../hive/ql/io/sarg/SearchArgumentFactory.java | 5 +-
.../hive/ql/io/sarg/SearchArgumentImpl.java | 7 +-
302 files changed, 7055 insertions(+), 5389 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
----------------------------------------------------------------------
diff --cc llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
index 4f7bb78,0000000..c934f39
mode 100644,000000..100644
--- a/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
+++ b/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
@@@ -1,949 -1,0 +1,950 @@@
+package org.apache.hadoop.hive.llap.io.encoded;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.CallableWithNdc;
+import org.apache.hadoop.hive.common.Pool;
+import org.apache.hadoop.hive.common.Pool.PoolObjectHelper;
+import org.apache.hadoop.hive.common.io.DataCache;
+import org.apache.hadoop.hive.common.io.Allocator;
+import org.apache.hadoop.hive.common.io.encoded.EncodedColumnBatch.ColumnStreamData;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.common.io.DiskRangeList;
+import org.apache.hadoop.hive.common.io.encoded.MemoryBuffer;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.ConsumerFeedback;
+import org.apache.hadoop.hive.llap.DebugUtils;
+import org.apache.hadoop.hive.llap.cache.Cache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache;
+import org.apache.hadoop.hive.llap.cache.LowLevelCache.Priority;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters;
+import org.apache.hadoop.hive.llap.counters.QueryFragmentCounters.Counter;
+import org.apache.hadoop.hive.llap.io.api.impl.LlapIoImpl;
+import org.apache.hadoop.hive.llap.io.decode.OrcEncodedDataConsumer;
+import org.apache.hadoop.hive.llap.io.metadata.OrcFileMetadata;
+import org.apache.hadoop.hive.llap.io.metadata.OrcMetadataCache;
+import org.apache.hadoop.hive.llap.io.metadata.OrcStripeMetadata;
+import org.apache.hadoop.hive.ql.exec.DDLTask;
+import org.apache.hadoop.hive.ql.io.AcidUtils;
+import org.apache.hadoop.hive.ql.io.HdfsUtils;
+import org.apache.hadoop.hive.ql.io.orc.CompressionKind;
+import org.apache.hadoop.hive.ql.io.orc.DataReader;
+import org.apache.hadoop.hive.ql.io.orc.MetadataReader;
+import org.apache.hadoop.hive.ql.io.orc.OrcFile;
+import org.apache.hadoop.hive.ql.io.orc.OrcFile.ReaderOptions;
+import org.apache.hadoop.hive.ql.io.orc.OrcConf;
+import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat;
+import org.apache.hadoop.hive.ql.io.orc.OrcProto;
+import org.apache.hadoop.hive.ql.io.orc.OrcSplit;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Reader;
+import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl;
+import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.SargApplier;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Consumer;
+import org.apache.hadoop.hive.ql.io.orc.encoded.EncodedOrcFile;
+import org.apache.hadoop.hive.ql.io.orc.encoded.EncodedReader;
+import org.apache.hadoop.hive.ql.io.orc.encoded.OrcBatchKey;
+import org.apache.hadoop.hive.ql.io.orc.encoded.OrcCacheKey;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Reader.OrcEncodedColumnBatch;
+import org.apache.hadoop.hive.ql.io.orc.encoded.Reader.PoolFactory;
+import org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils;
+import org.apache.hadoop.hive.ql.io.orc.StripeInformation;
+import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
+import org.apache.hadoop.mapred.FileSplit;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hive.common.util.FixedSizedObjectPool;
+
+/**
+ * This produces EncodedColumnBatch via ORC EncodedDataImpl.
+ * It serves as Consumer for EncodedColumnBatch too, for the high-level cache scenario where
+ * it inserts itself into the pipeline to put the data in cache, before passing it to the real
+ * consumer. It also serves as ConsumerFeedback that receives processed EncodedColumnBatch-es.
+ */
+public class OrcEncodedDataReader extends CallableWithNdc<Void>
+ implements ConsumerFeedback<OrcEncodedColumnBatch>, Consumer<OrcEncodedColumnBatch> {
+ private static final Log LOG = LogFactory.getLog(OrcEncodedDataReader.class);
+ public static final FixedSizedObjectPool<ColumnStreamData> CSD_POOL =
+ new FixedSizedObjectPool<>(8192, new PoolObjectHelper<ColumnStreamData>() {
+ @Override
+ public ColumnStreamData create() {
+ return new ColumnStreamData();
+ }
+ @Override
+ public void resetBeforeOffer(ColumnStreamData t) {
+ t.reset();
+ }
+ });
+ public static final FixedSizedObjectPool<OrcEncodedColumnBatch> ECB_POOL =
+ new FixedSizedObjectPool<>(1024, new PoolObjectHelper<OrcEncodedColumnBatch>() {
+ @Override
+ public OrcEncodedColumnBatch create() {
+ return new OrcEncodedColumnBatch();
+ }
+ @Override
+ public void resetBeforeOffer(OrcEncodedColumnBatch t) {
+ t.reset();
+ }
+ });
+ private static final PoolFactory POOL_FACTORY = new PoolFactory() {
+ @Override
+ public <T> Pool<T> createPool(int size, PoolObjectHelper<T> helper) {
+ return new FixedSizedObjectPool<>(size, helper);
+ }
+
+ @Override
+ public Pool<ColumnStreamData> createColumnStreamDataPool() {
+ return CSD_POOL;
+ }
+
+ @Override
+ public Pool<OrcEncodedColumnBatch> createEncodedColumnBatchPool() {
+ return ECB_POOL;
+ }
+ };
+
+ private final OrcMetadataCache metadataCache;
+ private final LowLevelCache lowLevelCache;
+ private final Configuration conf;
+ private final Cache<OrcCacheKey> cache;
+ private final FileSplit split;
+ private List<Integer> columnIds;
+ private final SearchArgument sarg;
+ private final String[] columnNames;
+ private final OrcEncodedDataConsumer consumer;
+ private final QueryFragmentCounters counters;
+
+ // Read state.
+ private int stripeIxFrom;
+ private OrcFileMetadata fileMetadata;
+ private Reader orcReader;
+ private MetadataReader metadataReader;
+ private EncodedReader stripeReader;
+ private long fileId;
+ private FileSystem fs;
+ /**
+ * readState[stripeIx'][colIx'] => boolean array (could be a bitmask) of rg-s that need to be
+ * read. Contains only stripes that are read, and only columns included. null => read all RGs.
+ */
+ private boolean[][][] readState;
+ private volatile boolean isStopped = false;
+ @SuppressWarnings("unused")
+ private volatile boolean isPaused = false;
+
+ public OrcEncodedDataReader(LowLevelCache lowLevelCache, Cache<OrcCacheKey> cache,
+ OrcMetadataCache metadataCache, Configuration conf, InputSplit split,
+ List<Integer> columnIds, SearchArgument sarg, String[] columnNames,
+ OrcEncodedDataConsumer consumer, QueryFragmentCounters counters) {
+ this.lowLevelCache = lowLevelCache;
+ this.metadataCache = metadataCache;
+ this.cache = cache;
+ this.conf = conf;
+ this.split = (FileSplit)split;
+ this.columnIds = columnIds;
+ if (this.columnIds != null) {
+ Collections.sort(this.columnIds);
+ }
+ this.sarg = sarg;
+ this.columnNames = columnNames;
+ this.consumer = consumer;
+ this.counters = counters;
+ }
+
+ @Override
+ public void stop() {
+ if (LOG.isInfoEnabled()) {
+ LOG.info("Encoded reader is being stopped");
+ }
+ isStopped = true;
+ }
+
+ @Override
+ public void pause() {
+ isPaused = true;
+ // TODO: pause fetching
+ }
+
+ @Override
+ public void unpause() {
+ isPaused = false;
+ // TODO: unpause fetching
+ }
+
+ @Override
+ protected Void callInternal() throws IOException {
+ long startTime = counters.startTimeCounter();
+ if (LlapIoImpl.LOGL.isInfoEnabled()) {
+ LlapIoImpl.LOG.info("Processing data for " + split.getPath());
+ }
+ if (processStop()) {
+ recordReaderTime(startTime);
+ return null;
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.TABLE, getDbAndTableName(split.getPath()));
+ orcReader = null;
+ // 1. Get file metadata from cache, or create the reader and read it.
+ // Don't cache the filesystem object for now; Tez closes it and FS cache will fix all that
+ fs = split.getPath().getFileSystem(conf);
+ fileId = determineFileId(fs, split);
+ counters.setDesc(QueryFragmentCounters.Desc.FILE, fileId);
+
+ try {
+ fileMetadata = getOrReadFileMetadata();
+ consumer.setFileMetadata(fileMetadata);
+ validateFileMetadata();
+ if (columnIds == null) {
+ columnIds = createColumnIds(fileMetadata);
+ }
+
+ // 2. Determine which stripes to read based on the split.
+ determineStripesToRead();
+ } catch (Throwable t) {
+ recordReaderTime(startTime);
+ consumer.setError(t);
+ return null;
+ }
+
+ if (readState.length == 0) {
+ consumer.setDone();
+ recordReaderTime(startTime);
+ return null; // No data to read.
+ }
+ counters.setDesc(QueryFragmentCounters.Desc.STRIPES, stripeIxFrom + "," + readState.length);
+
+ // 3. Apply SARG if needed, and otherwise determine what RGs to read.
+ int stride = fileMetadata.getRowIndexStride();
+ ArrayList<OrcStripeMetadata> stripeMetadatas = null;
+ boolean[] globalIncludes = null;
+ boolean[] sargColumns = null;
+ try {
+ globalIncludes = OrcInputFormat.genIncludedColumns(fileMetadata.getTypes(), columnIds, true);
+ if (sarg != null && stride != 0) {
+ // TODO: move this to a common method
- int[] filterColumns = RecordReaderImpl.mapSargColumns(sarg.getLeaves(), columnNames, 0);
++ int[] filterColumns = RecordReaderImpl.mapSargColumnsToOrcInternalColIdx(
++ sarg.getLeaves(), columnNames, 0);
+ // included will not be null, row options will fill the array with trues if null
+ sargColumns = new boolean[globalIncludes.length];
+ for (int i : filterColumns) {
+ // filter columns may have -1 as index which could be partition column in SARG.
+ if (i > 0) {
+ sargColumns[i] = true;
+ }
+ }
+
+ // If SARG is present, get relevant stripe metadata from cache or readers.
+ stripeMetadatas = readStripesMetadata(globalIncludes, sargColumns);
+ }
+
+ // Now, apply SARG if any; w/o sarg, this will just initialize readState.
+ boolean hasData = determineRgsToRead(globalIncludes, stride, stripeMetadatas);
+ if (!hasData) {
+ consumer.setDone();
+ recordReaderTime(startTime);
+ return null; // No data to read.
+ }
+ } catch (Throwable t) {
+ cleanupReaders();
+ consumer.setError(t);
+ recordReaderTime(startTime);
+ return null;
+ }
+
+ if (processStop()) {
+ cleanupReaders();
+ recordReaderTime(startTime);
+ return null;
+ }
+
+ // 4. Get data from high-level cache.
+ // If some cols are fully in cache, this will also give us the modified list of columns to
+ // read for every stripe (null means read all of them - the usual path). In any case,
+ // readState will be modified for column x rgs that were fetched from high-level cache.
+ List<Integer>[] stripeColsToRead = null;
+ if (cache != null) {
+ try {
+ stripeColsToRead = produceDataFromCache(stride);
+ } catch (Throwable t) {
+ // produceDataFromCache handles its own cleanup.
+ consumer.setError(t);
+ cleanupReaders();
+ recordReaderTime(startTime);
+ return null;
+ }
+ }
+
+ // 5. Create encoded data reader.
+ // In case if we have high-level cache, we will intercept the data and add it there;
+ // otherwise just pass the data directly to the consumer.
+ Consumer<OrcEncodedColumnBatch> dataConsumer = (cache == null) ? this.consumer : this;
+ try {
+ ensureOrcReader();
+ // Reader creating updates HDFS counters, don't do it here.
+ DataWrapperForOrc dw = new DataWrapperForOrc();
+ stripeReader = orcReader.encodedReader(fileId, dw, dw, POOL_FACTORY);
+ stripeReader.setDebugTracing(DebugUtils.isTraceOrcEnabled());
+ } catch (Throwable t) {
+ consumer.setError(t);
+ recordReaderTime(startTime);
+ cleanupReaders();
+ return null;
+ }
+
+ // 6. Read data.
+ // TODO: I/O threadpool could be here - one thread per stripe; for now, linear.
+ OrcBatchKey stripeKey = new OrcBatchKey(fileId, -1, 0);
+ for (int stripeIxMod = 0; stripeIxMod < readState.length; ++stripeIxMod) {
+ if (processStop()) {
+ cleanupReaders();
+ recordReaderTime(startTime);
+ return null;
+ }
+ int stripeIx = stripeIxFrom + stripeIxMod;
+ boolean[][] colRgs = null;
+ boolean[] stripeIncludes = null;
+ OrcStripeMetadata stripeMetadata = null;
+ StripeInformation stripe;
+ try {
+ List<Integer> cols = stripeColsToRead == null ? null : stripeColsToRead[stripeIxMod];
+ if (cols != null && cols.isEmpty()) continue; // No need to read this stripe.
+ stripe = fileMetadata.getStripes().get(stripeIx);
+
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Reading stripe " + stripeIx + ": "
+ + stripe.getOffset() + ", " + stripe.getLength());
+ }
+ colRgs = readState[stripeIxMod];
+ // We assume that NO_RGS value is only set from SARG filter and for all columns;
+ // intermediate changes for individual columns will unset values in the array.
+ // Skip this case for 0-column read. We could probably special-case it just like we do
+ // in EncodedReaderImpl, but for now it's not that important.
+ if (colRgs.length > 0 && colRgs[0] == SargApplier.READ_NO_RGS) continue;
+
+ // 6.1. Determine the columns to read (usually the same as requested).
+ if (cache == null || cols == null || cols.size() == colRgs.length) {
+ cols = columnIds;
+ stripeIncludes = globalIncludes;
+ } else {
+ // We are reading subset of the original columns, remove unnecessary bitmasks/etc.
+ // This will never happen w/o high-level cache.
+ stripeIncludes = OrcInputFormat.genIncludedColumns(fileMetadata.getTypes(), cols, true);
+ colRgs = genStripeColRgs(cols, colRgs);
+ }
+
+ // 6.2. Ensure we have stripe metadata. We might have read it before for RG filtering.
+ boolean isFoundInCache = false;
+ if (stripeMetadatas != null) {
+ stripeMetadata = stripeMetadatas.get(stripeIxMod);
+ } else {
+ stripeKey.stripeIx = stripeIx;
+ stripeMetadata = metadataCache.getStripeMetadata(stripeKey);
+ isFoundInCache = (stripeMetadata != null);
+ if (!isFoundInCache) {
+ counters.incrCounter(Counter.METADATA_CACHE_MISS);
+ ensureMetadataReader();
+ long startTimeHdfs = counters.startTimeCounter();
+ stripeMetadata = new OrcStripeMetadata(
+ stripeKey, metadataReader, stripe, stripeIncludes, sargColumns);
+ counters.incrTimeCounter(Counter.HDFS_TIME_US, startTimeHdfs);
+ stripeMetadata = metadataCache.putStripeMetadata(stripeMetadata);
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Caching stripe " + stripeKey.stripeIx
+ + " metadata with includes: " + DebugUtils.toString(stripeIncludes));
+ }
+ stripeKey = new OrcBatchKey(fileId, -1, 0);
+ }
+ consumer.setStripeMetadata(stripeMetadata);
+ }
+ if (!stripeMetadata.hasAllIndexes(stripeIncludes)) {
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Updating indexes in stripe " + stripeKey.stripeIx
+ + " metadata for includes: " + DebugUtils.toString(stripeIncludes));
+ }
+ assert isFoundInCache;
+ counters.incrCounter(Counter.METADATA_CACHE_MISS);
+ ensureMetadataReader();
+ updateLoadedIndexes(stripeMetadata, stripe, stripeIncludes, sargColumns);
+ } else if (isFoundInCache) {
+ counters.incrCounter(Counter.METADATA_CACHE_HIT);
+ }
+ } catch (Throwable t) {
+ consumer.setError(t);
+ cleanupReaders();
+ recordReaderTime(startTime);
+ return null;
+ }
+ if (processStop()) {
+ cleanupReaders();
+ recordReaderTime(startTime);
+ return null;
+ }
+
+ // 6.3. Finally, hand off to the stripe reader to produce the data.
+ // This is a sync call that will feed data to the consumer.
+ try {
+ // TODO: readEncodedColumns is not supposed to throw; errors should be propagated thru
+ // consumer. It is potentially holding locked buffers, and must perform its own cleanup.
+ // Also, currently readEncodedColumns is not stoppable. The consumer will discard the
+ // data it receives for one stripe. We could probably interrupt it, if it checked that.
+ stripeReader.readEncodedColumns(stripeIx, stripe, stripeMetadata.getRowIndexes(),
+ stripeMetadata.getEncodings(), stripeMetadata.getStreams(), stripeIncludes,
+ colRgs, dataConsumer);
+ } catch (Throwable t) {
+ consumer.setError(t);
+ cleanupReaders();
+ recordReaderTime(startTime);
+ return null;
+ }
+ }
+
+ // Done with all the things.
+ recordReaderTime(startTime);
+ dataConsumer.setDone();
+ if (DebugUtils.isTraceMttEnabled()) {
+ LlapIoImpl.LOG.info("done processing " + split);
+ }
+
+ // Close the stripe reader, we are done reading.
+ cleanupReaders();
+ return null;
+ }
+
+ private void recordReaderTime(long startTime) {
+ counters.incrTimeCounter(Counter.TOTAL_IO_TIME_US, startTime);
+ }
+
+ private static String getDbAndTableName(Path path) {
+ // Ideally, we'd get this from split; however, split doesn't contain any such thing and it's
+ // actually pretty hard to get cause even split generator only uses paths. We only need this
+ // for metrics; therefore, brace for BLACK MAGIC!
+ String[] parts = path.toUri().getPath().toString().split(Path.SEPARATOR);
+ int dbIx = -1;
+ // Try to find the default db postfix; don't check two last components - at least there
+ // should be a table and file (we could also try to throw away partition/bucket/acid stuff).
+ for (int i = 0; i < parts.length - 2; ++i) {
+ if (!parts[i].endsWith(DDLTask.DATABASE_PATH_SUFFIX)) continue;
+ if (dbIx >= 0) {
+ dbIx = -1; // Let's not guess.
+ break;
+ }
+ dbIx = i;
+ }
+ if (dbIx >= 0) {
+ return parts[dbIx].substring(0, parts[dbIx].length() - 3) + "." + parts[dbIx + 1];
+ }
+
+ // Just go from the back and throw away everything we think is wrong; skip last item, the file.
+ boolean isInPartFields = false;
+ for (int i = parts.length - 2; i >= 0; --i) {
+ String p = parts[i];
+ boolean isPartField = p.contains("=");
+ if ((isInPartFields && !isPartField) || (!isPartField && !p.startsWith(AcidUtils.BASE_PREFIX)
+ && !p.startsWith(AcidUtils.DELTA_PREFIX) && !p.startsWith(AcidUtils.BUCKET_PREFIX))) {
+ dbIx = i - 1;
+ break;
+ }
+ isInPartFields = isPartField;
+ }
+ // If we found something before we ran out of components, use it.
+ if (dbIx >= 0) {
+ String dbName = parts[dbIx];
+ if (dbName.endsWith(DDLTask.DATABASE_PATH_SUFFIX)) {
+ dbName = dbName.substring(0, dbName.length() - 3);
+ }
+ return dbName + "." + parts[dbIx + 1];
+ }
+ return "unknown";
+ }
+
+ private void validateFileMetadata() throws IOException {
+ if (fileMetadata.getCompressionKind() == CompressionKind.NONE) return;
+ int bufferSize = fileMetadata.getCompressionBufferSize();
+ int minAllocSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.LLAP_ORC_CACHE_MIN_ALLOC);
+ if (bufferSize < minAllocSize) {
+ LOG.warn("ORC compression buffer size (" + bufferSize + ") is smaller than LLAP low-level "
+ + "cache minimum allocation size (" + minAllocSize + "). Decrease the value for "
+ + HiveConf.ConfVars.LLAP_ORC_CACHE_MIN_ALLOC.toString() + " to avoid wasting memory");
+ }
+ }
+
+ private boolean processStop() {
+ if (!isStopped) return false;
+ LOG.info("Encoded data reader is stopping");
+ cleanupReaders();
+ return true;
+ }
+
+ private static long determineFileId(FileSystem fs, FileSplit split) throws IOException {
+ if (split instanceof OrcSplit) {
+ Long fileId = ((OrcSplit)split).getFileId();
+ if (fileId != null) {
+ return fileId;
+ }
+ }
+ LOG.warn("Split for " + split.getPath() + " (" + split.getClass() + ") does not have file ID");
+ return HdfsUtils.getFileId(fs, split.getPath());
+ }
+
+ private boolean[][] genStripeColRgs(List<Integer> stripeCols, boolean[][] globalColRgs) {
+ boolean[][] stripeColRgs = new boolean[stripeCols.size()][];
+ for (int i = 0, i2 = -1; i < globalColRgs.length; ++i) {
+ if (globalColRgs[i] == null) continue;
+ stripeColRgs[i2] = globalColRgs[i];
+ ++i2;
+ }
+ return stripeColRgs;
+ }
+
+ /**
+ * Puts all column indexes from metadata to make a column list to read all column.
+ */
+ private static List<Integer> createColumnIds(OrcFileMetadata metadata) {
+ List<Integer> columnIds = new ArrayList<Integer>(metadata.getTypes().size());
+ for (int i = 1; i < metadata.getTypes().size(); ++i) {
+ columnIds.add(i);
+ }
+ return columnIds;
+ }
+
+ /**
+ * In case if stripe metadata in cache does not have all indexes for current query, load
+ * the missing one. This is a temporary cludge until real metadata cache becomes available.
+ */
+ private void updateLoadedIndexes(OrcStripeMetadata stripeMetadata,
+ StripeInformation stripe, boolean[] stripeIncludes, boolean[] sargColumns) throws IOException {
+ // We only synchronize on write for now - design of metadata cache is very temporary;
+ // we pre-allocate the array and never remove entries; so readers should be safe.
+ synchronized (stripeMetadata) {
+ if (stripeMetadata.hasAllIndexes(stripeIncludes)) return;
+ long startTime = counters.startTimeCounter();
+ stripeMetadata.loadMissingIndexes(metadataReader, stripe, stripeIncludes, sargColumns);
+ counters.incrTimeCounter(Counter.HDFS_TIME_US, startTime);
+ }
+ }
+
+ /**
+ * Closes the stripe readers (on error).
+ */
+ private void cleanupReaders() {
+ if (metadataReader != null) {
+ try {
+ metadataReader.close();
+ } catch (IOException ex) {
+ // Ignore.
+ }
+ }
+ if (stripeReader != null) {
+ try {
+ stripeReader.close();
+ } catch (IOException ex) {
+ // Ignore.
+ }
+ }
+ }
+
+ /**
+ * Ensures orcReader is initialized for the split.
+ */
+ private void ensureOrcReader() throws IOException {
+ if (orcReader != null) return;
+ Path path = HdfsUtils.getFileIdPath(fs, split.getPath(), fileId);
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LOG.info("Creating reader for " + path + " (" + split.getPath() + ")");
+ }
+ long startTime = counters.startTimeCounter();
+ ReaderOptions opts = OrcFile.readerOptions(conf).filesystem(fs).fileMetadata(fileMetadata);
+ orcReader = EncodedOrcFile.createReader(path, opts);
+ counters.incrTimeCounter(Counter.HDFS_TIME_US, startTime);
+ }
+
+ /**
+ * Gets file metadata for the split from cache, or reads it from the file.
+ */
+ private OrcFileMetadata getOrReadFileMetadata() throws IOException {
+ OrcFileMetadata metadata = metadataCache.getFileMetadata(fileId);
+ if (metadata != null) {
+ counters.incrCounter(Counter.METADATA_CACHE_HIT);
+ return metadata;
+ }
+ counters.incrCounter(Counter.METADATA_CACHE_MISS);
+ ensureOrcReader();
+ // We assume this call doesn't touch HDFS because everything is already read; don't add time.
+ metadata = new OrcFileMetadata(fileId, orcReader);
+ return metadataCache.putFileMetadata(metadata);
+ }
+
+ /**
+ * Reads the metadata for all stripes in the file.
+ */
+ private ArrayList<OrcStripeMetadata> readStripesMetadata(
+ boolean[] globalInc, boolean[] sargColumns) throws IOException {
+ ArrayList<OrcStripeMetadata> result = new ArrayList<OrcStripeMetadata>(readState.length);
+ OrcBatchKey stripeKey = new OrcBatchKey(fileId, 0, 0);
+ for (int stripeIxMod = 0; stripeIxMod < readState.length; ++stripeIxMod) {
+ stripeKey.stripeIx = stripeIxMod + stripeIxFrom;
+ OrcStripeMetadata value = metadataCache.getStripeMetadata(stripeKey);
+ if (value == null || !value.hasAllIndexes(globalInc)) {
+ counters.incrCounter(Counter.METADATA_CACHE_MISS);
+ ensureMetadataReader();
+ StripeInformation si = fileMetadata.getStripes().get(stripeKey.stripeIx);
+ if (value == null) {
+ long startTime = counters.startTimeCounter();
+ value = new OrcStripeMetadata(stripeKey, metadataReader, si, globalInc, sargColumns);
+ counters.incrTimeCounter(Counter.HDFS_TIME_US, startTime);
+ value = metadataCache.putStripeMetadata(value);
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Caching stripe " + stripeKey.stripeIx
+ + " metadata with includes: " + DebugUtils.toString(globalInc));
+ }
+ // Create new key object to reuse for gets; we've used the old one to put in cache.
+ stripeKey = new OrcBatchKey(fileId, 0, 0);
+ }
+ // We might have got an old value from cache; recheck it has indexes.
+ if (!value.hasAllIndexes(globalInc)) {
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Updating indexes in stripe " + stripeKey.stripeIx
+ + " metadata for includes: " + DebugUtils.toString(globalInc));
+ }
+ updateLoadedIndexes(value, si, globalInc, sargColumns);
+ }
+ } else {
+ counters.incrCounter(Counter.METADATA_CACHE_HIT);
+ }
+ result.add(value);
+ consumer.setStripeMetadata(value);
+ }
+ return result;
+ }
+
+ private void ensureMetadataReader() throws IOException {
+ ensureOrcReader();
+ if (metadataReader != null) return;
+ long startTime = counters.startTimeCounter();
+ metadataReader = orcReader.metadata();
+ counters.incrTimeCounter(Counter.HDFS_TIME_US, startTime);
+ }
+
+ @Override
+ public void returnData(OrcEncodedColumnBatch ecb) {
+ for (ColumnStreamData[] datas : ecb.getColumnData()) {
+ if (datas == null) continue;
+ for (ColumnStreamData data : datas) {
+ if (data == null || data.decRef() != 0) continue;
+ if (DebugUtils.isTraceLockingEnabled()) {
+ for (MemoryBuffer buf : data.getCacheBuffers()) {
+ LlapIoImpl.LOG.info("Unlocking " + buf + " at the end of processing");
+ }
+ }
+ lowLevelCache.releaseBuffers(data.getCacheBuffers());
+ CSD_POOL.offer(data);
+ }
+ }
+ // We can offer ECB even with some streams not discarded; reset() will clear the arrays.
+ ECB_POOL.offer(ecb);
+ }
+
+ /**
+ * Determines which RGs need to be read, after stripes have been determined.
+ * SARG is applied, and readState is populated for each stripe accordingly.
+ * @param stripes All stripes in the file (field state is used to determine stripes to read).
+ */
+ private boolean determineRgsToRead(boolean[] globalIncludes, int rowIndexStride,
+ ArrayList<OrcStripeMetadata> metadata) throws IOException {
+ SargApplier sargApp = null;
+ if (sarg != null && rowIndexStride != 0) {
+ List<OrcProto.Type> types = fileMetadata.getTypes();
+ String[] colNamesForSarg = OrcInputFormat.getSargColumnNames(
+ columnNames, types, globalIncludes, fileMetadata.isOriginalFormat());
+ sargApp = new SargApplier(sarg, colNamesForSarg, rowIndexStride, types, globalIncludes.length);
+ }
+ boolean hasAnyData = false;
+ // readState should have been initialized by this time with an empty array.
+ for (int stripeIxMod = 0; stripeIxMod < readState.length; ++stripeIxMod) {
+ int stripeIx = stripeIxMod + stripeIxFrom;
+ StripeInformation stripe = fileMetadata.getStripes().get(stripeIx);
+ int rgCount = getRgCount(stripe, rowIndexStride);
+ boolean[] rgsToRead = null;
+ if (sargApp != null) {
+ OrcStripeMetadata stripeMetadata = metadata.get(stripeIxMod);
+ rgsToRead = sargApp.pickRowGroups(stripe, stripeMetadata.getRowIndexes(),
+ stripeMetadata.getBloomFilterIndexes(), true);
+ }
+ boolean isNone = rgsToRead == SargApplier.READ_NO_RGS,
+ isAll = rgsToRead == SargApplier.READ_ALL_RGS;
+ hasAnyData = hasAnyData || !isNone;
+ if (DebugUtils.isTraceOrcEnabled()) {
+ if (isNone) {
+ LlapIoImpl.LOG.info("SARG eliminated all RGs for stripe " + stripeIx);
+ } else if (!isAll) {
+ LlapIoImpl.LOG.info("SARG picked RGs for stripe " + stripeIx + ": "
+ + DebugUtils.toString(rgsToRead));
+ } else {
+ LlapIoImpl.LOG.info("Will read all " + rgCount + " RGs for stripe " + stripeIx);
+ }
+ }
+ assert isAll || isNone || rgsToRead.length == rgCount;
+ readState[stripeIxMod] = new boolean[columnIds.size()][];
+ for (int j = 0; j < columnIds.size(); ++j) {
+ readState[stripeIxMod][j] = (isAll || isNone) ? rgsToRead :
+ Arrays.copyOf(rgsToRead, rgsToRead.length);
+ }
+
+ adjustRgMetric(rgCount, rgsToRead, isNone, isAll);
+ }
+ return hasAnyData;
+ }
+
+ private void adjustRgMetric(int rgCount, boolean[] rgsToRead, boolean isNone,
+ boolean isAll) {
+ int count = 0;
+ if (!isAll) {
+ for (boolean b : rgsToRead) {
+ if (b)
+ count++;
+ }
+ } else if (!isNone) {
+ count = rgCount;
+ }
+ counters.setCounter(QueryFragmentCounters.Counter.SELECTED_ROWGROUPS, count);
+ }
+
+
+ private int getRgCount(StripeInformation stripe, int rowIndexStride) {
+ return (int)Math.ceil((double)stripe.getNumberOfRows() / rowIndexStride);
+ }
+
+ /**
+ * Determine which stripes to read for a split. Populates stripeIxFrom and readState.
+ */
+ public void determineStripesToRead() {
+ // The unit of caching for ORC is (rg x column) (see OrcBatchKey).
+ List<StripeInformation> stripes = fileMetadata.getStripes();
+ long offset = split.getStart(), maxOffset = offset + split.getLength();
+ stripeIxFrom = -1;
+ int stripeIxTo = -1;
+ if (LlapIoImpl.LOGL.isDebugEnabled()) {
+ String tmp = "FileSplit {" + split.getStart() + ", " + split.getLength() + "}; stripes ";
+ for (StripeInformation stripe : stripes) {
+ tmp += "{" + stripe.getOffset() + ", " + stripe.getLength() + "}, ";
+ }
+ LlapIoImpl.LOG.debug(tmp);
+ }
+
+ int stripeIx = 0;
+ for (StripeInformation stripe : stripes) {
+ long stripeStart = stripe.getOffset();
+ if (offset > stripeStart) {
+ // We assume splits will never start in the middle of the stripe.
+ ++stripeIx;
+ continue;
+ }
+ if (stripeIxFrom == -1) {
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Including stripes from " + stripeIx
+ + " (" + stripeStart + " >= " + offset + ")");
+ }
+ stripeIxFrom = stripeIx;
+ }
+ if (stripeStart >= maxOffset) {
+ stripeIxTo = stripeIx;
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Including stripes until " + stripeIxTo + " (" + stripeStart
+ + " >= " + maxOffset + "); " + (stripeIxTo - stripeIxFrom) + " stripes");
+ }
+ break;
+ }
+ ++stripeIx;
+ }
+ if (stripeIxTo == -1) {
+ stripeIxTo = stripeIx;
+ if (DebugUtils.isTraceOrcEnabled()) {
+ LlapIoImpl.LOG.info("Including stripes until " + stripeIx + " (end of file); "
+ + (stripeIxTo - stripeIxFrom) + " stripes");
+ }
+ }
+ readState = new boolean[stripeIxTo - stripeIxFrom][][];
+ }
+
+ // TODO: split by stripe? we do everything by stripe, and it might be faster
+ /**
+ * Takes the data from high-level cache for all stripes and returns to consumer.
+ * @return List of columns to read per stripe, if any columns were fully eliminated by cache.
+ */
+ private List<Integer>[] produceDataFromCache(int rowIndexStride) throws IOException {
+ OrcCacheKey key = new OrcCacheKey(fileId, -1, -1, -1);
+ // For each stripe, keep a list of columns that are not fully in cache (null => all of them).
+ @SuppressWarnings("unchecked")
+ List<Integer>[] stripeColsNotInCache = new List[readState.length];
+ for (int stripeIxMod = 0; stripeIxMod < readState.length; ++stripeIxMod) {
+ key.stripeIx = stripeIxFrom + stripeIxMod;
+ boolean[][] cols = readState[stripeIxMod];
+ boolean[] isMissingAnyRgs = new boolean[cols.length];
+ int totalRgCount = getRgCount(fileMetadata.getStripes().get(key.stripeIx), rowIndexStride);
+ for (int rgIx = 0; rgIx < totalRgCount; ++rgIx) {
+ OrcEncodedColumnBatch col = ECB_POOL.take();
+ col.init(fileId, key.stripeIx, rgIx, cols.length);
+ boolean hasAnyCached = false;
+ try {
+ key.rgIx = rgIx;
+ for (int colIxMod = 0; colIxMod < cols.length; ++colIxMod) {
+ boolean[] readMask = cols[colIxMod];
+ // Check if RG is eliminated by SARG
+ if ((readMask == SargApplier.READ_NO_RGS) || (readMask != SargApplier.READ_ALL_RGS
+ && (readMask.length <= rgIx || !readMask[rgIx]))) continue;
+ key.colIx = columnIds.get(colIxMod);
+ ColumnStreamData[] cached = cache.get(key);
+ if (cached == null) {
+ isMissingAnyRgs[colIxMod] = true;
+ continue;
+ }
+ assert cached.length == OrcEncodedColumnBatch.MAX_DATA_STREAMS;
+ col.setAllStreamsData(colIxMod, key.colIx, cached);
+ hasAnyCached = true;
+ if (readMask == SargApplier.READ_ALL_RGS) {
+ // We were going to read all RGs, but some were in cache, allocate the mask.
+ cols[colIxMod] = readMask = new boolean[totalRgCount];
+ Arrays.fill(readMask, true);
+ }
+ readMask[rgIx] = false; // Got from cache, don't read from disk.
+ }
+ } catch (Throwable t) {
+ // TODO: Any cleanup needed to release data in col back to cache should be here.
+ throw (t instanceof IOException) ? (IOException)t : new IOException(t);
+ }
+ if (hasAnyCached) {
+ consumer.consumeData(col);
+ }
+ }
+ boolean makeStripeColList = false; // By default assume we'll fetch all original columns.
+ for (int colIxMod = 0; colIxMod < cols.length; ++colIxMod) {
+ if (isMissingAnyRgs[colIxMod]) {
+ if (makeStripeColList) {
+ stripeColsNotInCache[stripeIxMod].add(columnIds.get(colIxMod));
+ }
+ } else if (!makeStripeColList) {
+ // Some columns were fully in cache. Make a per-stripe col list, add previous columns.
+ makeStripeColList = true;
+ stripeColsNotInCache[stripeIxMod] = new ArrayList<Integer>(cols.length - 1);
+ for (int i = 0; i < colIxMod; ++i) {
+ stripeColsNotInCache[stripeIxMod].add(columnIds.get(i));
+ }
+ }
+ }
+ }
+ return stripeColsNotInCache;
+ }
+
+ @Override
+ public void setDone() {
+ consumer.setDone();
+ }
+
+ @Override
+ public void consumeData(OrcEncodedColumnBatch data) {
+ // Store object in cache; create new key object - cannot be reused.
+ assert cache != null;
+ throw new UnsupportedOperationException("not implemented");
+ /*for (int i = 0; i < data.getColumnData().length; ++i) {
+ OrcCacheKey key = new OrcCacheKey(data.getBatchKey(), data.getColumnIxs()[i]);
+ ColumnStreamData[] toCache = data.getColumnData()[i];
+ ColumnStreamData[] cached = cache.cacheOrGet(key, toCache);
+ if (toCache != cached) {
+ for (ColumnStreamData sb : toCache) {
+ if (sb.decRef() != 0) continue;
+ lowLevelCache.releaseBuffers(sb.getCacheBuffers());
+ }
+ data.getColumnData()[i] = cached;
+ }
+ }
+ consumer.consumeData(data);*/
+ }
+
+ @Override
+ public void setError(Throwable t) {
+ consumer.setError(t);
+ }
+
+ private class DataWrapperForOrc implements DataReader, DataCache {
+ private DataReader orcDataReader;
+
+ public DataWrapperForOrc() {
+ boolean useZeroCopy = (conf != null) && OrcConf.USE_ZEROCOPY.getBoolean(conf);
+ if (useZeroCopy && !lowLevelCache.getAllocator().isDirectAlloc()) {
+ throw new UnsupportedOperationException("Cannot use zero-copy reader with non-direct cache "
+ + "buffers; either disable zero-copy or enable direct cache allocation");
+ }
+ this.orcDataReader = orcReader.createDefaultDataReader(useZeroCopy);
+ }
+
+ @Override
+ public DiskRangeList getFileData(long fileId, DiskRangeList range,
+ long baseOffset, DiskRangeListFactory factory, BooleanRef gotAllData) {
+ return lowLevelCache.getFileData(fileId, range, baseOffset, factory, counters, gotAllData);
+ }
+
+ @Override
+ public long[] putFileData(long fileId, DiskRange[] ranges,
+ MemoryBuffer[] data, long baseOffset) {
+ return lowLevelCache.putFileData(
+ fileId, ranges, data, baseOffset, Priority.NORMAL, counters);
+ }
+
+ @Override
+ public void releaseBuffer(MemoryBuffer buffer) {
+ lowLevelCache.releaseBuffer(buffer);
+ }
+
+ @Override
+ public void reuseBuffer(MemoryBuffer buffer) {
+ boolean isReused = lowLevelCache.reuseBuffer(buffer);
+ assert isReused;
+ }
+
+ @Override
+ public Allocator getAllocator() {
+ return lowLevelCache.getAllocator();
+ }
+
+ @Override
+ public void close() throws IOException {
+ orcDataReader.close();
+ }
+
+ @Override
+ public DiskRangeList readFileData(DiskRangeList range, long baseOffset,
+ boolean doForceDirect) throws IOException {
+ long startTime = counters.startTimeCounter();
+ DiskRangeList result = orcDataReader.readFileData(range, baseOffset, doForceDirect);
+ counters.recordHdfsTime(startTime);
+ if (DebugUtils.isTraceOrcEnabled() && LOG.isInfoEnabled()) {
+ LOG.info("Disk ranges after disk read (file " + fileId + ", base offset " + baseOffset
+ + "): " + RecordReaderUtils.stringifyDiskRanges(result));
+ }
+ return result;
+ }
+
+ @Override
+ public boolean isTrackingDiskRanges() {
+ return orcDataReader.isTrackingDiskRanges();
+ }
+
+ @Override
+ public void releaseBuffer(ByteBuffer buffer) {
+ orcDataReader.releaseBuffer(buffer);
+ }
+
+ @Override
+ public void open() throws IOException {
+ long startTime = counters.startTimeCounter();
+ orcDataReader.open();
+ counters.recordHdfsTime(startTime);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
index 5770bef,2500fb6..ffeaaa0
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
@@@ -56,9 -56,9 +56,10 @@@ import org.apache.hadoop.hive.ql.io.Aci
import org.apache.hadoop.hive.ql.io.AcidUtils.Directory;
import org.apache.hadoop.hive.ql.io.CombineHiveInputFormat;
import org.apache.hadoop.hive.ql.io.InputFormatChecker;
+import org.apache.hadoop.hive.ql.io.LlapWrappableInputFormatInterface;
import org.apache.hadoop.hive.ql.io.RecordIdentifier;
import org.apache.hadoop.hive.ql.io.StatsProvidingRecordReader;
+ import org.apache.hadoop.hive.ql.io.orc.OrcFile.WriterVersion;
import org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.Context;
import org.apache.hadoop.hive.ql.io.sarg.ConvertAstToSearchArg;
import org.apache.hadoop.hive.ql.io.sarg.PredicateLeaf;
@@@ -264,10 -265,9 +266,9 @@@ public class OrcInputFormat implement
boolean[] result = new boolean[numColumns];
result[0] = true;
OrcProto.Type root = types.get(rootColumn);
- for(int i=0; i < root.getSubtypesCount(); ++i) {
+ for (int i = 0; i < root.getSubtypesCount(); ++i) {
if (included.contains(i)) {
- includeColumnRecursive(types, result, root.getSubtypes(i),
- rootColumn);
+ includeColumnRecursive(types, result, root.getSubtypes(i), rootColumn);
}
}
return result;
@@@ -866,33 -901,11 +904,11 @@@
// we can't eliminate stripes if there are deltas because the
// deltas may change the rows making them match the predicate.
- if (deltas.isEmpty()) {
- Reader.Options options = new Reader.Options();
- options.include(includedCols);
- setSearchArgument(options, types, context.conf, isOriginal);
- // only do split pruning if HIVE-8732 has been fixed in the writer
- if (options.getSearchArgument() != null &&
- writerVersion != OrcFile.WriterVersion.ORIGINAL) {
- SearchArgument sarg = options.getSearchArgument();
- List<PredicateLeaf> sargLeaves = sarg.getLeaves();
- int[] filterColumns = RecordReaderImpl.mapSargColumns(sargLeaves,
- options.getColumnNames(), getRootColumn(isOriginal));
-
- if (stripeStats != null) {
- // eliminate stripes that doesn't satisfy the predicate condition
- includeStripe = new boolean[stripes.size()];
- for (int i = 0; i < stripes.size(); ++i) {
- includeStripe[i] = (i >= stripeStats.size()) ||
- isStripeSatisfyPredicate(stripeStats.get(i), sarg,
- filterColumns);
- if (isDebugEnabled && !includeStripe[i]) {
- LOG.debug("Eliminating ORC stripe-" + i + " of file '" +
- fileWithId.getFileStatus().getPath() + "' as it did not satisfy " +
- "predicate condition.");
- }
- }
- }
- }
+ if (deltas.isEmpty() && canCreateSargFromConf(context.conf)) {
+ SearchArgument sarg = ConvertAstToSearchArg.createFromConf(context.conf);
+ String[] sargColNames = extractNeededColNames(types, context.conf, includedCols, isOriginal);
+ includeStripe = pickStripes(sarg, sargColNames, writerVersion, isOriginal,
- metadata.getStripeStatistics(), stripes.size(), file.getPath());
++ stripeStats, stripes.size(), file.getPath());
}
// if we didn't have predicate pushdown, read everything
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vector_count_distinct.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vector_decimal_aggregate.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
index c974d00,52c00f9..563213a
--- a/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
+++ b/ql/src/test/results/clientpositive/spark/vector_distinct_2.q.out
@@@ -143,20 -143,24 +143,32 @@@ STAGE PLANS
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
++<<<<<<< HEAD
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
++=======
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
++>>>>>>> master
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
index 9687ec1,2255f72..b2402db
--- a/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
+++ b/ql/src/test/results/clientpositive/spark/vector_groupby_3.q.out
@@@ -153,13 -152,18 +153,25 @@@ STAGE PLANS
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: tinyint), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
++<<<<<<< HEAD
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
++=======
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Execution mode: vectorized
++>>>>>>> master
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
index a3a44df,bbc66fc..6308cee
--- a/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
+++ b/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
@@@ -65,22 -65,17 +65,18 @@@ STAGE PLANS
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reducer 4
+ Execution mode: vectorized
Local Work:
Map Reduce Local Work
Reduce Operator Tree:
@@@ -270,22 -266,17 +266,18 @@@ STAGE PLANS
Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int)
+ Group By Operator
+ keys: l_partkey (type: int)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reducer 4
+ Execution mode: vectorized
Local Work:
Map Reduce Local Work
Reduce Operator Tree:
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vector_orderby_5.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vectorization_0.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vectorization_13.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vectorization_15.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vectorization_short_regress.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
index 7308fb2,316ed63..bf23ae4
--- a/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
@@@ -97,24 -97,19 +97,27 @@@ STAGE PLANS
input vertices:
1 Map 4
Statistics: Num rows: 7433 Data size: 228226 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: double)
+ Group By Operator
+ aggregations: sum(_col1)
+ mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 7433 Data size: 228226 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col0)
- mode: hash
- outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
++<<<<<<< HEAD
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: double)
+ Execution mode: vectorized
++=======
+ value expressions: _col0 (type: double)
++>>>>>>> master
Local Work:
Map Reduce Local Work
- Execution mode: vectorized
Reducer 3
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0)
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/spark/vectorized_timestamp_funcs.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out
index 2e87e2c,63e6ade..8e4a501
--- a/ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out
+++ b/ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out
@@@ -1591,37 -1591,37 +1591,37 @@@ STAGE PLANS
Filter Operator
predicate: (t is null or (t = 27)) (type: boolean)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: si (type: smallint), i (type: int), b (type: bigint), f (type: float), t (type: tinyint)
+ Group By Operator
+ keys: t (type: tinyint), si (type: smallint), i (type: int), b (type: bigint), f (type: float)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
+ sort order: +++++
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- sort order: +++++
- Map-reduce partition columns: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: smallint), KEY._col1 (type: int), KEY._col2 (type: bigint), KEY._col3 (type: float), KEY._col4 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: smallint), KEY._col2 (type: int), KEY._col3 (type: bigint), KEY._col4 (type: float)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.over1k_part2_orc
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.over1k_part2_orc
- Execution mode: vectorized
Stage: Stage-2
Dependency Collection
@@@ -1670,50 -1669,37 +1669,37 @@@ STAGE PLANS
Filter Operator
predicate: (t is null or (t = 27)) (type: boolean)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: si (type: smallint), i (type: int), b (type: bigint), f (type: float), t (type: tinyint)
+ Group By Operator
+ keys: t (type: tinyint), si (type: smallint), i (type: int), b (type: bigint), f (type: float)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ Reduce Output Operator
+ key expressions: _col0 (type: tinyint), _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float)
+ sort order: +++++
+ Map-reduce partition columns: _col0 (type: tinyint)
Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- sort order: +++++
- Map-reduce partition columns: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Statistics: Num rows: 1048 Data size: 310873 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: smallint), KEY._col1 (type: int), KEY._col2 (type: bigint), KEY._col3 (type: float), KEY._col4 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: smallint), KEY._col2 (type: int), KEY._col3 (type: bigint), KEY._col4 (type: float)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col4 (type: tinyint)
- sort order: +
- Map-reduce partition columns: _col4 (type: tinyint)
- Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: smallint), _col1 (type: int), _col2 (type: bigint), _col3 (type: float), _col4 (type: tinyint)
- Reducer 3
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col0 (type: smallint), VALUE._col1 (type: int), VALUE._col2 (type: bigint), VALUE._col3 (type: float), VALUE._col4 (type: tinyint)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
- Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: smallint), _col2 (type: int), _col3 (type: bigint), _col4 (type: float), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- name: default.over1k_part2_orc
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 524 Data size: 155436 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ name: default.over1k_part2_orc
- Execution mode: vectorized
Stage: Stage-2
Dependency Collection
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/mrr.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_binary_join_groupby.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_count_distinct.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_decimal_aggregate.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_decimal_udf.q.out
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f324305a/ql/src/test/results/clientpositive/tez/vector_distinct_2.q.out
----------------------------------------------------------------------
diff --cc ql/src/test/results/clientpositive/tez/vector_distinct_2.q.out
index a1063ab,6c31294..44d207b
--- a/ql/src/test/results/clientpositive/tez/vector_distinct_2.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_distinct_2.q.out
@@@ -143,20 -143,24 +143,24 @@@ STAGE PLANS
Statistics: Num rows: 2000 Data size: 918712 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: tinyint)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: tinyint)
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1000 Data size: 459356 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Execution mode: vectorized
Stage: Stage-0
Fetch Operator
[31/51] [abbrv] hive git commit: HIVE-12036 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12036 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b28aec9a
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b28aec9a
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b28aec9a
Branch: refs/heads/master
Commit: b28aec9a0f5e2507f05f9bb4c235bb6136bb5df2
Parents: c5ccf66 0ca9ff8
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 5 12:42:39 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 5 12:42:39 2015 -0700
----------------------------------------------------------------------
.../apache/hive/beeline/HiveSchemaHelper.java | 4 +-
.../hadoop/hive/common/StatsSetupConst.java | 8 -
.../org/apache/hadoop/hive/conf/HiveConf.java | 20 +-
.../hive/common/type/TestHiveDecimal.java | 12 +-
data/files/decimal_1_1.txt | 30 +
data/files/sample2.json | 2 +
.../hadoop/hive/hbase/HBaseStatsAggregator.java | 128 -
.../hadoop/hive/hbase/HBaseStatsPublisher.java | 154 -
.../hive/hbase/HBaseStatsSetupConstants.java | 34 -
.../hadoop/hive/hbase/HBaseStatsUtils.java | 135 -
.../src/test/queries/positive/hbase_stats.q | 30 -
.../src/test/queries/positive/hbase_stats2.q | 31 -
.../positive/hbase_stats_empty_partition.q | 13 -
.../src/test/results/positive/hbase_stats.q.out | 311 -
.../test/results/positive/hbase_stats2.q.out | 311 -
.../positive/hbase_stats_empty_partition.q.out | 63 -
hcatalog/conf/proto-hive-site.xml | 2 +-
.../templates/conf/hive-site.xml.template | 2 +-
hcatalog/streaming/pom.xml | 7 +
.../streaming/AbstractRecordWriter.java | 93 +-
.../streaming/DelimitedInputWriter.java | 54 +-
.../hcatalog/streaming/StrictJsonWriter.java | 46 +-
.../hive/hcatalog/streaming/TestStreaming.java | 698 +-
.../test/resources/testconfiguration.properties | 1 +
.../hadoop/hive/metastore/HiveAlterHandler.java | 2 +-
.../hadoop/hive/metastore/HiveMetaStore.java | 30 +-
.../hive/metastore/HiveMetaStoreClient.java | 94 +
.../hadoop/hive/metastore/IMetaStoreClient.java | 20 +
.../hadoop/hive/metastore/MetaStoreUtils.java | 40 +-
.../apache/hadoop/hive/metastore/Warehouse.java | 4 +-
.../hive/metastore/hbase/HBaseReadWrite.java | 9 +-
.../metastore/txn/CompactionTxnHandler.java | 36 +-
.../metastore/AlternateFailurePreListener.java | 2 +-
pom.xml | 3 +-
ql/pom.xml | 3 +
.../java/org/apache/hadoop/hive/ql/Driver.java | 2 +-
.../org/apache/hadoop/hive/ql/exec/DDLTask.java | 8 +-
.../hive/ql/exec/tez/HashTableLoader.java | 25 +-
.../hive/ql/exec/tez/HiveSplitGenerator.java | 4 +-
.../hadoop/hive/ql/io/orc/OrcInputFormat.java | 366 +-
.../hadoop/hive/ql/io/orc/ReaderImpl.java | 3 +-
.../serde/ArrayWritableObjectInspector.java | 7 +
.../hadoop/hive/ql/lockmgr/DummyTxnManager.java | 3 +-
.../hadoop/hive/ql/lockmgr/HiveLockObject.java | 6 +-
.../apache/hadoop/hive/ql/metadata/Hive.java | 48 +-
.../apache/hadoop/hive/ql/metadata/Table.java | 5 +-
.../hive/ql/optimizer/ConvertJoinMapJoin.java | 4 +
.../optimizer/SortedDynPartitionOptimizer.java | 7 +-
.../correlation/CorrelationUtilities.java | 33 -
.../RewriteQueryUsingAggregateIndexCtx.java | 2 +-
.../ql/parse/ColumnStatsSemanticAnalyzer.java | 2 +-
.../hive/ql/parse/LoadSemanticAnalyzer.java | 12 -
.../hive/ql/txn/AcidHouseKeeperService.java | 5 +-
.../hadoop/hive/ql/txn/compactor/Cleaner.java | 3 +-
.../hive/ql/txn/compactor/CompactorMR.java | 10 +-
.../hive/ql/txn/compactor/CompactorThread.java | 9 +-
.../hadoop/hive/ql/txn/compactor/Initiator.java | 5 +-
.../hadoop/hive/ql/metadata/StringAppender.java | 2 +-
.../hadoop/hive/ql/metadata/TestHive.java | 2 +-
.../special_character_in_tabnames_1.q | 13 +
ql/src/test/queries/clientpositive/cross_join.q | 8 +
.../test/queries/clientpositive/decimal_1_1.q | 9 +
.../dynpart_sort_opt_vectorization.q | 2 -
.../clientpositive/dynpart_sort_optimization.q | 2 -
.../clientpositive/dynpart_sort_optimization2.q | 2 -
.../queries/clientpositive/load_non_hdfs_path.q | 6 +
.../parquet_mixed_partition_formats2.q | 31 +
.../special_character_in_tabnames_1.q | 1075 +
.../special_character_in_tabnames_2.q | 40 +
.../special_character_in_tabnames_1.q.out | 10 +
.../results/clientpositive/cross_join.q.out | 196 +
.../results/clientpositive/decimal_1_1.q.out | 104 +
.../clientpositive/load_non_hdfs_path.q.out | 16 +
.../parquet_mixed_partition_formats2.q.out | 99 +
.../clientpositive/spark/cross_join.q.out | 211 +
.../clientpositive/spark/decimal_1_1.q.out | 104 +
.../special_character_in_tabnames_1.q.out | 19550 +++++++++++++++++
.../special_character_in_tabnames_2.q.out | 304 +
.../results/clientpositive/tez/auto_join0.q.out | 1 -
.../tez/auto_sortmerge_join_12.q.out | 1 -
.../results/clientpositive/tez/cross_join.q.out | 187 +
.../tez/cross_product_check_2.q.out | 6 -
.../tez/dynamic_partition_pruning.q.out | 1 -
.../vectorized_dynamic_partition_pruning.q.out | 1 -
.../objectinspector/ObjectInspectorUtils.java | 29 +
service/pom.xml | 6 +
.../auth/LdapAuthenticationProviderImpl.java | 19 +-
.../cli/operation/LogDivertAppender.java | 2 +-
.../auth/TestLdapAtnProviderWithLdapServer.java | 215 +
.../org/apache/hive/service/auth/ldapdata.ldif | 59 +
.../hadoop/hive/common/type/HiveDecimal.java | 10 +-
.../hive/ptest/execution/JIRAService.java | 115 +-
.../org/apache/hive/ptest/execution/PTest.java | 11 +-
.../hive/ptest/execution/TestCheckPhase.java | 77 +
.../ptest/execution/TestTestCheckPhase.java | 91 +
.../src/test/resources/HIVE-10761.6.patch | 2539 +++
.../src/test/resources/HIVE-11271.4.patch | 606 +
.../ptest2/src/test/resources/HIVE-9377.1.patch | 25 +
.../ptest2/src/test/resources/remove-test.patch | 33 +
99 files changed, 27175 insertions(+), 1646 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/ql/pom.xml
----------------------------------------------------------------------
diff --cc ql/pom.xml
index 99c22a3,587e2ee..83b9ebf
--- a/ql/pom.xml
+++ b/ql/pom.xml
@@@ -709,7 -704,7 +709,8 @@@
<include>org.apache.hive:hive-common</include>
<include>org.apache.hive:hive-exec</include>
<include>org.apache.hive:hive-serde</include>
+ <include>org.apache.hive:hive-llap-client</include>
+ <include>org.apache.hive:hive-metastore</include>
<include>com.esotericsoftware.kryo:kryo</include>
<include>org.apache.parquet:parquet-hadoop-bundle</include>
<include>org.apache.thrift:libthrift</include>
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HiveSplitGenerator.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
index 8941db1,ef62a23..6ffec30
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
@@@ -108,10 -114,10 +114,10 @@@ import com.google.common.util.concurren
* delete events have null for row.
*/
public class OrcInputFormat implements InputFormat<NullWritable, OrcStruct>,
- InputFormatChecker, VectorizedInputFormatInterface,
+ InputFormatChecker, VectorizedInputFormatInterface, LlapWrappableInputFormatInterface,
AcidInputFormat<NullWritable, OrcStruct>, CombineHiveInputFormat.AvoidSplitCombination {
- static enum SplitStrategyKind{
+ static enum SplitStrategyKind {
HYBRID,
BI,
ETL
@@@ -811,7 -821,7 +823,8 @@@
this.file = this.fileWithId.getFileStatus();
this.blockSize = this.file.getBlockSize();
this.fileInfo = splitInfo.fileInfo;
- locations = SHIMS.getLocationsWithOffset(fs, fileWithId.getFileStatus());
- locations = SHIMS.getLocationsWithOffset(fs, file); // TODO: potential DFS call
++ // TODO: potential DFS call
++ this.locations = SHIMS.getLocationsWithOffset(fs, fileWithId.getFileStatus());
this.isOriginal = splitInfo.isOriginal;
this.deltas = splitInfo.deltas;
this.hasBase = splitInfo.hasBase;
@@@ -993,41 -1003,51 +1006,51 @@@
}
private void populateAndCacheStripeDetails() throws IOException {
- Reader orcReader = OrcFile.createReader(fileWithId.getFileStatus().getPath(),
- OrcFile.readerOptions(context.conf).filesystem(fs));
+ // Only create OrcReader if we are missing some information.
- OrcProto.Footer footer;
++ List<OrcProto.ColumnStatistics> colStatsLocal;
++ List<OrcProto.Type> typesLocal;
if (fileInfo != null) {
stripes = fileInfo.stripeInfos;
+ stripeStats = fileInfo.stripeStats;
fileMetaInfo = fileInfo.fileMetaInfo;
- types = fileInfo.types;
- metadata = fileInfo.metadata;
- types = fileInfo.footer.getTypesList();
++ typesLocal = types = fileInfo.types;
++ colStatsLocal = fileInfo.fileStats;
writerVersion = fileInfo.writerVersion;
- footer = fileInfo.footer;
// For multiple runs, in case sendSplitsInFooter changes
if (fileMetaInfo == null && context.footerInSplits) {
+ Reader orcReader = createOrcReader();
fileInfo.fileMetaInfo = ((ReaderImpl) orcReader).getFileMetaInfo();
- fileInfo.types = orcReader.getTypes();
- fileInfo.writerVersion = orcReader.getWriterVersion();
- assert fileInfo.metadata != null && fileInfo.footer != null
++ assert fileInfo.stripeStats != null && fileInfo.types != null
+ && fileInfo.writerVersion != null;
- footer = fileInfo.footer;
+ // We assume that if we needed to create a reader, we need to cache it to meta cache.
+ // TODO: This will also needlessly overwrite it in local cache for now.
+ Context.footerCache.put(fileWithId.getFileId(), file, fileInfo.fileMetaInfo, orcReader);
}
} else {
+ Reader orcReader = createOrcReader();
stripes = orcReader.getStripes();
- metadata = orcReader.getMetadata();
-- types = orcReader.getTypes();
++ typesLocal = types = orcReader.getTypes();
++ colStatsLocal = orcReader.getOrcProtoFileStatistics();
writerVersion = orcReader.getWriterVersion();
+ stripeStats = orcReader.getStripeStatistics();
fileMetaInfo = context.footerInSplits ?
((ReaderImpl) orcReader).getFileMetaInfo() : null;
- footer = orcReader.getFooter();
if (context.cacheStripeDetails) {
- // Populate into cache.
- Context.footerCache.put(fileWithId.getFileStatus().getPath(),
- new FileInfo(fileWithId.getFileStatus().getModificationTime(),
- fileWithId.getFileStatus().getLen(), stripes,
- stripeStats, types, fileMetaInfo, writerVersion));
+ Long fileId = fileWithId.getFileId();
+ Context.footerCache.put(fileId, file, fileMetaInfo, orcReader);
}
}
includedCols = genIncludedColumns(types, context.conf, isOriginal);
- projColsUncompressedSize = computeProjectionSize(orcReader, includedCols, isOriginal);
- projColsUncompressedSize = computeProjectionSize(footer, includedCols, isOriginal);
++ projColsUncompressedSize = computeProjectionSize(typesLocal, colStatsLocal, includedCols, isOriginal);
+ }
+
+ private Reader createOrcReader() throws IOException {
+ return OrcFile.createReader(file.getPath(),
+ OrcFile.readerOptions(context.conf).filesystem(fs));
}
- private long computeProjectionSize(final Reader orcReader, final boolean[] includedCols,
- final boolean isOriginal) {
- private long computeProjectionSize(
- OrcProto.Footer footer, final boolean[] includedCols, final boolean isOriginal) {
++ private long computeProjectionSize(List<OrcProto.Type> types,
++ List<OrcProto.ColumnStatistics> stats, boolean[] includedCols, boolean isOriginal) {
final int rootIdx = getRootColumn(isOriginal);
List<Integer> internalColIds = Lists.newArrayList();
if (includedCols != null) {
@@@ -1037,7 -1057,7 +1060,7 @@@
}
}
}
- return orcReader.getRawDataSizeFromColIndices(internalColIds);
- return ReaderImpl.getRawDataSizeFromColIndices(internalColIds, footer);
++ return ReaderImpl.getRawDataSizeFromColIndices(internalColIds, types, stats);
}
}
@@@ -1048,9 -1068,12 +1071,13 @@@
static List<OrcSplit> generateSplitsInfo(Configuration conf, int numSplits)
throws IOException {
- // use threads to resolve directories into splits
+ // Use threads to resolve directories into splits.
+ if (HiveConf.getBoolVar(conf, HiveConf.ConfVars.HIVE_ORC_MS_FOOTER_CACHE_ENABLED)) {
+ // Create HiveConf once, since this is expensive.
+ conf = new HiveConf(conf, OrcInputFormat.class);
+ }
Context context = new Context(conf, numSplits);
+ boolean useFileIds = HiveConf.getBoolVar(conf, ConfVars.HIVE_ORC_INCLUDE_FILE_ID_IN_SPLITS);
List<OrcSplit> splits = Lists.newArrayList();
List<Future<AcidDirInfo>> pathFutures = Lists.newArrayList();
List<Future<List<OrcSplit>>> splitFutures = Lists.newArrayList();
@@@ -1137,26 -1163,28 +1166,29 @@@
*
*/
private static class FileInfo {
- long modificationTime;
- long size;
- List<StripeInformation> stripeInfos;
- FileMetaInfo fileMetaInfo;
- List<StripeStatistics> stripeStats;
- List<OrcProto.Type> types;
+ private final long modificationTime;
+ private final long size;
+ private final Long fileId;
+ private final List<StripeInformation> stripeInfos;
- private ReaderImpl.FileMetaInfo fileMetaInfo;
- private Metadata metadata;
- private OrcProto.Footer footer;
++ private FileMetaInfo fileMetaInfo;
++ private List<StripeStatistics> stripeStats;
++ private List<OrcProto.ColumnStatistics> fileStats;
++ private List<OrcProto.Type> types;
private OrcFile.WriterVersion writerVersion;
-- FileInfo(long modificationTime, long size,
-- List<StripeInformation> stripeInfos,
- Metadata metadata, OrcProto.Footer footer,
- ReaderImpl.FileMetaInfo fileMetaInfo,
++ FileInfo(long modificationTime, long size, List<StripeInformation> stripeInfos,
+ List<StripeStatistics> stripeStats, List<OrcProto.Type> types,
- FileMetaInfo fileMetaInfo,
- OrcFile.WriterVersion writerVersion) {
++ List<OrcProto.ColumnStatistics> fileStats, FileMetaInfo fileMetaInfo,
+ OrcFile.WriterVersion writerVersion, Long fileId) {
this.modificationTime = modificationTime;
this.size = size;
+ this.fileId = fileId;
this.stripeInfos = stripeInfos;
this.fileMetaInfo = fileMetaInfo;
- this.metadata = metadata;
- this.footer = footer;
+ this.stripeStats = stripeStats;
+ this.types = types;
++ this.fileStats = fileStats;
this.writerVersion = writerVersion;
}
}
@@@ -1516,5 -1544,186 +1548,187 @@@
bucket, validTxnList, new Reader.Options(), deltaDirectory);
}
+ /**
+ * Represents footer cache.
+ */
+ public interface FooterCache {
+ FileInfo[] getAndValidate(List<HdfsFileStatusWithId> files) throws IOException;
+ void put(Long fileId, FileStatus file, FileMetaInfo fileMetaInfo, Reader orcReader)
+ throws IOException;
+ }
+
+ /** Local footer cache using Guava. Stores convoluted Java objects. */
+ private static class LocalCache implements FooterCache {
+ private Cache<Path, FileInfo> cache;
+
+ public LocalCache(int numThreads, int cacheStripeDetailsSize) {
+ cache = CacheBuilder.newBuilder()
+ .concurrencyLevel(numThreads)
+ .initialCapacity(cacheStripeDetailsSize)
+ .maximumSize(cacheStripeDetailsSize)
+ .softValues()
+ .build();
+ }
+ @Override
+ public FileInfo[] getAndValidate(List<HdfsFileStatusWithId> files) {
+ // TODO: should local cache also be by fileId? Preserve the original logic for now.
+ FileInfo[] result = new FileInfo[files.size()];
+ int i = -1;
+ for (HdfsFileStatusWithId fileWithId : files) {
+ ++i;
+ FileStatus file = fileWithId.getFileStatus();
+ Path path = file.getPath();
+ Long fileId = fileWithId.getFileId();
+ FileInfo fileInfo = cache.getIfPresent(path);
+ if (isDebugEnabled) {
+ LOG.debug("Info " + (fileInfo == null ? "not " : "") + "cached for path: " + path);
+ }
+ if (fileInfo == null) continue;
+ if ((fileId != null && fileInfo.fileId != null && fileId == fileInfo.fileId)
+ || (fileInfo.modificationTime == file.getModificationTime() &&
+ fileInfo.size == file.getLen())) {
+ result[i] = fileInfo;
+ continue;
+ }
+ // Invalidate
+ cache.invalidate(path);
+ if (isDebugEnabled) {
+ LOG.debug("Meta-Info for : " + path + " changed. CachedModificationTime: "
+ + fileInfo.modificationTime + ", CurrentModificationTime: "
+ + file.getModificationTime() + ", CachedLength: " + fileInfo.size
+ + ", CurrentLength: " + file.getLen());
+ }
+ }
+ return result;
+ }
+
+ public void put(Path path, FileInfo fileInfo) {
+ cache.put(path, fileInfo);
+ }
+
+ @Override
+ public void put(Long fileId, FileStatus file, FileMetaInfo fileMetaInfo, Reader orcReader)
+ throws IOException {
+ cache.put(file.getPath(), new FileInfo(file.getModificationTime(), file.getLen(),
- orcReader.getStripes(), orcReader.getMetadata(), orcReader.getFooter(), fileMetaInfo,
- orcReader.getWriterVersion(), fileId));
++ orcReader.getStripes(), orcReader.getStripeStatistics(), orcReader.getTypes(),
++ orcReader.getOrcProtoFileStatistics(), fileMetaInfo, orcReader.getWriterVersion(),
++ fileId));
+ }
+ }
+
+ /** Metastore-based footer cache storing serialized footers. Also has a local cache. */
+ public static class MetastoreCache implements FooterCache {
+ private final LocalCache localCache;
+ private boolean isWarnLogged = false;
+ private HiveConf conf;
+
+ public MetastoreCache(LocalCache lc) {
+ localCache = lc;
+ }
+
+ @Override
+ public FileInfo[] getAndValidate(List<HdfsFileStatusWithId> files) throws IOException {
+ // First, check the local cache.
+ FileInfo[] result = localCache.getAndValidate(files);
+ assert result.length == files.size();
+ // This is an unfortunate consequence of batching/iterating thru MS results.
+ // TODO: maybe have a direct map call for small lists if this becomes a perf issue.
+ HashMap<Long, Integer> posMap = new HashMap<>(files.size());
+ for (int i = 0; i < result.length; ++i) {
+ if (result[i] != null) continue;
+ HdfsFileStatusWithId file = files.get(i);
+ Long fileId = file.getFileId();
+ if (fileId == null) {
+ if (!isWarnLogged || isDebugEnabled) {
+ LOG.warn("Not using metastore cache because fileId is missing: "
+ + file.getFileStatus().getPath());
+ isWarnLogged = true;
+ }
+ continue;
+ }
+ posMap.put(fileId, i);
+ }
+ Iterator<Entry<Long, ByteBuffer>> iter = null;
+ Hive hive;
+ try {
+ hive = getHive();
+ iter = hive.getFileMetadata(Lists.newArrayList(posMap.keySet()), conf).iterator();
+ } catch (HiveException ex) {
+ throw new IOException(ex);
+ }
+ List<Long> corruptIds = null;
+ while (iter.hasNext()) {
+ Entry<Long, ByteBuffer> e = iter.next();
+ int ix = posMap.get(e.getKey());
+ assert result[ix] == null;
+ HdfsFileStatusWithId file = files.get(ix);
+ assert file.getFileId() == e.getKey();
+ result[ix] = createFileInfoFromMs(file, e.getValue());
+ if (result[ix] == null) {
+ if (corruptIds == null) {
+ corruptIds = new ArrayList<>();
+ }
+ corruptIds.add(file.getFileId());
+ } else {
+ localCache.put(file.getFileStatus().getPath(), result[ix]);
+ }
+ }
+ if (corruptIds != null) {
+ try {
+ hive.clearFileMetadata(corruptIds);
+ } catch (HiveException ex) {
+ LOG.error("Failed to clear corrupt cache data", ex);
+ }
+ }
+ return result;
+ }
+
+ private Hive getHive() throws HiveException {
+ // TODO: we wish we could cache the Hive object, but it's not thread safe, and each
+ // threadlocal we "cache" would need to be reinitialized for every query. This is
+ // a huge PITA. Hive object will be cached internally, but the compat check will be
+ // done every time inside get().
+ return Hive.getWithFastCheck(conf);
+ }
+
+ private static FileInfo createFileInfoFromMs(
+ HdfsFileStatusWithId file, ByteBuffer bb) throws IOException {
+ FileStatus fs = file.getFileStatus();
+ ReaderImpl.FooterInfo fi = null;
+ ByteBuffer original = bb.duplicate();
+ try {
+ fi = ReaderImpl.extractMetaInfoFromFooter(bb, fs.getPath());
+ } catch (Exception ex) {
+ byte[] data = new byte[original.remaining()];
+ System.arraycopy(original.array(), original.arrayOffset() + original.position(),
+ data, 0, data.length);
+ String msg = "Failed to parse the footer stored in cache for file ID "
+ + file.getFileId() + " " + original + " [ " + Hex.encodeHexString(data) + " ]";
+ LOG.error(msg, ex);
+ return null;
+ }
- return new FileInfo(fs.getModificationTime(), fs.getLen(), fi.getStripes(),
- fi.getMetadata(), fi.getFooter(), fi.getFileMetaInfo(),
++ return new FileInfo(fs.getModificationTime(), fs.getLen(), fi.getStripes(), fi.getMetadata(),
++ fi.getFooter().getTypesList(), fi.getFooter().getStatisticsList(), fi.getFileMetaInfo(),
+ fi.getFileMetaInfo().writerVersion, file.getFileId());
+ }
+
+ @Override
+ public void put(Long fileId, FileStatus file, FileMetaInfo fileMetaInfo, Reader orcReader)
+ throws IOException {
+ localCache.put(fileId, file, fileMetaInfo, orcReader);
+ if (fileId != null) {
+ try {
+ getHive().putFileMetadata(Lists.newArrayList(fileId),
+ Lists.newArrayList(((ReaderImpl)orcReader).getSerializedFileFooter()));
+ } catch (HiveException e) {
+ throw new IOException(e);
+ }
+ }
+ }
+
+ public void configure(HiveConf queryConfig) {
+ this.conf = queryConfig;
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b28aec9a/ql/src/test/results/clientpositive/tez/vectorized_dynamic_partition_pruning.q.out
----------------------------------------------------------------------
[49/51] [abbrv] hive git commit: HIVE-12153 : LLAP: update errata for
bad branch commits (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12153 : LLAP: update errata for bad branch commits (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b6de889b
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b6de889b
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b6de889b
Branch: refs/heads/master
Commit: b6de889b0ecb26b23d0530b9c594e2f1a9c65e87
Parents: 3cfcad6
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 12 16:51:42 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 12 16:51:42 2015 -0700
----------------------------------------------------------------------
errata.txt | 67 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 67 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b6de889b/errata.txt
----------------------------------------------------------------------
diff --git a/errata.txt b/errata.txt
index 7607ef1..7df4b9e 100644
--- a/errata.txt
+++ b/errata.txt
@@ -9,3 +9,70 @@ f3ab5fda6af57afff31c29ad048d906fd095d5fb branch-1.2 HIVE-10885 https://issues.ap
dcf21cd6fa98fb5db01ef661bb3b9f94d9ca2d15 master HIVE-10021 https://issues.apache.org/jira/browse/HIVE-10021
9763c9dd31bd5939db3ca50e75bb97955b411f6d master HIVE-11536 https://issues.apache.org/jira/browse/HIVE-11536
52a934f911c63fda5d69cb6036cb4e917c799259 llap HIVE-11871 https://issues.apache.org/jira/browse/HIVE-11871
+1d0881e04e1aa9dd10dde8425427f29a53bee97a llap HIVE-12125 https://issues.apache.org/jira/browse/HIVE-12125
+7f21a4254dff893ff6d882ab66e018075a37d484 llap HIVE-12126 https://issues.apache.org/jira/browse/HIVE-12126
+a4e32580f7dd0d7cda08695af0a1feae6b175709 llap HIVE-12127 https://issues.apache.org/jira/browse/HIVE-12127
+aebb82888e53676312a46587577ac67ad0b78579 llap HIVE-12128 https://issues.apache.org/jira/browse/HIVE-12128
+b0860a48b75069dd24a413dca701a2685577c1cf llap HIVE-12129 https://issues.apache.org/jira/browse/HIVE-12129
+ebb8fec397e098702e85ae68919af63f6cad2ac0 llap HIVE-12130 https://issues.apache.org/jira/browse/HIVE-12130
+b7c53456dfa49ec08952f2f1237dffb59bd3e8a9 llap HIVE-12131 https://issues.apache.org/jira/browse/HIVE-12131
+00045de70f85f7e8c843bc7bee7846339c9781b4 llap HIVE-12132 https://issues.apache.org/jira/browse/HIVE-12132
+3dc2dd9c195e5985712e4fba968f12fdb1c5ec2e llap HIVE-12133 https://issues.apache.org/jira/browse/HIVE-12133
+ac52a81f72c8ce26f13554a682ee7ae41a6a7015 llap HIVE-12134 https://issues.apache.org/jira/browse/HIVE-12134
+b4df77b013d7e5b33c4a3eddee0c1d009e2f117a llap HIVE-12135 https://issues.apache.org/jira/browse/HIVE-12135
+d4db62fb5f6779d9989f9a8153f1771895255982 llap HIVE-12136 https://issues.apache.org/jira/browse/HIVE-12136
+a12191237578abbaafb35934d094dbf1278d1412 llap HIVE-12137 https://issues.apache.org/jira/browse/HIVE-12137
+255b3a6fc621b96f32fad68437b3d8caf04823ec llap HIVE-12138 https://issues.apache.org/jira/browse/HIVE-12138
+29cd5c8d7817eb0618e4f115329882d2c4a20417 llap HIVE-12139 https://issues.apache.org/jira/browse/HIVE-12139
+f314e577c1f411ee5b434ba8d81e30e937e40c68 llap HIVE-12140 https://issues.apache.org/jira/browse/HIVE-12140
+4bbaca8b33ed31cb67862f7c815ea7b6bbe5a2b4 llap HIVE-12141 https://issues.apache.org/jira/browse/HIVE-12141
+289863694d242a16cdd5e8ed82bc8b4ef460bfdc llap HIVE-12142 https://issues.apache.org/jira/browse/HIVE-12142
+c4b2a13a5e9bccadf1ca430c2a110cbe5d68a66b llap HIVE-12143 https://issues.apache.org/jira/browse/HIVE-12143
+8b6c34eaa02f87806e6567a27baeb56c74f94926 llap HIVE-12144 https://issues.apache.org/jira/browse/HIVE-12144
+7b5f80f1273f82f864ff4a36c7d640021e7d3d6a llap HIVE-12145 https://issues.apache.org/jira/browse/HIVE-12145
+7ebf999e2fdee5de00a145653a1e58de53650602 llap HIVE-12146 https://issues.apache.org/jira/browse/HIVE-12146
+e6b1556e39f81dc2861f612733b2ba61c17ff698 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+b8acbb6ef7b97502b772569641917e6ef973b25e llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+35c18a3d9b67013bf8cb2185a27391390aacc1e4 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+a31d7c8a5346fe3f26ad241e4be17a09a41583dc llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+08969c8517861d820ed353db7b1e98e9f1799d64 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+7dc3cf966745feaffef742a2ea4d74c89d44e766 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+f11222583c2b62248832010fbb7181eee369fbca llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+a7f77a73d89ee6503d5671258f74f5d7183d805e llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+35a314bc1bb2e7e7b29232cb63d1c5adbe26234e llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+5feb58db2c99627cb41a747a097a0ec4b019d60c llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+2dadf56692fe33e1a67f162e57ba9d36bd26b84a llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+e6965be3df0c74061c44e0a6aee5f74ce9d7c113 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+76432fbe2e463c20b0230839366f8e35a0948f0f llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+42acf2b77b0f160629d9457774f5b109bc0b1fbe llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+0737a4a5aed16374e1ee504f147c96ecc6636f6a llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+d487f800a09936562dd41b6d8a039904c14dfaff llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+ccb63a31cb5f9003221341bad592080918627565 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+74c4bdfeb03f2119e01a439cc86e384ddd2bfcde llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+371c2ba38cfd90feca4be2878daf030cf8a85bfb llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+52589882b8a69577f38dbe64a1b64e51bb5f6b52 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+f9bb03441c2e50c31d29582083d467e32bc5e088 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+e5bea303829174a4999b03bbcee5b0ad57a3bcf3 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+94f3b0590ac749b2f13c2841d0b3c47c16c5d8b7 llap HIVE-12147 https://issues.apache.org/jira/browse/HIVE-12147
+69886819281100327ffb9527a001a7956ffc8daf llap HIVE-12148 https://issues.apache.org/jira/browse/HIVE-12148
+055ed8dc679d0f59645f2cf1b118ab125e24d4f5 llap HIVE-12148 https://issues.apache.org/jira/browse/HIVE-12148
+05630792e1adf24e1ead0a3b03fcf0d4af689909 llap HIVE-12149 https://issues.apache.org/jira/browse/HIVE-12149
+744dc9c36dd50d2c7ef8b54a76aba1d4109f1b23 llap HIVE-12149 https://issues.apache.org/jira/browse/HIVE-12149
+7775f7cbad687ee39b78538c38bb0a5c0329e076 llap HIVE-12150 https://issues.apache.org/jira/browse/HIVE-12150
+541fcbe720df8c62e3bd4e00311c9a8c95bb12a4 llap HIVE-12150 https://issues.apache.org/jira/browse/HIVE-12150
+53094ba7190b326d32be5e43ed4d992823c5dd4e llap HIVE-12150 https://issues.apache.org/jira/browse/HIVE-12150
+0f556fe3723ebb67dc22793fbfa4cc0e2e248f35 llap HIVE-12150 https://issues.apache.org/jira/browse/HIVE-12150
+4104b2c35eaac2669e862f6703dc003e94aba0f6 llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+ac4baea04ffc801bd2c972d7628deba0eb9ae4a8 llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+200749619c929474333c5d540eadd3751d7ecb19 llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+f27bcd9ca3a4296625079e2caf7408e855a197db llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+9b3756902e5d70f36540d11b50234c3d9a2adb39 llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+7d1ea695819ccdcaa86efe8d095323b5007df7f1 llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+2dfd8457b7ee415f1b28c5de2650b3f2457f20ea llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+fc6be8faf5c97901ccad33edca8f8f80023b308a llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+20acdb661a12f1bc472633d89428917275b6364d llap HIVE-12151 https://issues.apache.org/jira/browse/HIVE-12151
+3a2e8ee7e47bd31745dfc5f6a29c602e09747f24 llap HIVE-12152 https://issues.apache.org/jira/browse/HIVE-12152
+8ed270cb9d8a9c49cccf99402ca92e3df3304d9f llap HIVE-12152 https://issues.apache.org/jira/browse/HIVE-12152
+c6565f5d65da9ed5cb452db7e313d0ce7abc1105 llap HIVE-9729 https://issues.apache.org/jira/browse/HIVE-9729
+d8298e1c85a515150562b0df68af89c18c468638 llap HIVE-9418 https://issues.apache.org/jira/browse/HIVE-9729
+
[37/51] [abbrv] hive git commit: HIVE-12047 : LLAP: merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12047 : LLAP: merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b87f63ca
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b87f63ca
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b87f63ca
Branch: refs/heads/master
Commit: b87f63cad8773a40498efa5d5fcc3c0b754cd8e0
Parents: 86e1c89 bcff871
Author: Sergey Shelukhin <se...@apache.org>
Authored: Tue Oct 6 10:52:40 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Tue Oct 6 10:52:40 2015 -0700
----------------------------------------------------------------------
.../org/apache/hadoop/hive/conf/HiveConf.java | 4 +
.../org/apache/hive/jdbc/miniHS2/MiniHS2.java | 10 +-
.../apache/hive/jdbc/TestJdbcWithMiniHS2.java | 55 +
metastore/if/hive_metastore.thrift | 10 +-
.../gen/thrift/gen-cpp/ThriftHiveMetastore.cpp | 1744 ++++++++--------
.../gen/thrift/gen-cpp/ThriftHiveMetastore.h | 16 +-
.../gen/thrift/gen-cpp/hive_metastore_types.cpp | 557 +++---
.../gen/thrift/gen-cpp/hive_metastore_types.h | 47 +-
.../hive/metastore/api/AbortTxnRequest.java | 2 +-
.../metastore/api/AddDynamicPartitions.java | 2 +-
.../metastore/api/AddPartitionsRequest.java | 2 +-
.../hive/metastore/api/AddPartitionsResult.java | 2 +-
.../hadoop/hive/metastore/api/AggrStats.java | 2 +-
.../metastore/api/AlreadyExistsException.java | 2 +-
.../metastore/api/BinaryColumnStatsData.java | 2 +-
.../metastore/api/BooleanColumnStatsData.java | 2 +-
.../hive/metastore/api/CheckLockRequest.java | 2 +-
.../metastore/api/ClearFileMetadataRequest.java | 34 +-
.../metastore/api/ClearFileMetadataResult.java | 2 +-
.../hive/metastore/api/ColumnStatistics.java | 2 +-
.../metastore/api/ColumnStatisticsDesc.java | 2 +-
.../hive/metastore/api/ColumnStatisticsObj.java | 2 +-
.../hive/metastore/api/CommitTxnRequest.java | 2 +-
.../hive/metastore/api/CompactionRequest.java | 2 +-
.../api/ConfigValSecurityException.java | 2 +-
.../api/CurrentNotificationEventId.java | 2 +-
.../hadoop/hive/metastore/api/Database.java | 2 +-
.../apache/hadoop/hive/metastore/api/Date.java | 2 +-
.../hive/metastore/api/DateColumnStatsData.java | 2 +-
.../hadoop/hive/metastore/api/Decimal.java | 2 +-
.../metastore/api/DecimalColumnStatsData.java | 2 +-
.../metastore/api/DoubleColumnStatsData.java | 2 +-
.../hive/metastore/api/DropPartitionsExpr.java | 2 +-
.../metastore/api/DropPartitionsRequest.java | 2 +-
.../metastore/api/DropPartitionsResult.java | 2 +-
.../hive/metastore/api/EnvironmentContext.java | 2 +-
.../hadoop/hive/metastore/api/FieldSchema.java | 2 +-
.../hive/metastore/api/FireEventRequest.java | 2 +-
.../hive/metastore/api/FireEventResponse.java | 2 +-
.../hadoop/hive/metastore/api/Function.java | 2 +-
.../metastore/api/GetAllFunctionsResponse.java | 38 +-
.../api/GetFileMetadataByExprRequest.java | 143 +-
.../api/GetFileMetadataByExprResult.java | 186 +-
.../metastore/api/GetFileMetadataRequest.java | 34 +-
.../metastore/api/GetFileMetadataResult.java | 46 +-
.../metastore/api/GetOpenTxnsInfoResponse.java | 2 +-
.../hive/metastore/api/GetOpenTxnsResponse.java | 2 +-
.../api/GetPrincipalsInRoleRequest.java | 2 +-
.../api/GetPrincipalsInRoleResponse.java | 2 +-
.../api/GetRoleGrantsForPrincipalRequest.java | 2 +-
.../api/GetRoleGrantsForPrincipalResponse.java | 2 +-
.../api/GrantRevokePrivilegeRequest.java | 2 +-
.../api/GrantRevokePrivilegeResponse.java | 2 +-
.../metastore/api/GrantRevokeRoleRequest.java | 2 +-
.../metastore/api/GrantRevokeRoleResponse.java | 2 +-
.../hive/metastore/api/HeartbeatRequest.java | 2 +-
.../metastore/api/HeartbeatTxnRangeRequest.java | 2 +-
.../api/HeartbeatTxnRangeResponse.java | 2 +-
.../hive/metastore/api/HiveObjectPrivilege.java | 2 +-
.../hive/metastore/api/HiveObjectRef.java | 2 +-
.../apache/hadoop/hive/metastore/api/Index.java | 2 +-
.../api/IndexAlreadyExistsException.java | 2 +-
.../metastore/api/InsertEventRequestData.java | 2 +-
.../metastore/api/InvalidInputException.java | 2 +-
.../metastore/api/InvalidObjectException.java | 2 +-
.../api/InvalidOperationException.java | 2 +-
.../api/InvalidPartitionException.java | 2 +-
.../hive/metastore/api/LockComponent.java | 2 +-
.../hadoop/hive/metastore/api/LockRequest.java | 2 +-
.../hadoop/hive/metastore/api/LockResponse.java | 2 +-
.../hive/metastore/api/LongColumnStatsData.java | 2 +-
.../hive/metastore/api/MetaException.java | 2 +-
.../hive/metastore/api/MetadataPpdResult.java | 107 +-
.../hive/metastore/api/NoSuchLockException.java | 2 +-
.../metastore/api/NoSuchObjectException.java | 2 +-
.../hive/metastore/api/NoSuchTxnException.java | 2 +-
.../hive/metastore/api/NotificationEvent.java | 2 +-
.../metastore/api/NotificationEventRequest.java | 2 +-
.../api/NotificationEventResponse.java | 2 +-
.../hive/metastore/api/OpenTxnRequest.java | 2 +-
.../hive/metastore/api/OpenTxnsResponse.java | 2 +-
.../apache/hadoop/hive/metastore/api/Order.java | 2 +-
.../hadoop/hive/metastore/api/Partition.java | 2 +-
.../api/PartitionListComposingSpec.java | 2 +-
.../hive/metastore/api/PartitionSpec.java | 2 +-
.../api/PartitionSpecWithSharedSD.java | 2 +-
.../hive/metastore/api/PartitionWithoutSD.java | 2 +-
.../metastore/api/PartitionsByExprRequest.java | 2 +-
.../metastore/api/PartitionsByExprResult.java | 2 +-
.../metastore/api/PartitionsStatsRequest.java | 2 +-
.../metastore/api/PartitionsStatsResult.java | 2 +-
.../metastore/api/PrincipalPrivilegeSet.java | 2 +-
.../hadoop/hive/metastore/api/PrivilegeBag.java | 2 +-
.../hive/metastore/api/PrivilegeGrantInfo.java | 2 +-
.../metastore/api/PutFileMetadataRequest.java | 66 +-
.../metastore/api/PutFileMetadataResult.java | 2 +-
.../hadoop/hive/metastore/api/ResourceUri.java | 2 +-
.../apache/hadoop/hive/metastore/api/Role.java | 2 +-
.../hive/metastore/api/RolePrincipalGrant.java | 2 +-
.../hadoop/hive/metastore/api/Schema.java | 2 +-
.../hadoop/hive/metastore/api/SerDeInfo.java | 2 +-
.../api/SetPartitionsStatsRequest.java | 2 +-
.../hive/metastore/api/ShowCompactRequest.java | 2 +-
.../hive/metastore/api/ShowCompactResponse.java | 2 +-
.../api/ShowCompactResponseElement.java | 2 +-
.../hive/metastore/api/ShowLocksRequest.java | 2 +-
.../hive/metastore/api/ShowLocksResponse.java | 2 +-
.../metastore/api/ShowLocksResponseElement.java | 2 +-
.../hadoop/hive/metastore/api/SkewedInfo.java | 2 +-
.../hive/metastore/api/StorageDescriptor.java | 2 +-
.../metastore/api/StringColumnStatsData.java | 2 +-
.../apache/hadoop/hive/metastore/api/Table.java | 2 +-
.../hive/metastore/api/TableStatsRequest.java | 2 +-
.../hive/metastore/api/TableStatsResult.java | 2 +-
.../hive/metastore/api/ThriftHiveMetastore.java | 1870 +++++++++---------
.../hive/metastore/api/TxnAbortedException.java | 2 +-
.../hadoop/hive/metastore/api/TxnInfo.java | 2 +-
.../hive/metastore/api/TxnOpenException.java | 2 +-
.../apache/hadoop/hive/metastore/api/Type.java | 2 +-
.../hive/metastore/api/UnknownDBException.java | 2 +-
.../api/UnknownPartitionException.java | 2 +-
.../metastore/api/UnknownTableException.java | 2 +-
.../hive/metastore/api/UnlockRequest.java | 2 +-
.../hadoop/hive/metastore/api/Version.java | 2 +-
.../gen-php/metastore/ThriftHiveMetastore.php | 1052 +++++-----
.../src/gen/thrift/gen-php/metastore/Types.php | 214 +-
.../hive_metastore/ThriftHiveMetastore.py | 708 +++----
.../gen/thrift/gen-py/hive_metastore/ttypes.py | 140 +-
.../gen/thrift/gen-rb/hive_metastore_types.rb | 15 +-
.../hive/metastore/MetaStoreDirectSql.java | 62 +-
.../hadoop/hive/metastore/ObjectStore.java | 33 +-
.../hive/metastore/StatObjectConverter.java | 20 +-
.../metastore/txn/CompactionTxnHandler.java | 50 +-
.../model/MPartitionColumnStatistics.java | 12 -
.../metastore/model/MTableColumnStatistics.java | 8 -
.../hive/metastore/VerifyingObjectStore.java | 28 +-
.../hadoop/hive/ql/plan/api/Adjacency.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Graph.java | 2 +-
.../hadoop/hive/ql/plan/api/Operator.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Query.java | 2 +-
.../hadoop/hive/ql/plan/api/QueryPlan.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Stage.java | 2 +-
.../apache/hadoop/hive/ql/plan/api/Task.java | 2 +-
.../clientpositive/llap/explainuser_1.q.out | 70 +-
.../test/results/clientpositive/orc_llap.q.out | 50 +-
.../hadoop/hive/serde/test/InnerStruct.java | 2 +-
.../hadoop/hive/serde/test/ThriftTestObj.java | 2 +-
.../hadoop/hive/serde2/thrift/test/Complex.java | 2 +-
.../hive/serde2/thrift/test/IntString.java | 2 +-
.../hive/serde2/thrift/test/MegaStruct.java | 2 +-
.../hive/serde2/thrift/test/MiniStruct.java | 2 +-
.../hive/serde2/thrift/test/SetIntString.java | 2 +-
.../hadoop/hive/service/HiveClusterStatus.java | 2 +-
.../hive/service/HiveServerException.java | 2 +-
.../apache/hadoop/hive/service/ThriftHive.java | 2 +-
.../service/cli/thrift/TArrayTypeEntry.java | 2 +-
.../hive/service/cli/thrift/TBinaryColumn.java | 2 +-
.../hive/service/cli/thrift/TBoolColumn.java | 2 +-
.../hive/service/cli/thrift/TBoolValue.java | 2 +-
.../hive/service/cli/thrift/TByteColumn.java | 2 +-
.../hive/service/cli/thrift/TByteValue.java | 2 +-
.../hive/service/cli/thrift/TCLIService.java | 2 +-
.../cli/thrift/TCancelDelegationTokenReq.java | 2 +-
.../cli/thrift/TCancelDelegationTokenResp.java | 2 +-
.../service/cli/thrift/TCancelOperationReq.java | 2 +-
.../cli/thrift/TCancelOperationResp.java | 2 +-
.../service/cli/thrift/TCloseOperationReq.java | 2 +-
.../service/cli/thrift/TCloseOperationResp.java | 2 +-
.../service/cli/thrift/TCloseSessionReq.java | 2 +-
.../service/cli/thrift/TCloseSessionResp.java | 2 +-
.../hive/service/cli/thrift/TColumnDesc.java | 2 +-
.../hive/service/cli/thrift/TDoubleColumn.java | 2 +-
.../hive/service/cli/thrift/TDoubleValue.java | 2 +-
.../cli/thrift/TExecuteStatementReq.java | 2 +-
.../cli/thrift/TExecuteStatementResp.java | 2 +-
.../service/cli/thrift/TFetchResultsReq.java | 2 +-
.../service/cli/thrift/TFetchResultsResp.java | 2 +-
.../service/cli/thrift/TGetCatalogsReq.java | 2 +-
.../service/cli/thrift/TGetCatalogsResp.java | 2 +-
.../hive/service/cli/thrift/TGetColumnsReq.java | 2 +-
.../service/cli/thrift/TGetColumnsResp.java | 2 +-
.../cli/thrift/TGetDelegationTokenReq.java | 2 +-
.../cli/thrift/TGetDelegationTokenResp.java | 2 +-
.../service/cli/thrift/TGetFunctionsReq.java | 2 +-
.../service/cli/thrift/TGetFunctionsResp.java | 2 +-
.../hive/service/cli/thrift/TGetInfoReq.java | 2 +-
.../hive/service/cli/thrift/TGetInfoResp.java | 2 +-
.../cli/thrift/TGetOperationStatusReq.java | 2 +-
.../cli/thrift/TGetOperationStatusResp.java | 2 +-
.../cli/thrift/TGetResultSetMetadataReq.java | 2 +-
.../cli/thrift/TGetResultSetMetadataResp.java | 2 +-
.../hive/service/cli/thrift/TGetSchemasReq.java | 2 +-
.../service/cli/thrift/TGetSchemasResp.java | 2 +-
.../service/cli/thrift/TGetTableTypesReq.java | 2 +-
.../service/cli/thrift/TGetTableTypesResp.java | 2 +-
.../hive/service/cli/thrift/TGetTablesReq.java | 2 +-
.../hive/service/cli/thrift/TGetTablesResp.java | 2 +-
.../service/cli/thrift/TGetTypeInfoReq.java | 2 +-
.../service/cli/thrift/TGetTypeInfoResp.java | 2 +-
.../service/cli/thrift/THandleIdentifier.java | 2 +-
.../hive/service/cli/thrift/TI16Column.java | 2 +-
.../hive/service/cli/thrift/TI16Value.java | 2 +-
.../hive/service/cli/thrift/TI32Column.java | 2 +-
.../hive/service/cli/thrift/TI32Value.java | 2 +-
.../hive/service/cli/thrift/TI64Column.java | 2 +-
.../hive/service/cli/thrift/TI64Value.java | 2 +-
.../hive/service/cli/thrift/TMapTypeEntry.java | 2 +-
.../service/cli/thrift/TOpenSessionReq.java | 2 +-
.../service/cli/thrift/TOpenSessionResp.java | 2 +-
.../service/cli/thrift/TOperationHandle.java | 2 +-
.../service/cli/thrift/TPrimitiveTypeEntry.java | 2 +-
.../cli/thrift/TRenewDelegationTokenReq.java | 2 +-
.../cli/thrift/TRenewDelegationTokenResp.java | 2 +-
.../apache/hive/service/cli/thrift/TRow.java | 2 +-
.../apache/hive/service/cli/thrift/TRowSet.java | 2 +-
.../hive/service/cli/thrift/TSessionHandle.java | 2 +-
.../apache/hive/service/cli/thrift/TStatus.java | 2 +-
.../hive/service/cli/thrift/TStringColumn.java | 2 +-
.../hive/service/cli/thrift/TStringValue.java | 2 +-
.../service/cli/thrift/TStructTypeEntry.java | 2 +-
.../hive/service/cli/thrift/TTableSchema.java | 2 +-
.../hive/service/cli/thrift/TTypeDesc.java | 2 +-
.../service/cli/thrift/TTypeQualifiers.java | 2 +-
.../service/cli/thrift/TUnionTypeEntry.java | 2 +-
.../cli/thrift/TUserDefinedTypeEntry.java | 2 +-
.../cli/thrift/ThriftHttpCLIService.java | 7 +
226 files changed, 3943 insertions(+), 3881 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b87f63ca/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b87f63ca/itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java
----------------------------------------------------------------------
[45/51] [abbrv] hive git commit: HIVE-12042 : LLAP: update some out
files (Sergey Shelukhin) ADDENDUM4
Posted by se...@apache.org.
HIVE-12042 : LLAP: update some out files (Sergey Shelukhin) ADDENDUM4
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/49972ef0
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/49972ef0
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/49972ef0
Branch: refs/heads/master
Commit: 49972ef0926e632de786af6dab231b5f059053e6
Parents: 4cc2089
Author: Sergey Shelukhin <se...@apache.org>
Authored: Fri Oct 9 11:02:25 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Fri Oct 9 11:02:25 2015 -0700
----------------------------------------------------------------------
.../clientpositive/llap/explainuser_1.q.out | 70 ++++++++++----------
1 file changed, 35 insertions(+), 35 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/49972ef0/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index d57c271..141a80b 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -6031,36 +6031,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key AND a.value <=> b.value
PREHOOK: type: QUERY
@@ -6078,34 +6078,34 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int), value (type: int)","1":"key (type: int), value (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int), value (type: int)
| Map-reduce partition columns:key (type: int), value (type: int)
| sort order:++
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int), value (type: int)
Map-reduce partition columns:key (type: int), value (type: int)
sort order:++
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(a) */ * FROM smb_input1 a RIGHT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6123,36 +6123,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Right Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6170,36 +6170,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Inner Join 0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: explain select /*+ MAPJOIN(b) */ * FROM smb_input1 a LEFT OUTER JOIN smb_input1 b ON a.key <=> b.key
PREHOOK: type: QUERY
@@ -6217,36 +6217,36 @@ Stage-0
Reducer 2
File Output Operator [FS_6]
compressed:true
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
table:{"input format:":"org.apache.hadoop.mapred.TextInputFormat","output format:":"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"}
Select Operator [SEL_5]
outputColumnNames:["_col0","_col1","_col2","_col3"]
- Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
Merge Join Operator [MERGEJOIN_7]
| condition map:[{"":"Left Outer Join0 to 1"}]
| keys:{"0":"key (type: int)","1":"key (type: int)"}
| outputColumnNames:["_col0","_col1","_col5","_col6"]
- | Statistics:Num rows: 24 Data size: 174 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 28 Data size: 209 Basic stats: COMPLETE Column stats: NONE
|<-Map 1 [SIMPLE_EDGE]
| Reduce Output Operator [RS_2]
| key expressions:key (type: int)
| Map-reduce partition columns:key (type: int)
| sort order:+
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
| value expressions:value (type: int)
| TableScan [TS_0]
| alias:a
- | Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ | Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
|<-Map 3 [SIMPLE_EDGE]
Reduce Output Operator [RS_3]
key expressions:key (type: int)
Map-reduce partition columns:key (type: int)
sort order:+
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
value expressions:value (type: int)
TableScan [TS_1]
alias:b
- Statistics:Num rows: 22 Data size: 159 Basic stats: COMPLETE Column stats: NONE
+ Statistics:Num rows: 26 Data size: 190 Basic stats: COMPLETE Column stats: NONE
PREHOOK: query: drop table sales
PREHOOK: type: DROPTABLE
[32/51] [abbrv] hive git commit: HIVE-12034 : HIVE-4243 broke things
for llap branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12034 : HIVE-4243 broke things for llap branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/edd3079f
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/edd3079f
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/edd3079f
Branch: refs/heads/master
Commit: edd3079f771d4b70914d79221bc0df8141e1957a
Parents: b28aec9
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 5 12:43:23 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 5 12:43:23 2015 -0700
----------------------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/edd3079f/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
index 7d9353a..68d3255 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcFile.java
@@ -107,6 +107,7 @@ public final class OrcFile {
ORIGINAL(0),
HIVE_8732(1), // corrupted stripe/file maximum column statistics
HIVE_4243(2), // use real column names from Hive tables
+// Don't use any magic numbers here except for the below:
FUTURE(Integer.MAX_VALUE); // a version from a future writer
private final int id;
@@ -125,7 +126,7 @@ public final class OrcFile {
int max = Integer.MIN_VALUE;
for (WriterVersion v : WriterVersion.values()) {
if (v.id < 0) throw new AssertionError();
- if (v.id > max) {
+ if (v.id > max && FUTURE.id != v.id) {
max = v.id;
}
}
@@ -136,6 +137,7 @@ public final class OrcFile {
}
public static WriterVersion from(int val) {
+ if (val == FUTURE.id) return FUTURE; // Special handling for the magic value.
return values[val];
}
}
[38/51] [abbrv] hive git commit: HIVE-12043. Fix UGI usage in IO
elevator threads. (Siddharth Seth)
Posted by se...@apache.org.
HIVE-12043. Fix UGI usage in IO elevator threads. (Siddharth Seth)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/b861b9ef
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/b861b9ef
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/b861b9ef
Branch: refs/heads/master
Commit: b861b9efbd516d596414e7b68d746c905472b225
Parents: b87f63c
Author: Siddharth Seth <ss...@apache.org>
Authored: Tue Oct 6 18:19:54 2015 -0700
Committer: Siddharth Seth <ss...@apache.org>
Committed: Tue Oct 6 18:19:54 2015 -0700
----------------------------------------------------------------------
.../llap/io/encoded/OrcEncodedDataReader.java | 20 ++++++++++++++++++--
1 file changed, 18 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/b861b9ef/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
----------------------------------------------------------------------
diff --git a/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java b/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
index c934f39..33277ae 100644
--- a/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
+++ b/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
@@ -2,6 +2,7 @@ package org.apache.hadoop.hive.llap.io.encoded;
import java.io.IOException;
import java.nio.ByteBuffer;
+import java.security.PrivilegedExceptionAction;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
@@ -61,6 +62,7 @@ import org.apache.hadoop.hive.ql.io.orc.StripeInformation;
import org.apache.hadoop.hive.ql.io.sarg.SearchArgument;
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hive.common.util.FixedSizedObjectPool;
/**
@@ -121,6 +123,7 @@ public class OrcEncodedDataReader extends CallableWithNdc<Void>
private final String[] columnNames;
private final OrcEncodedDataConsumer consumer;
private final QueryFragmentCounters counters;
+ private final UserGroupInformation ugi;
// Read state.
private int stripeIxFrom;
@@ -156,6 +159,11 @@ public class OrcEncodedDataReader extends CallableWithNdc<Void>
this.columnNames = columnNames;
this.consumer = consumer;
this.counters = counters;
+ try {
+ this.ugi = UserGroupInformation.getCurrentUser();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
}
@Override
@@ -179,7 +187,16 @@ public class OrcEncodedDataReader extends CallableWithNdc<Void>
}
@Override
- protected Void callInternal() throws IOException {
+ protected Void callInternal() throws IOException, InterruptedException {
+ return ugi.doAs(new PrivilegedExceptionAction<Void>() {
+ @Override
+ public Void run() throws Exception {
+ return performDataRead();
+ }
+ });
+ }
+
+ protected Void performDataRead() throws IOException {
long startTime = counters.startTimeCounter();
if (LlapIoImpl.LOGL.isInfoEnabled()) {
LlapIoImpl.LOG.info("Processing data for " + split.getPath());
@@ -653,7 +670,6 @@ public class OrcEncodedDataReader extends CallableWithNdc<Void>
/**
* Determines which RGs need to be read, after stripes have been determined.
* SARG is applied, and readState is populated for each stripe accordingly.
- * @param stripes All stripes in the file (field state is used to determine stripes to read).
*/
private boolean determineRgsToRead(boolean[] globalIncludes, int rowIndexStride,
ArrayList<OrcStripeMetadata> metadata) throws IOException {
[22/51] [abbrv] hive git commit: HIVE-11978 : LLAP: NPE in Expr
toString (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-11978 : LLAP: NPE in Expr toString (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/434b7d47
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/434b7d47
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/434b7d47
Branch: refs/heads/master
Commit: 434b7d47dce3be14f11c0b0eb2400038e6db06c3
Parents: 3c5b4ce
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Sep 28 12:09:59 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Sep 28 12:09:59 2015 -0700
----------------------------------------------------------------------
.../org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/434b7d47/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java b/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
index 655cfb3..b5d2ddf 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java
@@ -141,7 +141,7 @@ public class ExprNodeGenericFuncDesc extends ExprNodeDesc implements
if (i > 0) {
sb.append(", ");
}
- sb.append(chidren.get(i).toString());
+ sb.append(chidren.get(i));
}
}
sb.append(")");
[29/51] [abbrv] hive git commit: HIVE-12013 : LLAP: disable most llap
tests before merge (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12013 : LLAP: disable most llap tests before merge (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/a1bc2ef4
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/a1bc2ef4
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/a1bc2ef4
Branch: refs/heads/master
Commit: a1bc2ef4cda1fb9a42a58b9433bf60737519d32c
Parents: f272ccb
Author: Sergey Shelukhin <se...@apache.org>
Authored: Thu Oct 1 19:29:32 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Thu Oct 1 19:29:32 2015 -0700
----------------------------------------------------------------------
itests/qtest/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/a1bc2ef4/itests/qtest/pom.xml
----------------------------------------------------------------------
diff --git a/itests/qtest/pom.xml b/itests/qtest/pom.xml
index 8c41b5a..bb5b1a1 100644
--- a/itests/qtest/pom.xml
+++ b/itests/qtest/pom.xml
@@ -538,7 +538,7 @@
templatePath="${basedir}/${hive.path.to.root}/ql/src/test/templates/" template="TestCliDriver.vm"
queryDirectory="${basedir}/${hive.path.to.root}/ql/src/test/queries/clientpositive/"
queryFile="${qfile}"
- includeQueryFile="${minitez.query.files},${minitez.query.files.shared}"
+ includeQueryFile="${minitez.query.files}"
queryFileRegex="${qfile_regex}"
clusterMode="llap"
runDisabled="${run_disabled}"
[09/51] [abbrv] hive git commit: HIVE-11893: LLAP: Update llap golden
files after master merge (Prasanth Jayachandran)
Posted by se...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_13.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_13.q.out b/ql/src/test/results/clientpositive/llap/vectorization_13.q.out
index f6aefa4..23e0f5f 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_13.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_13.q.out
@@ -89,19 +89,19 @@ STAGE PLANS:
predicate: (((cfloat < 3569.0) and (10.175 >= cdouble) and (cboolean1 <> 1)) or ((UDFToDouble(ctimestamp1) > 11.0) and (UDFToDouble(ctimestamp2) <> 12.0) and (UDFToDouble(ctinyint) < 9763215.5639))) (type: boolean)
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cboolean1 (type: boolean), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ expressions: ctinyint (type: tinyint), cfloat (type: float), cstring1 (type: string), ctimestamp1 (type: timestamp), cboolean1 (type: boolean)
+ outputColumnNames: ctinyint, cfloat, cstring1, ctimestamp1, cboolean1
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1), sum(_col3), stddev_pop(_col3), stddev_pop(_col1), max(_col3), min(_col1)
- keys: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string)
+ aggregations: max(ctinyint), sum(cfloat), stddev_pop(cfloat), stddev_pop(ctinyint), max(cfloat), min(ctinyint)
+ keys: ctinyint (type: tinyint), cfloat (type: float), cstring1 (type: string), ctimestamp1 (type: timestamp), cboolean1 (type: boolean)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string)
+ key expressions: _col0 (type: tinyint), _col1 (type: float), _col2 (type: string), _col3 (type: timestamp), _col4 (type: boolean)
sort order: +++++
- Map-reduce partition columns: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string)
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: float), _col2 (type: string), _col3 (type: timestamp), _col4 (type: boolean)
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
value expressions: _col5 (type: tinyint), _col6 (type: double), _col7 (type: struct<count:bigint,sum:double,variance:double>), _col8 (type: struct<count:bigint,sum:double,variance:double>), _col9 (type: float), _col10 (type: tinyint)
Execution mode: vectorized, llap
@@ -110,12 +110,12 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0), sum(VALUE._col1), stddev_pop(VALUE._col2), stddev_pop(VALUE._col3), max(VALUE._col4), min(VALUE._col5)
- keys: KEY._col0 (type: boolean), KEY._col1 (type: tinyint), KEY._col2 (type: timestamp), KEY._col3 (type: float), KEY._col4 (type: string)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: float), KEY._col2 (type: string), KEY._col3 (type: timestamp), KEY._col4 (type: boolean)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string), (- _col1) (type: tinyint), _col5 (type: tinyint), ((- _col1) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col1) + _col5))) (type: double), (- _col6) (type: double), (79.553 * UDFToDouble(_col3)) (type: double), _col7 (type: double), (- _col6) (type: double), _col8 (type: double), (UDFToDouble(((- _col1) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col1) + _col5))) / UDFToDouble(_col1)) (type: double), _col10 (type: tinyint)
+ expressions: _col4 (type: boolean), _col0 (type: tinyint), _col3 (type: timestamp), _col1 (type: float), _col2 (type: string), (- _col0) (type: tinyint), _col5 (type: tinyint), ((- _col0) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col0) + _col5))) (type: double), (- _col6) (type: double), (79.553 * UDFToDouble(_col1)) (type: double), _col7 (type: double), (- _col6) (type: double), _col8 (type: double), (UDFToDouble(((- _col0) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col0) + _col5))) / UDFToDouble(_col0)) (type: double), _col10 (type: tinyint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20
Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
@@ -343,19 +343,19 @@ STAGE PLANS:
predicate: (((cfloat < 3569.0) and (10.175 >= cdouble) and (cboolean1 <> 1)) or ((UDFToDouble(ctimestamp1) > -1.388) and (UDFToDouble(ctimestamp2) <> -1.3359999999999999) and (UDFToDouble(ctinyint) < 9763215.5639))) (type: boolean)
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cboolean1 (type: boolean), ctinyint (type: tinyint), ctimestamp1 (type: timestamp), cfloat (type: float), cstring1 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ expressions: ctinyint (type: tinyint), cfloat (type: float), cstring1 (type: string), ctimestamp1 (type: timestamp), cboolean1 (type: boolean)
+ outputColumnNames: ctinyint, cfloat, cstring1, ctimestamp1, cboolean1
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1), sum(_col3), stddev_pop(_col3), stddev_pop(_col1), max(_col3), min(_col1)
- keys: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string)
+ aggregations: max(ctinyint), sum(cfloat), stddev_pop(cfloat), stddev_pop(ctinyint), max(cfloat), min(ctinyint)
+ keys: ctinyint (type: tinyint), cfloat (type: float), cstring1 (type: string), ctimestamp1 (type: timestamp), cboolean1 (type: boolean)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string)
+ key expressions: _col0 (type: tinyint), _col1 (type: float), _col2 (type: string), _col3 (type: timestamp), _col4 (type: boolean)
sort order: +++++
- Map-reduce partition columns: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string)
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: float), _col2 (type: string), _col3 (type: timestamp), _col4 (type: boolean)
Statistics: Num rows: 2730 Data size: 586959 Basic stats: COMPLETE Column stats: NONE
value expressions: _col5 (type: tinyint), _col6 (type: double), _col7 (type: struct<count:bigint,sum:double,variance:double>), _col8 (type: struct<count:bigint,sum:double,variance:double>), _col9 (type: float), _col10 (type: tinyint)
Execution mode: vectorized, llap
@@ -364,12 +364,12 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0), sum(VALUE._col1), stddev_pop(VALUE._col2), stddev_pop(VALUE._col3), max(VALUE._col4), min(VALUE._col5)
- keys: KEY._col0 (type: boolean), KEY._col1 (type: tinyint), KEY._col2 (type: timestamp), KEY._col3 (type: float), KEY._col4 (type: string)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: float), KEY._col2 (type: string), KEY._col3 (type: timestamp), KEY._col4 (type: boolean)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: boolean), _col1 (type: tinyint), _col2 (type: timestamp), _col3 (type: float), _col4 (type: string), (- _col1) (type: tinyint), _col5 (type: tinyint), ((- _col1) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col1) + _col5))) (type: double), (- _col6) (type: double), (79.553 * UDFToDouble(_col3)) (type: double), _col7 (type: double), (- _col6) (type: double), _col8 (type: double), (UDFToDouble(((- _col1) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col1) + _col5))) / UDFToDouble(_col1)) (type: double), _col10 (type: tinyint)
+ expressions: _col4 (type: boolean), _col0 (type: tinyint), _col3 (type: timestamp), _col1 (type: float), _col2 (type: string), (- _col0) (type: tinyint), _col5 (type: tinyint), ((- _col0) + _col5) (type: tinyint), _col6 (type: double), (_col6 * UDFToDouble(((- _col0) + _col5))) (type: double), (- _col6) (type: double), (79.553 * UDFToDouble(_col1)) (type: double), _col7 (type: double), (- _col6) (type: double), _col8 (type: double), (UDFToDouble(((- _col0) + _col5)) - 10.175) (type: double), (- (- _col6)) (type: double), (-26.28 / (- (- _col6))) (type: double), _col9 (type: float), ((_col6 * UDFToDouble(((- _col0) + _col5))) / UDFToDouble(_col0)) (type: double), _col10 (type: tinyint)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20
Statistics: Num rows: 1365 Data size: 293479 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_15.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_15.q.out b/ql/src/test/results/clientpositive/llap/vectorization_15.q.out
index 0b48cd6..062b63d 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_15.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_15.q.out
@@ -85,19 +85,19 @@ STAGE PLANS:
predicate: ((cstring2 like '%ss%') or (cstring1 like '10%') or ((cint >= -75) and (UDFToShort(ctinyint) = csmallint) and (cdouble >= -3728.0))) (type: boolean)
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cfloat (type: float), cboolean1 (type: boolean), cdouble (type: double), cstring1 (type: string), ctinyint (type: tinyint), cint (type: int), ctimestamp1 (type: timestamp)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
+ expressions: ctinyint (type: tinyint), cint (type: int), cfloat (type: float), cdouble (type: double), cstring1 (type: string), ctimestamp1 (type: timestamp), cboolean1 (type: boolean)
+ outputColumnNames: ctinyint, cint, cfloat, cdouble, cstring1, ctimestamp1, cboolean1
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: stddev_samp(_col0), min(_col2), stddev_samp(_col4), var_pop(_col4), var_samp(_col5), stddev_pop(_col5)
- keys: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp)
+ aggregations: stddev_samp(cfloat), min(cdouble), stddev_samp(ctinyint), var_pop(ctinyint), var_samp(cint), stddev_pop(cint)
+ keys: ctinyint (type: tinyint), cint (type: int), cfloat (type: float), cdouble (type: double), cstring1 (type: string), ctimestamp1 (type: timestamp), cboolean1 (type: boolean)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp)
+ key expressions: _col0 (type: tinyint), _col1 (type: int), _col2 (type: float), _col3 (type: double), _col4 (type: string), _col5 (type: timestamp), _col6 (type: boolean)
sort order: +++++++
- Map-reduce partition columns: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp)
+ Map-reduce partition columns: _col0 (type: tinyint), _col1 (type: int), _col2 (type: float), _col3 (type: double), _col4 (type: string), _col5 (type: timestamp), _col6 (type: boolean)
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
value expressions: _col7 (type: struct<count:bigint,sum:double,variance:double>), _col8 (type: double), _col9 (type: struct<count:bigint,sum:double,variance:double>), _col10 (type: struct<count:bigint,sum:double,variance:double>), _col11 (type: struct<count:bigint,sum:double,variance:double>), _col12 (type: struct<count:bigint,sum:double,variance:double>)
Execution mode: vectorized, llap
@@ -106,12 +106,12 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: stddev_samp(VALUE._col0), min(VALUE._col1), stddev_samp(VALUE._col2), var_pop(VALUE._col3), var_samp(VALUE._col4), stddev_pop(VALUE._col5)
- keys: KEY._col0 (type: float), KEY._col1 (type: boolean), KEY._col2 (type: double), KEY._col3 (type: string), KEY._col4 (type: tinyint), KEY._col5 (type: int), KEY._col6 (type: timestamp)
+ keys: KEY._col0 (type: tinyint), KEY._col1 (type: int), KEY._col2 (type: float), KEY._col3 (type: double), KEY._col4 (type: string), KEY._col5 (type: timestamp), KEY._col6 (type: boolean)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: float), _col1 (type: boolean), _col2 (type: double), _col3 (type: string), _col4 (type: tinyint), _col5 (type: int), _col6 (type: timestamp), _col7 (type: double), (-26.28 - UDFToDouble(_col5)) (type: double), _col8 (type: double), (_col2 * 79.553) (type: double), (33.0 % _col0) (type: float), _col9 (type: double), _col10 (type: double), (-23.0 % _col2) (type: double), (- _col4) (type: tinyint), _col11 (type: double), (UDFToFloat(_col5) - _col0) (type: float), (-23 % UDFToInteger(_col4)) (type: int), (- (-26.28 - UDFToDouble(_col5))) (type: double), _col12 (type: double)
+ expressions: _col2 (type: float), _col6 (type: boolean), _col3 (type: double), _col4 (type: string), _col0 (type: tinyint), _col1 (type: int), _col5 (type: timestamp), _col7 (type: double), (-26.28 - UDFToDouble(_col1)) (type: double), _col8 (type: double), (_col3 * 79.553) (type: double), (33.0 % _col2) (type: float), _col9 (type: double), _col10 (type: double), (-23.0 % _col3) (type: double), (- _col0) (type: tinyint), _col11 (type: double), (UDFToFloat(_col1) - _col2) (type: float), (-23 % UDFToInteger(_col0)) (type: int), (- (-26.28 - UDFToDouble(_col1))) (type: double), _col12 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_16.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_16.q.out b/ql/src/test/results/clientpositive/llap/vectorization_16.q.out
index 501f3fa..c1cdfe8 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_16.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_16.q.out
@@ -62,19 +62,19 @@ STAGE PLANS:
predicate: ((cstring2 like '%b%') and ((cdouble >= -1.389) or (cstring1 < 'a'))) (type: boolean)
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cstring1 (type: string), cdouble (type: double), ctimestamp1 (type: timestamp)
- outputColumnNames: _col0, _col1, _col2
+ expressions: cdouble (type: double), cstring1 (type: string), ctimestamp1 (type: timestamp)
+ outputColumnNames: cdouble, cstring1, ctimestamp1
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1), stddev_samp(_col1), min(_col1)
- keys: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp)
+ aggregations: count(cdouble), stddev_samp(cdouble), min(cdouble)
+ keys: cdouble (type: double), cstring1 (type: string), ctimestamp1 (type: timestamp)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp)
+ key expressions: _col0 (type: double), _col1 (type: string), _col2 (type: timestamp)
sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp)
+ Map-reduce partition columns: _col0 (type: double), _col1 (type: string), _col2 (type: timestamp)
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
value expressions: _col3 (type: bigint), _col4 (type: struct<count:bigint,sum:double,variance:double>), _col5 (type: double)
Execution mode: vectorized, llap
@@ -83,12 +83,12 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), stddev_samp(VALUE._col1), min(VALUE._col2)
- keys: KEY._col0 (type: string), KEY._col1 (type: double), KEY._col2 (type: timestamp)
+ keys: KEY._col0 (type: double), KEY._col1 (type: string), KEY._col2 (type: timestamp)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp), (_col1 - 9763215.5639) (type: double), (- (_col1 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col1) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double)
+ expressions: _col1 (type: string), _col0 (type: double), _col2 (type: timestamp), (_col0 - 9763215.5639) (type: double), (- (_col0 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col0) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE
File Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_9.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_9.q.out b/ql/src/test/results/clientpositive/llap/vectorization_9.q.out
index 501f3fa..c1cdfe8 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_9.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_9.q.out
@@ -62,19 +62,19 @@ STAGE PLANS:
predicate: ((cstring2 like '%b%') and ((cdouble >= -1.389) or (cstring1 < 'a'))) (type: boolean)
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cstring1 (type: string), cdouble (type: double), ctimestamp1 (type: timestamp)
- outputColumnNames: _col0, _col1, _col2
+ expressions: cdouble (type: double), cstring1 (type: string), ctimestamp1 (type: timestamp)
+ outputColumnNames: cdouble, cstring1, ctimestamp1
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col1), stddev_samp(_col1), min(_col1)
- keys: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp)
+ aggregations: count(cdouble), stddev_samp(cdouble), min(cdouble)
+ keys: cdouble (type: double), cstring1 (type: string), ctimestamp1 (type: timestamp)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp)
+ key expressions: _col0 (type: double), _col1 (type: string), _col2 (type: timestamp)
sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp)
+ Map-reduce partition columns: _col0 (type: double), _col1 (type: string), _col2 (type: timestamp)
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
value expressions: _col3 (type: bigint), _col4 (type: struct<count:bigint,sum:double,variance:double>), _col5 (type: double)
Execution mode: vectorized, llap
@@ -83,12 +83,12 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), stddev_samp(VALUE._col1), min(VALUE._col2)
- keys: KEY._col0 (type: string), KEY._col1 (type: double), KEY._col2 (type: timestamp)
+ keys: KEY._col0 (type: double), KEY._col1 (type: string), KEY._col2 (type: timestamp)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: string), _col1 (type: double), _col2 (type: timestamp), (_col1 - 9763215.5639) (type: double), (- (_col1 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col1) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double)
+ expressions: _col1 (type: string), _col0 (type: double), _col2 (type: timestamp), (_col0 - 9763215.5639) (type: double), (- (_col0 - 9763215.5639)) (type: double), _col3 (type: bigint), _col4 (type: double), (- _col4) (type: double), (_col4 * UDFToDouble(_col3)) (type: double), _col5 (type: double), (9763215.5639 / _col0) (type: double), (UDFToDouble(_col3) / -1.389) (type: double), _col4 (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12
Statistics: Num rows: 2048 Data size: 440327 Basic stats: COMPLETE Column stats: NONE
File Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out b/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
index ece6719..1d2f8dd 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_limit.q.out
@@ -252,10 +252,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint)
- outputColumnNames: _col0
+ outputColumnNames: ctinyint
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: tinyint)
+ keys: ctinyint (type: tinyint)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
@@ -343,10 +343,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint), cdouble (type: double)
- outputColumnNames: _col0, _col1
+ outputColumnNames: ctinyint, cdouble
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: tinyint), _col1 (type: double)
+ keys: ctinyint (type: tinyint), cdouble (type: double)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
@@ -471,11 +471,11 @@ STAGE PLANS:
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cdouble (type: double), ctinyint (type: tinyint)
- outputColumnNames: _col0, _col1
+ outputColumnNames: cdouble, ctinyint
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col1)
- keys: _col0 (type: double)
+ aggregations: sum(ctinyint)
+ keys: cdouble (type: double)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_pushdown.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_pushdown.q.out b/ql/src/test/results/clientpositive/llap/vectorization_pushdown.q.out
index a02cd59..6ae1d71 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_pushdown.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_pushdown.q.out
@@ -25,10 +25,10 @@ STAGE PLANS:
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cbigint (type: bigint)
- outputColumnNames: _col0
+ outputColumnNames: cbigint
Statistics: Num rows: 4096 Data size: 880654 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(_col0)
+ aggregations: avg(cbigint)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out b/ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out
index d5ea930..82f5b12 100644
--- a/ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorization_short_regress.q.out
@@ -156,10 +156,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cint (type: int), cdouble (type: double), csmallint (type: smallint), cfloat (type: float), ctinyint (type: tinyint)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ outputColumnNames: cint, cdouble, csmallint, cfloat, ctinyint
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(_col0), sum(_col1), stddev_pop(_col0), stddev_samp(_col2), var_samp(_col0), avg(_col3), stddev_samp(_col0), min(_col4), count(_col2)
+ aggregations: avg(cint), sum(cdouble), stddev_pop(cint), stddev_samp(csmallint), var_samp(cint), avg(cfloat), stddev_samp(cint), min(ctinyint), count(csmallint)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: NONE
@@ -369,10 +369,10 @@ STAGE PLANS:
Statistics: Num rows: 6826 Data size: 1467614 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cint (type: int), cbigint (type: bigint), csmallint (type: smallint), cdouble (type: double), ctinyint (type: tinyint)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ outputColumnNames: cint, cbigint, csmallint, cdouble, ctinyint
Statistics: Num rows: 6826 Data size: 1467614 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0), var_pop(_col1), stddev_pop(_col2), max(_col3), avg(_col4), min(_col0), min(_col3), stddev_samp(_col2), var_samp(_col0)
+ aggregations: max(cint), var_pop(cbigint), stddev_pop(csmallint), max(cdouble), avg(ctinyint), min(cint), min(cdouble), stddev_samp(csmallint), var_samp(cint)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 1 Data size: 24 Basic stats: COMPLETE Column stats: NONE
@@ -573,10 +573,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cbigint (type: bigint), ctinyint (type: tinyint), csmallint (type: smallint), cint (type: int), cdouble (type: double)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4
+ outputColumnNames: cbigint, ctinyint, csmallint, cint, cdouble
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: var_pop(_col0), count(), max(_col1), stddev_pop(_col2), max(_col3), stddev_samp(_col4), count(_col1), avg(_col1)
+ aggregations: var_pop(cbigint), count(), max(ctinyint), stddev_pop(csmallint), max(cint), stddev_samp(cdouble), count(ctinyint), avg(ctinyint)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 1 Data size: 24 Basic stats: COMPLETE Column stats: NONE
@@ -756,10 +756,10 @@ STAGE PLANS:
Statistics: Num rows: 8874 Data size: 1907941 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint), cbigint (type: bigint), cint (type: int), cfloat (type: float)
- outputColumnNames: _col0, _col1, _col2, _col3
+ outputColumnNames: ctinyint, cbigint, cint, cfloat
Statistics: Num rows: 8874 Data size: 1907941 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(_col0), max(_col1), stddev_samp(_col2), var_pop(_col2), var_pop(_col1), max(_col3)
+ aggregations: avg(ctinyint), max(cbigint), stddev_samp(cint), var_pop(cint), var_pop(cbigint), max(cfloat)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
@@ -1886,11 +1886,11 @@ STAGE PLANS:
Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: csmallint (type: smallint), cbigint (type: bigint), ctinyint (type: tinyint)
- outputColumnNames: _col0, _col1, _col2
+ outputColumnNames: csmallint, cbigint, ctinyint
Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: stddev_samp(_col0), sum(_col1), var_pop(_col2), count()
- keys: _col0 (type: smallint)
+ aggregations: stddev_samp(csmallint), sum(cbigint), var_pop(ctinyint), count()
+ keys: csmallint (type: smallint)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4
Statistics: Num rows: 2503 Data size: 538153 Basic stats: COMPLETE Column stats: NONE
@@ -2094,11 +2094,11 @@ STAGE PLANS:
Statistics: Num rows: 2654 Data size: 570619 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cdouble (type: double), cfloat (type: float)
- outputColumnNames: _col0, _col1
+ outputColumnNames: cdouble, cfloat
Statistics: Num rows: 2654 Data size: 570619 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: var_samp(_col0), count(_col1), sum(_col1), var_pop(_col0), stddev_pop(_col0), sum(_col0)
- keys: _col0 (type: double)
+ aggregations: var_samp(cdouble), count(cfloat), sum(cfloat), var_pop(cdouble), stddev_pop(cdouble), sum(cdouble)
+ keys: cdouble (type: double)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 2654 Data size: 570619 Basic stats: COMPLETE Column stats: NONE
@@ -2349,19 +2349,19 @@ STAGE PLANS:
predicate: ((UDFToDouble(ctimestamp1) <> 0.0) and (((-257 <> UDFToInteger(ctinyint)) and cboolean2 is not null and cstring1 regexp '.*ss' and (-3.0 < UDFToDouble(ctimestamp1))) or (UDFToDouble(ctimestamp2) = -5.0) or ((UDFToDouble(ctimestamp1) < 0.0) and (cstring2 like '%b%')) or (cdouble = UDFToDouble(cint)) or (cboolean1 is null and (cfloat < UDFToFloat(cint))))) (type: boolean)
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ctimestamp1 (type: timestamp), cstring1 (type: string), cint (type: int), csmallint (type: smallint), ctinyint (type: tinyint), cfloat (type: float), cdouble (type: double)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
+ expressions: cstring1 (type: string), ctimestamp1 (type: timestamp), cint (type: int), csmallint (type: smallint), ctinyint (type: tinyint), cfloat (type: float), cdouble (type: double)
+ outputColumnNames: cstring1, ctimestamp1, cint, csmallint, ctinyint, cfloat, cdouble
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: stddev_pop(_col2), avg(_col3), count(), min(_col4), var_samp(_col3), var_pop(_col5), avg(_col2), var_samp(_col5), avg(_col5), min(_col6), var_pop(_col3), stddev_pop(_col4), sum(_col2)
- keys: _col0 (type: timestamp), _col1 (type: string)
+ aggregations: stddev_pop(cint), avg(csmallint), count(), min(ctinyint), var_samp(csmallint), var_pop(cfloat), avg(cint), var_samp(cfloat), avg(cfloat), min(cdouble), var_pop(csmallint), stddev_pop(ctinyint), sum(cint)
+ keys: cstring1 (type: string), ctimestamp1 (type: timestamp)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: timestamp), _col1 (type: string)
+ key expressions: _col0 (type: string), _col1 (type: timestamp)
sort order: ++
- Map-reduce partition columns: _col0 (type: timestamp), _col1 (type: string)
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: timestamp)
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: struct<count:bigint,sum:double,variance:double>), _col3 (type: struct<count:bigint,sum:double,input:smallint>), _col4 (type: bigint), _col5 (type: tinyint), _col6 (type: struct<count:bigint,sum:double,variance:double>), _col7 (type: struct<count:bigint,sum:double,variance:double>), _col8 (type: struct<count:bigint,sum:double,input:int>), _col9 (type: struct<count:bigint,sum:double,variance:double>), _col10 (type: struct<count:bigint,sum:double,input:float>), _col11 (type: double), _col12 (type: struct<count:bigint,sum:double,variance:double>), _col13 (type: struct<count:bigint,sum:double,variance:double>), _col14 (type: bigint)
Execution mode: vectorized, llap
@@ -2370,12 +2370,12 @@ STAGE PLANS:
Reduce Operator Tree:
Group By Operator
aggregations: stddev_pop(VALUE._col0), avg(VALUE._col1), count(VALUE._col2), min(VALUE._col3), var_samp(VALUE._col4), var_pop(VALUE._col5), avg(VALUE._col6), var_samp(VALUE._col7), avg(VALUE._col8), min(VALUE._col9), var_pop(VALUE._col10), stddev_pop(VALUE._col11), sum(VALUE._col12)
- keys: KEY._col0 (type: timestamp), KEY._col1 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: timestamp)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: timestamp), _col1 (type: string), _col2 (type: double), (_col2 * 10.175) (type: double), (- _col2) (type: double), _col3 (type: double), (- _col2) (type: double), (-26.28 - _col2) (type: double), _col4 (type: bigint), (- _col4) (type: bigint), ((-26.28 - _col2) * (- _col2)) (type: double), _col5 (type: tinyint), (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4))) (type: double), (- (_col2 * 10.175)) (type: double), _col6 (type: double), (_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- _col2)) (type: double), (UDFToDouble((- _col4)) / _col2) (type: double), _col7 (type: double), (10.175 / _col3) (type: double), _col8 (type: double), _col9 (type: double), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- (_col2 * 10.175))) (type: double), _col10 (type: double), (((_col6 + (((-26.28 - _col2) * (-
_col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) * 10.175) (type: double), (10.175 % (10.175 / _col3)) (type: double), (- _col5) (type: tinyint), _col11 (type: double), _col12 (type: double), (- ((-26.28 - _col2) * (- _col2))) (type: double), ((- _col2) % _col10) (type: double), (-26.28 / UDFToDouble((- _col5))) (type: double), _col13 (type: double), _col14 (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) / _col7) (type: double), (- (- _col4)) (type: bigint), _col4 (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) % -26.28) (type: double)
+ expressions: _col1 (type: timestamp), _col0 (type: string), _col2 (type: double), (_col2 * 10.175) (type: double), (- _col2) (type: double), _col3 (type: double), (- _col2) (type: double), (-26.28 - _col2) (type: double), _col4 (type: bigint), (- _col4) (type: bigint), ((-26.28 - _col2) * (- _col2)) (type: double), _col5 (type: tinyint), (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4))) (type: double), (- (_col2 * 10.175)) (type: double), _col6 (type: double), (_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- _col2)) (type: double), (UDFToDouble((- _col4)) / _col2) (type: double), _col7 (type: double), (10.175 / _col3) (type: double), _col8 (type: double), _col9 (type: double), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) (type: double), (- (- (_col2 * 10.175))) (type: double), _col10 (type: double), (((_col6 + (((-26.28 - _col2) * (-
_col2)) * UDFToDouble((- _col4)))) - (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) * 10.175) (type: double), (10.175 % (10.175 / _col3)) (type: double), (- _col5) (type: tinyint), _col11 (type: double), _col12 (type: double), (- ((-26.28 - _col2) * (- _col2))) (type: double), ((- _col2) % _col10) (type: double), (-26.28 / UDFToDouble((- _col5))) (type: double), _col13 (type: double), _col14 (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) / _col7) (type: double), (- (- _col4)) (type: bigint), _col4 (type: bigint), ((_col6 + (((-26.28 - _col2) * (- _col2)) * UDFToDouble((- _col4)))) % -26.28) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col37, _col38
Statistics: Num rows: 6144 Data size: 1320982 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
@@ -2683,11 +2683,11 @@ STAGE PLANS:
Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cboolean1 (type: boolean), cfloat (type: float), cbigint (type: bigint), cint (type: int), cdouble (type: double), ctinyint (type: tinyint), csmallint (type: smallint)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
+ outputColumnNames: cboolean1, cfloat, cbigint, cint, cdouble, ctinyint, csmallint
Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1), sum(_col2), var_samp(_col3), avg(_col4), min(_col2), var_pop(_col2), sum(_col3), stddev_samp(_col5), stddev_pop(_col6), avg(_col3)
- keys: _col0 (type: boolean)
+ aggregations: max(cfloat), sum(cbigint), var_samp(cint), avg(cdouble), min(cbigint), var_pop(cbigint), sum(cint), stddev_samp(ctinyint), stddev_pop(csmallint), avg(cint)
+ keys: cboolean1 (type: boolean)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10
Statistics: Num rows: 4778 Data size: 1027287 Basic stats: COMPLETE Column stats: NONE
@@ -2923,10 +2923,10 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: i (type: int)
- outputColumnNames: _col0
+ outputColumnNames: i
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(i)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -3101,10 +3101,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint)
- outputColumnNames: _col0
+ outputColumnNames: ctinyint
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(ctinyint)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -3168,10 +3168,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cint (type: int)
- outputColumnNames: _col0
+ outputColumnNames: cint
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(cint)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -3235,10 +3235,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cfloat (type: float)
- outputColumnNames: _col0
+ outputColumnNames: cfloat
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(cfloat)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -3302,10 +3302,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cstring1 (type: string)
- outputColumnNames: _col0
+ outputColumnNames: cstring1
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(cstring1)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -3369,10 +3369,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cboolean1 (type: boolean)
- outputColumnNames: _col0
+ outputColumnNames: cboolean1
Statistics: Num rows: 12288 Data size: 168 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(_col0)
+ aggregations: count(cboolean1)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorized_distinct_gby.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_distinct_gby.q.out b/ql/src/test/results/clientpositive/llap/vectorized_distinct_gby.q.out
index df4a301..67dea8e 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_distinct_gby.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_distinct_gby.q.out
@@ -38,10 +38,10 @@ STAGE PLANS:
Statistics: Num rows: 5 Data size: 40 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: a (type: int)
- outputColumnNames: _col0
+ outputColumnNames: a
Statistics: Num rows: 5 Data size: 40 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: a (type: int)
mode: final
outputColumnNames: _col0
Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: NONE
@@ -109,10 +109,10 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: cint (type: int)
- outputColumnNames: _col0
+ outputColumnNames: cint
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: cint (type: int)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 12288 Data size: 2641964 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out b/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
index 41331e7..182d6f7 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_dynamic_partition_pruning.q.out
@@ -59,10 +59,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
@@ -1831,7 +1831,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11
#### A masked pattern was here ####
1000
-Warning: Shuffle Join MERGEJOIN[23][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[21][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: -- parent is reduce tasks
EXPLAIN select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
@@ -1871,10 +1871,10 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: '2008-04-08' (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
@@ -1938,7 +1938,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[23][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
+Warning: Shuffle Join MERGEJOIN[21][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
PREHOOK: Input: default@srcpart
@@ -2835,10 +2835,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -2854,10 +2854,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3039,10 +3039,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3058,10 +3058,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3230,20 +3230,16 @@ STAGE PLANS:
alias: srcpart
filterExpr: ds is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: ds (type: string)
+ Group By Operator
+ keys: ds (type: string)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 10
Map Operator Tree:
@@ -3252,10 +3248,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -3270,20 +3266,16 @@ STAGE PLANS:
alias: srcpart
filterExpr: ds is not null (type: boolean)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: ds (type: string)
+ Group By Operator
+ keys: ds (type: string)
+ mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Execution mode: llap
Map 7
Map Operator Tree:
@@ -3292,10 +3284,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -4270,7 +4262,7 @@ POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-09/hr=11
#### A masked pattern was here ####
1000
-Warning: Map Join MAPJOIN[23][bigTable=?] in task 'Map 1' is a cross product
+Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
PREHOOK: query: -- parent is reduce tasks
EXPLAIN select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
@@ -4326,10 +4318,10 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: '2008-04-08' (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: string)
+ keys: ds (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
@@ -4374,7 +4366,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Map Join MAPJOIN[23][bigTable=?] in task 'Map 1' is a cross product
+Warning: Map Join MAPJOIN[21][bigTable=?] in task 'Map 1' is a cross product
PREHOOK: query: select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'
PREHOOK: type: QUERY
PREHOOK: Input: default@srcpart
@@ -4984,10 +4976,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col0)
+ aggregations: max(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
@@ -5003,10 +4995,10 @@ STAGE PLANS:
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ds (type: string)
- outputColumnNames: _col0
+ outputColumnNames: ds
Statistics: Num rows: 2000 Data size: 21248 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0)
+ aggregations: min(ds)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 84 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out b/ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out
index b28ef86..899967b 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out
@@ -75,19 +75,15 @@ STAGE PLANS:
1 Map 4
Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE
HybridGraceHashJoin: true
- Select Operator
- expressions: _col1 (type: double)
+ Group By Operator
+ aggregations: sum(_col1)
+ mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 7433 Data size: 1598388 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col0)
- mode: hash
- outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: double)
+ value expressions: _col0 (type: double)
Execution mode: vectorized, llap
Map 4
Map Operator Tree:
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out b/ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out
index 9ddbb91..dfb14d0 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_parquet.q.out
@@ -135,11 +135,11 @@ STAGE PLANS:
Statistics: Num rows: 12288 Data size: 73728 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctinyint (type: tinyint), cint (type: int), csmallint (type: smallint), cstring1 (type: string), cfloat (type: float), cdouble (type: double)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ outputColumnNames: ctinyint, cint, csmallint, cstring1, cfloat, cdouble
Statistics: Num rows: 12288 Data size: 73728 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(_col1), min(_col2), count(_col3), avg(_col4), stddev_pop(_col5)
- keys: _col0 (type: tinyint)
+ aggregations: max(cint), min(csmallint), count(cstring1), avg(cfloat), stddev_pop(cdouble)
+ keys: ctinyint (type: tinyint)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 12288 Data size: 73728 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/llap/vectorized_timestamp_funcs.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vectorized_timestamp_funcs.q.out b/ql/src/test/results/clientpositive/llap/vectorized_timestamp_funcs.q.out
index 41a252d..b0fe5e2 100644
--- a/ql/src/test/results/clientpositive/llap/vectorized_timestamp_funcs.q.out
+++ b/ql/src/test/results/clientpositive/llap/vectorized_timestamp_funcs.q.out
@@ -644,10 +644,10 @@ STAGE PLANS:
Statistics: Num rows: 40 Data size: 84 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctimestamp1 (type: timestamp)
- outputColumnNames: _col0
+ outputColumnNames: ctimestamp1
Statistics: Num rows: 40 Data size: 84 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col0), max(_col0), count(_col0), count()
+ aggregations: min(ctimestamp1), max(ctimestamp1), count(ctimestamp1), count()
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 1 Data size: 96 Basic stats: COMPLETE Column stats: NONE
@@ -725,10 +725,10 @@ STAGE PLANS:
Statistics: Num rows: 40 Data size: 84 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctimestamp1 (type: timestamp)
- outputColumnNames: _col0
+ outputColumnNames: ctimestamp1
Statistics: Num rows: 40 Data size: 84 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col0)
+ aggregations: sum(ctimestamp1)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
@@ -816,10 +816,10 @@ STAGE PLANS:
Statistics: Num rows: 40 Data size: 84 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: ctimestamp1 (type: timestamp)
- outputColumnNames: _col0
+ outputColumnNames: ctimestamp1
Statistics: Num rows: 40 Data size: 84 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: avg(_col0), variance(_col0), var_pop(_col0), var_samp(_col0), std(_col0), stddev(_col0), stddev_pop(_col0), stddev_samp(_col0)
+ aggregations: avg(ctimestamp1), variance(ctimestamp1), var_pop(ctimestamp1), var_samp(ctimestamp1), std(ctimestamp1), stddev(ctimestamp1), stddev_pop(ctimestamp1), stddev_samp(ctimestamp1)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/73eb9239/ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out b/ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out
index bcae2fc..5ed3d9e 100644
--- a/ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out
+++ b/ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out
@@ -14393,6 +14393,38 @@ POSTHOOK: Lineage: src10_3.key SIMPLE [(src)src.FieldSchema(name:key, type:strin
POSTHOOK: Lineage: src10_3.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
POSTHOOK: Lineage: src10_4.key SIMPLE [(src)src.FieldSchema(name:key, type:string, comment:default), ]
POSTHOOK: Lineage: src10_4.value SIMPLE [(src)src.FieldSchema(name:value, type:string, comment:default), ]
+PREHOOK: query: analyze table src10_1 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_1
+PREHOOK: Output: default@src10_1
+POSTHOOK: query: analyze table src10_1 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_1
+POSTHOOK: Output: default@src10_1
+PREHOOK: query: analyze table src10_2 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_2
+PREHOOK: Output: default@src10_2
+POSTHOOK: query: analyze table src10_2 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_2
+POSTHOOK: Output: default@src10_2
+PREHOOK: query: analyze table src10_3 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_3
+PREHOOK: Output: default@src10_3
+POSTHOOK: query: analyze table src10_3 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_3
+POSTHOOK: Output: default@src10_3
+PREHOOK: query: analyze table src10_4 compute statistics
+PREHOOK: type: QUERY
+PREHOOK: Input: default@src10_4
+PREHOOK: Output: default@src10_4
+POSTHOOK: query: analyze table src10_4 compute statistics
+POSTHOOK: type: QUERY
+POSTHOOK: Input: default@src10_4
+POSTHOOK: Output: default@src10_4
PREHOOK: query: -- When we convert the Join of sub1 and sub0 into a MapJoin,
-- we can use a single MR job to evaluate this entire query.
explain
[28/51] [abbrv] hive git commit: HIVE-12000 : LLAP: Merge master into
branch (Sergey Shelukhin)
Posted by se...@apache.org.
HIVE-12000 : LLAP: Merge master into branch (Sergey Shelukhin)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/f272ccb2
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/f272ccb2
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/f272ccb2
Branch: refs/heads/master
Commit: f272ccb25bc495d600a713058f6c2082ebd6d966
Parents: 7273a4c 064e37c
Author: Sergey Shelukhin <se...@apache.org>
Authored: Wed Sep 30 12:22:32 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Wed Sep 30 12:22:32 2015 -0700
----------------------------------------------------------------------
.../common/metrics/common/MetricsConstant.java | 1 +
.../hive/hcatalog/templeton/AppConfig.java | 21 +++
.../apache/hive/hcatalog/templeton/Server.java | 12 +-
itests/qtest/pom.xml | 2 +-
.../test/resources/testconfiguration.properties | 27 ++-
.../hadoop/hive/ql/io/orc/MetadataReader.java | 2 +-
.../hive/ql/io/orc/MetadataReaderImpl.java | 20 ++-
.../hadoop/hive/ql/io/orc/OrcInputFormat.java | 49 ++++++
.../hadoop/hive/ql/io/orc/ReaderImpl.java | 12 +-
.../hadoop/hive/ql/io/orc/RecordReaderImpl.java | 35 ++++
.../hive/ql/io/orc/RecordReaderUtils.java | 5 +-
.../hive/ql/optimizer/StatsOptimizer.java | 46 ++++-
.../ql/optimizer/calcite/RelOptHiveTable.java | 15 ++
.../ql/optimizer/physical/Vectorizer.java.rej | 86 ----------
.../hadoop/hive/ql/session/SessionState.java | 9 +-
.../test/queries/clientpositive/add_jar_pfile.q | 8 +
.../clientpositive/metadata_only_queries.q | 15 ++
.../results/clientpositive/add_jar_pfile.q.out | 12 ++
.../clientpositive/metadata_only_queries.q.out | 158 +++++++++++++++++
.../spark/metadata_only_queries.q.out | 170 +++++++++++++++++++
.../tez/metadata_only_queries.q.out | 170 +++++++++++++++++++
.../tez/vector_groupby_reduce.q.out | 70 +++++---
.../clientpositive/vector_groupby_reduce.q.out | 69 +++++---
.../hive/service/cli/operation/Operation.java | 11 ++
.../service/cli/operation/OperationManager.java | 11 ++
.../service/cli/session/HiveSessionProxy.java | 6 +
.../service/cli/session/SessionManager.java | 6 +-
.../thrift/EmbeddedThriftBinaryCLIService.java | 2 +-
.../thrift/ThreadPoolExecutorWithOomHook.java | 55 ++++++
.../cli/thrift/ThriftBinaryCLIService.java | 12 +-
.../service/cli/thrift/ThriftCLIService.java | 3 +
.../cli/thrift/ThriftHttpCLIService.java | 10 +-
.../apache/hive/service/server/HiveServer2.java | 12 +-
.../hive/service/auth/TestPlainSaslHelper.java | 2 +-
.../session/TestPluggableHiveSessionImpl.java | 2 +-
.../cli/session/TestSessionGlobalInitFile.java | 2 +-
36 files changed, 957 insertions(+), 191 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/itests/qtest/pom.xml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/itests/src/test/resources/testconfiguration.properties
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java
index 35cc05f,43d2933..cea324c
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReader.java
@@@ -18,17 -18,102 +18,17 @@@
package org.apache.hadoop.hive.ql.io.orc;
import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.List;
-import org.apache.hadoop.fs.FSDataInputStream;
-import org.apache.hadoop.fs.FileSystem;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hive.common.DiskRange;
-import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.BufferChunk;
+import org.apache.hadoop.hive.ql.io.orc.OrcProto.BloomFilterIndex;
+import org.apache.hadoop.hive.ql.io.orc.OrcProto.RowIndex;
+import org.apache.hadoop.hive.ql.io.orc.OrcProto.StripeFooter;
-import com.google.common.collect.Lists;
+public interface MetadataReader {
+ RecordReaderImpl.Index readRowIndex(StripeInformation stripe, StripeFooter footer,
+ boolean[] included, RowIndex[] indexes, boolean[] sargColumns,
+ BloomFilterIndex[] bloomFilterIndices) throws IOException;
-public class MetadataReader {
- private final FSDataInputStream file;
- private final CompressionCodec codec;
- private final int bufferSize;
- private final int typeCount;
+ StripeFooter readStripeFooter(StripeInformation stripe) throws IOException;
- public MetadataReader(FileSystem fileSystem, Path path,
- CompressionCodec codec, int bufferSize, int typeCount) throws IOException {
- this(fileSystem.open(path), codec, bufferSize, typeCount);
- }
-
- public MetadataReader(FSDataInputStream file,
- CompressionCodec codec, int bufferSize, int typeCount) {
- this.file = file;
- this.codec = codec;
- this.bufferSize = bufferSize;
- this.typeCount = typeCount;
- }
-
- public RecordReaderImpl.Index readRowIndex(StripeInformation stripe, OrcProto.StripeFooter footer,
- boolean[] included, OrcProto.RowIndex[] indexes, boolean[] sargColumns,
- OrcProto.BloomFilterIndex[] bloomFilterIndices) throws IOException {
- if (footer == null) {
- footer = readStripeFooter(stripe);
- }
- if (indexes == null) {
- indexes = new OrcProto.RowIndex[typeCount];
- }
- if (bloomFilterIndices == null) {
- bloomFilterIndices = new OrcProto.BloomFilterIndex[typeCount];
- }
- long offset = stripe.getOffset();
- List<OrcProto.Stream> streams = footer.getStreamsList();
- for (int i = 0; i < streams.size(); i++) {
- OrcProto.Stream stream = streams.get(i);
- OrcProto.Stream nextStream = null;
- if (i < streams.size() - 1) {
- nextStream = streams.get(i+1);
- }
- int col = stream.getColumn();
- int len = (int) stream.getLength();
- // row index stream and bloom filter are interlaced, check if the sarg column contains bloom
- // filter and combine the io to read row index and bloom filters for that column together
- if (stream.hasKind() && (stream.getKind() == OrcProto.Stream.Kind.ROW_INDEX)) {
- boolean readBloomFilter = false;
- if (sargColumns != null && sargColumns[col] &&
- nextStream.getKind() == OrcProto.Stream.Kind.BLOOM_FILTER) {
- len += nextStream.getLength();
- i += 1;
- readBloomFilter = true;
- }
- if ((included == null || included[col]) && indexes[col] == null) {
- byte[] buffer = new byte[len];
- file.readFully(offset, buffer, 0, buffer.length);
- ByteBuffer[] bb = new ByteBuffer[] {ByteBuffer.wrap(buffer)};
- indexes[col] = OrcProto.RowIndex.parseFrom(InStream.create("index",
- bb, new long[]{0}, stream.getLength(), codec, bufferSize));
- if (readBloomFilter) {
- bb[0].position((int) stream.getLength());
- bloomFilterIndices[col] = OrcProto.BloomFilterIndex.parseFrom(
- InStream.create("bloom_filter", bb, new long[]{0}, nextStream.getLength(),
- codec, bufferSize));
- }
- }
- }
- offset += len;
- }
-
- RecordReaderImpl.Index index = new RecordReaderImpl.Index(indexes, bloomFilterIndices);
- return index;
- }
-
- public OrcProto.StripeFooter readStripeFooter(StripeInformation stripe) throws IOException {
- long offset = stripe.getOffset() + stripe.getIndexLength() + stripe.getDataLength();
- int tailLength = (int) stripe.getFooterLength();
-
- // read the footer
- ByteBuffer tailBuf = ByteBuffer.allocate(tailLength);
- file.readFully(offset, tailBuf.array(), tailBuf.arrayOffset(), tailLength);
- return OrcProto.StripeFooter.parseFrom(InStream.create("footer",
- Lists.<DiskRange>newArrayList(new BufferChunk(tailBuf, 0)),
- tailLength, codec, bufferSize));
- }
-
- public void close() throws IOException {
- file.close();
- }
+ void close() throws IOException;
- }
+ }
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
----------------------------------------------------------------------
diff --cc ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
index 83594f7,0000000..1456df3
mode 100644,000000..100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/MetadataReaderImpl.java
@@@ -1,119 -1,0 +1,123 @@@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.io.orc;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.List;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.common.io.DiskRange;
+import org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.BufferChunk;
+
+import com.google.common.collect.Lists;
+
+public class MetadataReaderImpl implements MetadataReader {
+ private final FSDataInputStream file;
+ private final CompressionCodec codec;
+ private final int bufferSize;
+ private final int typeCount;
+
- public MetadataReaderImpl(FileSystem fileSystem, Path path, CompressionCodec codec,
- int bufferSize, int typeCount) throws IOException {
- this.file = fileSystem.open(path);
++ public MetadataReaderImpl(FileSystem fileSystem, Path path,
++ CompressionCodec codec, int bufferSize, int typeCount) throws IOException {
++ this(fileSystem.open(path), codec, bufferSize, typeCount);
++ }
++
++ public MetadataReaderImpl(FSDataInputStream file,
++ CompressionCodec codec, int bufferSize, int typeCount) {
++ this.file = file;
+ this.codec = codec;
+ this.bufferSize = bufferSize;
+ this.typeCount = typeCount;
+ }
+
+ @Override
+ public RecordReaderImpl.Index readRowIndex(StripeInformation stripe,
+ OrcProto.StripeFooter footer, boolean[] included, OrcProto.RowIndex[] indexes,
+ boolean[] sargColumns, OrcProto.BloomFilterIndex[] bloomFilterIndices) throws IOException {
+ if (footer == null) {
+ footer = readStripeFooter(stripe);
+ }
+ if (indexes == null) {
+ indexes = new OrcProto.RowIndex[typeCount];
+ }
+ if (bloomFilterIndices == null) {
+ bloomFilterIndices = new OrcProto.BloomFilterIndex[typeCount];
+ }
+ long offset = stripe.getOffset();
+ List<OrcProto.Stream> streams = footer.getStreamsList();
+ for (int i = 0; i < streams.size(); i++) {
+ OrcProto.Stream stream = streams.get(i);
+ OrcProto.Stream nextStream = null;
+ if (i < streams.size() - 1) {
+ nextStream = streams.get(i+1);
+ }
+ int col = stream.getColumn();
+ int len = (int) stream.getLength();
+ // row index stream and bloom filter are interlaced, check if the sarg column contains bloom
+ // filter and combine the io to read row index and bloom filters for that column together
+ if (stream.hasKind() && (stream.getKind() == OrcProto.Stream.Kind.ROW_INDEX)) {
+ boolean readBloomFilter = false;
+ if (sargColumns != null && sargColumns[col] &&
+ nextStream.getKind() == OrcProto.Stream.Kind.BLOOM_FILTER) {
+ len += nextStream.getLength();
+ i += 1;
+ readBloomFilter = true;
+ }
+ if ((included == null || included[col]) && indexes[col] == null) {
+ byte[] buffer = new byte[len];
++ file.readFully(offset, buffer, 0, buffer.length);
+ ByteBuffer bb = ByteBuffer.wrap(buffer);
- file.seek(offset);
- file.readFully(buffer);
+ indexes[col] = OrcProto.RowIndex.parseFrom(InStream.create(null, "index",
+ Lists.<DiskRange>newArrayList(new BufferChunk(bb, 0)), stream.getLength(),
- codec, bufferSize));
++ codec, bufferSize));
+ if (readBloomFilter) {
+ bb.position((int) stream.getLength());
+ bloomFilterIndices[col] = OrcProto.BloomFilterIndex.parseFrom(InStream.create(
+ null, "bloom_filter", Lists.<DiskRange>newArrayList(new BufferChunk(bb, 0)),
+ nextStream.getLength(), codec, bufferSize));
+ }
+ }
+ }
+ offset += len;
+ }
+
+ RecordReaderImpl.Index index = new RecordReaderImpl.Index(indexes, bloomFilterIndices);
+ return index;
+ }
+
+ @Override
+ public OrcProto.StripeFooter readStripeFooter(StripeInformation stripe) throws IOException {
+ long offset = stripe.getOffset() + stripe.getIndexLength() + stripe.getDataLength();
+ int tailLength = (int) stripe.getFooterLength();
++
+ // read the footer
+ ByteBuffer tailBuf = ByteBuffer.allocate(tailLength);
- file.seek(offset);
- file.readFully(tailBuf.array(), tailBuf.arrayOffset(), tailLength);
++ file.readFully(offset, tailBuf.array(), tailBuf.arrayOffset(), tailLength);
+ return OrcProto.StripeFooter.parseFrom(InStream.create(null, "footer",
+ Lists.<DiskRange>newArrayList(new BufferChunk(tailBuf, 0)),
+ tailLength, codec, bufferSize));
+ }
+
+ @Override
+ public void close() throws IOException {
+ file.close();
+ }
+}
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderImpl.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/java/org/apache/hadoop/hive/ql/io/orc/RecordReaderUtils.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/f272ccb2/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
----------------------------------------------------------------------
[51/51] [abbrv] hive git commit: HIVE-12153 : LLAP: update errata for
bad branch commits (Sergey Shelukhin) ADDENDUM
Posted by se...@apache.org.
HIVE-12153 : LLAP: update errata for bad branch commits (Sergey Shelukhin) ADDENDUM
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/23ede5dd
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/23ede5dd
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/23ede5dd
Branch: refs/heads/master
Commit: 23ede5dd15e39f95228ce78e13be9759658707f0
Parents: eb28deb
Author: Sergey Shelukhin <se...@apache.org>
Authored: Mon Oct 12 16:54:51 2015 -0700
Committer: Sergey Shelukhin <se...@apache.org>
Committed: Mon Oct 12 16:54:51 2015 -0700
----------------------------------------------------------------------
errata.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/23ede5dd/errata.txt
----------------------------------------------------------------------
diff --git a/errata.txt b/errata.txt
index 7df4b9e..1ae5238 100644
--- a/errata.txt
+++ b/errata.txt
@@ -74,5 +74,5 @@ fc6be8faf5c97901ccad33edca8f8f80023b308a llap HIVE-12151 https://issues.ap
3a2e8ee7e47bd31745dfc5f6a29c602e09747f24 llap HIVE-12152 https://issues.apache.org/jira/browse/HIVE-12152
8ed270cb9d8a9c49cccf99402ca92e3df3304d9f llap HIVE-12152 https://issues.apache.org/jira/browse/HIVE-12152
c6565f5d65da9ed5cb452db7e313d0ce7abc1105 llap HIVE-9729 https://issues.apache.org/jira/browse/HIVE-9729
-d8298e1c85a515150562b0df68af89c18c468638 llap HIVE-9418 https://issues.apache.org/jira/browse/HIVE-9729
+d8298e1c85a515150562b0df68af89c18c468638 llap HIVE-9418 https://issues.apache.org/jira/browse/HIVE-9418