You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by px...@apache.org on 2015/09/30 19:06:56 UTC
hive git commit: HIVE-11989: vector_groupby_reduce.q is failing on
CLI and MiniTez drivers on master (Pengcheng Xiong, reviewed by Matt McCline)
Repository: hive
Updated Branches:
refs/heads/master 1f0878939 -> e9b4d7e4e
HIVE-11989: vector_groupby_reduce.q is failing on CLI and MiniTez drivers on master (Pengcheng Xiong, reviewed by Matt McCline)
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/e9b4d7e4
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/e9b4d7e4
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/e9b4d7e4
Branch: refs/heads/master
Commit: e9b4d7e4e89cd68eeda58b2e2e6014ff24d0a690
Parents: 1f08789
Author: Pengcheng Xiong <px...@apache.org>
Authored: Wed Sep 30 10:06:36 2015 -0700
Committer: Pengcheng Xiong <px...@apache.org>
Committed: Wed Sep 30 10:06:36 2015 -0700
----------------------------------------------------------------------
.../tez/vector_groupby_reduce.q.out | 70 +++++++++++++-------
.../clientpositive/vector_groupby_reduce.q.out | 69 +++++++++++++------
2 files changed, 94 insertions(+), 45 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/e9b4d7e4/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out b/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
index fe7e829..1635462 100644
--- a/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
+++ b/ql/src/test/results/clientpositive/tez/vector_groupby_reduce.q.out
@@ -399,7 +399,7 @@ STAGE PLANS:
sort order: +
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
- Reducer 3
+ Reducer 3
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: int)
@@ -562,31 +562,32 @@ STAGE PLANS:
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
- Map 1
+ Map 1
Map Operator Tree:
TableScan
alias: store_sales
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_ticket_number (type: int), ss_item_sk (type: int), ss_quantity (type: int)
- outputColumnNames: _col0, _col1, _col2
+ expressions: ss_item_sk (type: int), ss_ticket_number (type: int), ss_quantity (type: int)
+ outputColumnNames: ss_item_sk, ss_ticket_number, ss_quantity
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col2)
- keys: _col0 (type: int), _col1 (type: int)
+ aggregations: min(ss_quantity)
+ keys: ss_item_sk (type: int), ss_ticket_number (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type: int)
sort order: ++
- Map-reduce partition columns: _col0 (type: int)
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: int)
Execution mode: vectorized
- Reducer 2
+ Reducer 2
Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
@@ -595,18 +596,33 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col1), sum(_col2)
- keys: _col0 (type: int)
- mode: complete
+ aggregations: sum(_col0), sum(_col2)
+ keys: _col1 (type: int)
+ mode: hash
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
- Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
Execution mode: vectorized
- Reducer 3
+ Reducer 3
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: sum(VALUE._col0), sum(VALUE._col1)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Execution mode: vectorized
+ Reducer 4
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: int), VALUE._col0 (type: bigint), VALUE._col1 (type: bigint)
@@ -771,18 +787,18 @@ STAGE PLANS:
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
- Map 1
+ Map 1
Map Operator Tree:
TableScan
alias: store_sales
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_ticket_number (type: int), ss_item_sk (type: int), ss_quantity (type: int)
- outputColumnNames: _col0, _col1, _col2
+ expressions: ss_item_sk (type: int), ss_ticket_number (type: int), ss_quantity (type: int)
+ outputColumnNames: ss_item_sk, ss_ticket_number, ss_quantity
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col2)
- keys: _col0 (type: int), _col1 (type: int)
+ aggregations: min(ss_quantity)
+ keys: ss_item_sk (type: int), ss_ticket_number (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
@@ -793,7 +809,7 @@ STAGE PLANS:
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: int)
Execution mode: vectorized
- Reducer 2
+ Reducer 2
Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
@@ -807,13 +823,17 @@ STAGE PLANS:
mode: complete
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: int), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: bigint)
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: bigint)
Execution mode: vectorized
- Reducer 3
+ Reducer 3
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: int), KEY.reducesinkkey1 (type: int), VALUE._col0 (type: bigint)
http://git-wip-us.apache.org/repos/asf/hive/blob/e9b4d7e4/ql/src/test/results/clientpositive/vector_groupby_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/vector_groupby_reduce.q.out b/ql/src/test/results/clientpositive/vector_groupby_reduce.q.out
index fc1997c..3e7077e 100644
--- a/ql/src/test/results/clientpositive/vector_groupby_reduce.q.out
+++ b/ql/src/test/results/clientpositive/vector_groupby_reduce.q.out
@@ -550,7 +550,8 @@ POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
- Stage-0 depends on stages: Stage-2
+ Stage-3 depends on stages: Stage-2
+ Stage-0 depends on stages: Stage-3
STAGE PLANS:
Stage: Stage-1
@@ -560,19 +561,19 @@ STAGE PLANS:
alias: store_sales
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_ticket_number (type: int), ss_item_sk (type: int), ss_quantity (type: int)
- outputColumnNames: _col0, _col1, _col2
+ expressions: ss_item_sk (type: int), ss_ticket_number (type: int), ss_quantity (type: int)
+ outputColumnNames: ss_item_sk, ss_ticket_number, ss_quantity
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col2)
- keys: _col0 (type: int), _col1 (type: int)
+ aggregations: min(ss_quantity)
+ keys: ss_item_sk (type: int), ss_ticket_number (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int), _col1 (type: int)
sort order: ++
- Map-reduce partition columns: _col0 (type: int)
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: int)
Execution mode: vectorized
@@ -584,11 +585,11 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col1), sum(_col2)
- keys: _col0 (type: int)
- mode: complete
+ aggregations: sum(_col0), sum(_col2)
+ keys: _col1 (type: int)
+ mode: hash
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -603,6 +604,30 @@ STAGE PLANS:
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 500 Data size: 44138 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: sum(VALUE._col0), sum(VALUE._col1)
+ keys: KEY._col0 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+
+ Stage: Stage-3
+ Map Reduce
+ Map Operator Tree:
+ TableScan
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reduce Operator Tree:
@@ -769,12 +794,12 @@ STAGE PLANS:
alias: store_sales
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_ticket_number (type: int), ss_item_sk (type: int), ss_quantity (type: int)
- outputColumnNames: _col0, _col1, _col2
+ expressions: ss_item_sk (type: int), ss_ticket_number (type: int), ss_quantity (type: int)
+ outputColumnNames: ss_item_sk, ss_ticket_number, ss_quantity
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: min(_col2)
- keys: _col0 (type: int), _col1 (type: int)
+ aggregations: min(ss_quantity)
+ keys: ss_item_sk (type: int), ss_ticket_number (type: int)
mode: hash
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 1000 Data size: 88276 Basic stats: COMPLETE Column stats: NONE
@@ -798,12 +823,16 @@ STAGE PLANS:
mode: complete
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: int), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 250 Data size: 22069 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-2
Map Reduce