You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by se...@apache.org on 2017/10/13 00:07:10 UTC
[19/23] hive git commit: HIVE-17726: Using exists may lead to
incorrect results (Vineet Garg, reviewed by Ashutosh Chauhan)
http://git-wip-us.apache.org/repos/asf/hive/blob/9375cf3c/ql/src/test/results/clientpositive/perf/spark/query16.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query16.q.out b/ql/src/test/results/clientpositive/perf/spark/query16.q.out
index 205173a..b74d721 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query16.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query16.q.out
@@ -1,3 +1,4 @@
+Warning: Shuffle Join JOIN[33][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_4]] in Work 'Reducer 17' is a cross product
PREHOOK: query: explain
select
count(distinct cs_order_number) as `order count`
@@ -69,7 +70,7 @@ STAGE PLANS:
Spark
#### A masked pattern was here ####
Vertices:
- Map 9
+ Map 10
Map Operator Tree:
TableScan
alias: call_center
@@ -92,7 +93,7 @@ STAGE PLANS:
Spark
#### A masked pattern was here ####
Vertices:
- Map 7
+ Map 8
Map Operator Tree:
TableScan
alias: date_dim
@@ -114,13 +115,17 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 11 <- Map 10 (GROUP, 24)
- Reducer 13 <- Map 12 (GROUP, 305)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 494), Map 8 (PARTITION-LEVEL SORT, 494)
- Reducer 3 <- Reducer 11 (PARTITION-LEVEL SORT, 570), Reducer 13 (PARTITION-LEVEL SORT, 570), Reducer 2 (PARTITION-LEVEL SORT, 570)
- Reducer 4 <- Reducer 3 (GROUP, 447)
- Reducer 5 <- Reducer 4 (GROUP, 1)
- Reducer 6 <- Reducer 5 (SORT, 1)
+ Reducer 12 <- Map 11 (GROUP, 24)
+ Reducer 14 <- Map 13 (PARTITION-LEVEL SORT, 1009), Reducer 18 (PARTITION-LEVEL SORT, 1009)
+ Reducer 15 <- Reducer 14 (GROUP, 1009)
+ Reducer 17 <- Map 16 (PARTITION-LEVEL SORT, 1), Map 19 (PARTITION-LEVEL SORT, 1), Map 20 (PARTITION-LEVEL SORT, 1), Map 21 (PARTITION-LEVEL SORT, 1)
+ Reducer 18 <- Reducer 17 (GROUP, 1009)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 494), Map 9 (PARTITION-LEVEL SORT, 494)
+ Reducer 3 <- Reducer 12 (PARTITION-LEVEL SORT, 418), Reducer 2 (PARTITION-LEVEL SORT, 418)
+ Reducer 4 <- Reducer 15 (PARTITION-LEVEL SORT, 1009), Reducer 3 (PARTITION-LEVEL SORT, 1009)
+ Reducer 5 <- Reducer 4 (GROUP, 1009)
+ Reducer 6 <- Reducer 5 (GROUP, 1)
+ Reducer 7 <- Reducer 6 (SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -143,7 +148,7 @@ STAGE PLANS:
1 _col0 (type: int)
outputColumnNames: _col1, _col2, _col3, _col4, _col5, _col6
input vertices:
- 1 Map 7
+ 1 Map 8
Statistics: Num rows: 316788826 Data size: 42899570777 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
@@ -153,7 +158,7 @@ STAGE PLANS:
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: int), _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2))
Local Work:
Map Reduce Local Work
- Map 10
+ Map 11
Map Operator Tree:
TableScan
alias: cr1
@@ -171,25 +176,65 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 28798881 Data size: 3057234680 Basic stats: COMPLETE Column stats: NONE
- Map 12
+ Map 13
Map Operator Tree:
TableScan
alias: cs2
Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (cs_order_number is not null and cs_warehouse_sk is not null) (type: boolean)
+ Select Operator
+ expressions: cs_warehouse_sk (type: int), cs_order_number (type: int)
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: cs_order_number (type: int), cs_warehouse_sk (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col1 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
- Map 8
+ value expressions: _col0 (type: int)
+ Map 16
+ Map Operator Tree:
+ TableScan
+ alias: date_dim
+ Statistics: Num rows: 73049 Data size: 81741831 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 73049 Data size: 292196 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 73049 Data size: 292196 Basic stats: COMPLETE Column stats: COMPLETE
+ Map 19
+ Map Operator Tree:
+ TableScan
+ alias: call_center
+ Statistics: Num rows: 60 Data size: 122700 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 60 Data size: 240 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 60 Data size: 240 Basic stats: COMPLETE Column stats: COMPLETE
+ Map 20
+ Map Operator Tree:
+ TableScan
+ alias: customer_address
+ Statistics: Num rows: 40000000 Data size: 40595195284 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 40000000 Data size: 160000000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 40000000 Data size: 160000000 Basic stats: COMPLETE Column stats: COMPLETE
+ Map 21
+ Map Operator Tree:
+ TableScan
+ alias: cs1
+ Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: cs_warehouse_sk (type: int), cs_order_number (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int), _col1 (type: int)
+ Map 9
Map Operator Tree:
TableScan
alias: customer_address
@@ -206,7 +251,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 20000000 Data size: 20297597642 Basic stats: COMPLETE Column stats: NONE
- Reducer 11
+ Reducer 12
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -223,22 +268,85 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 14399440 Data size: 1528617286 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: boolean)
- Reducer 13
+ Reducer 14
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: int)
+ 1 _col1 (type: int)
+ outputColumnNames: _col0, _col2, _col3
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (_col2 <> _col0) (type: boolean)
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col2 (type: int), _col3 (type: int)
+ outputColumnNames: _col2, _col3
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col2 (type: int), _col3 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Reducer 15
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: int), KEY._col1 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2536427365110644736 Data size: 2536427365110644736 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 2536427365110644736 Data size: 2536427365110644736 Basic stats: COMPLETE Column stats: NONE
+ Reducer 17
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ Inner Join 0 to 2
+ Inner Join 0 to 3
+ keys:
+ 0
+ 1
+ 2
+ 3
+ outputColumnNames: _col3, _col4
+ Statistics: Num rows: 9223372036854775807 Data size: 9223372036854775807 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col4 (type: int), _col3 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 9223372036854775807 Data size: 9223372036854775807 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 9223372036854775807 Data size: 9223372036854775807 Basic stats: COMPLETE Column stats: NONE
+ Reducer 18
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 143994918 Data size: 19499804476 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4611686018427387903 Data size: 4611686018427387904 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), _col0 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 143994918 Data size: 19499804476 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4611686018427387903 Data size: 4611686018427387904 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 143994918 Data size: 19499804476 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4611686018427387903 Data size: 4611686018427387904 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Reducer 2
Local Work:
@@ -260,7 +368,7 @@ STAGE PLANS:
1 _col0 (type: int)
outputColumnNames: _col3, _col4, _col5, _col6
input vertices:
- 1 Map 9
+ 1 Map 10
Statistics: Num rows: 383314495 Data size: 51908482889 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col4 (type: int)
@@ -273,45 +381,58 @@ STAGE PLANS:
Join Operator
condition map:
Left Outer Join 0 to 1
- Inner Join 0 to 2
keys:
0 _col4 (type: int)
1 _col0 (type: int)
- 2 _col1 (type: int)
- outputColumnNames: _col3, _col4, _col5, _col6, _col14, _col15
- residual filter predicates: {(_col3 <> _col15)}
- Statistics: Num rows: 843291907 Data size: 114198664830 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col3, _col4, _col5, _col6, _col14
+ Statistics: Num rows: 421645953 Data size: 57099332415 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col3 (type: int), _col4 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col3 (type: int), _col4 (type: int)
+ Statistics: Num rows: 421645953 Data size: 57099332415 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2)), _col14 (type: boolean)
+ Reducer 4
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col3 (type: int), _col4 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
+ outputColumnNames: _col4, _col5, _col6, _col14
+ Statistics: Num rows: 2790070162094850048 Data size: 2790070162094850048 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col4 (type: int), _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2)), _col14 (type: boolean)
outputColumnNames: _col4, _col5, _col6, _col16
- Statistics: Num rows: 843291907 Data size: 114198664830 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2790070162094850048 Data size: 2790070162094850048 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col16 is null (type: boolean)
- Statistics: Num rows: 421645953 Data size: 57099332347 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col4 (type: int), _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2))
outputColumnNames: _col4, _col5, _col6
- Statistics: Num rows: 421645953 Data size: 57099332347 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: sum(_col5), sum(_col6)
keys: _col4 (type: int)
mode: hash
outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 421645953 Data size: 57099332347 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 421645953 Data size: 57099332347 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: decimal(17,2)), _col3 (type: decimal(17,2))
- Reducer 4
+ Reducer 5
Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0), sum(VALUE._col1)
keys: KEY._col0 (type: int)
mode: partial2
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 421645953 Data size: 57099332347 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(_col0), sum(_col1), sum(_col2)
mode: partial2
@@ -321,7 +442,7 @@ STAGE PLANS:
sort order:
Statistics: Num rows: 1 Data size: 344 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col1 (type: decimal(17,2)), _col2 (type: decimal(17,2))
- Reducer 5
+ Reducer 6
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), sum(VALUE._col1), sum(VALUE._col2)
@@ -338,7 +459,7 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 344 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col1 (type: decimal(17,2)), _col2 (type: decimal(17,2))
- Reducer 6
+ Reducer 7
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: bigint), VALUE._col0 (type: decimal(17,2)), VALUE._col1 (type: decimal(17,2))
http://git-wip-us.apache.org/repos/asf/hive/blob/9375cf3c/ql/src/test/results/clientpositive/perf/spark/query94.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query94.q.out b/ql/src/test/results/clientpositive/perf/spark/query94.q.out
index 4ee2ed8..43b8c77 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query94.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query94.q.out
@@ -1,3 +1,4 @@
+Warning: Shuffle Join JOIN[33][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_4]] in Work 'Reducer 17' is a cross product
PREHOOK: query: explain
select
count(distinct ws_order_number) as `order count`
@@ -65,7 +66,7 @@ STAGE PLANS:
Spark
#### A masked pattern was here ####
Vertices:
- Map 9
+ Map 10
Map Operator Tree:
TableScan
alias: web_site
@@ -88,7 +89,7 @@ STAGE PLANS:
Spark
#### A masked pattern was here ####
Vertices:
- Map 7
+ Map 8
Map Operator Tree:
TableScan
alias: date_dim
@@ -110,13 +111,17 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 11 <- Map 10 (GROUP, 11)
- Reducer 13 <- Map 12 (GROUP, 153)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 327), Map 8 (PARTITION-LEVEL SORT, 327)
- Reducer 3 <- Reducer 11 (PARTITION-LEVEL SORT, 286), Reducer 13 (PARTITION-LEVEL SORT, 286), Reducer 2 (PARTITION-LEVEL SORT, 286)
- Reducer 4 <- Reducer 3 (GROUP, 224)
- Reducer 5 <- Reducer 4 (GROUP, 1)
- Reducer 6 <- Reducer 5 (SORT, 1)
+ Reducer 12 <- Map 11 (GROUP, 11)
+ Reducer 14 <- Map 13 (PARTITION-LEVEL SORT, 1009), Reducer 18 (PARTITION-LEVEL SORT, 1009)
+ Reducer 15 <- Reducer 14 (GROUP, 1009)
+ Reducer 17 <- Map 16 (PARTITION-LEVEL SORT, 1), Map 19 (PARTITION-LEVEL SORT, 1), Map 20 (PARTITION-LEVEL SORT, 1), Map 21 (PARTITION-LEVEL SORT, 1)
+ Reducer 18 <- Reducer 17 (GROUP, 1009)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 327), Map 9 (PARTITION-LEVEL SORT, 327)
+ Reducer 3 <- Reducer 12 (PARTITION-LEVEL SORT, 209), Reducer 2 (PARTITION-LEVEL SORT, 209)
+ Reducer 4 <- Reducer 15 (PARTITION-LEVEL SORT, 1009), Reducer 3 (PARTITION-LEVEL SORT, 1009)
+ Reducer 5 <- Reducer 4 (GROUP, 1009)
+ Reducer 6 <- Reducer 5 (GROUP, 1)
+ Reducer 7 <- Reducer 6 (SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -139,7 +144,7 @@ STAGE PLANS:
1 _col0 (type: int)
outputColumnNames: _col1, _col2, _col3, _col4, _col5, _col6
input vertices:
- 1 Map 7
+ 1 Map 8
Statistics: Num rows: 158402938 Data size: 21538218500 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
@@ -149,7 +154,7 @@ STAGE PLANS:
value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: int), _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2))
Local Work:
Map Reduce Local Work
- Map 10
+ Map 11
Map Operator Tree:
TableScan
alias: wr1
@@ -167,25 +172,65 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 14398467 Data size: 1325194184 Basic stats: COMPLETE Column stats: NONE
- Map 12
+ Map 13
Map Operator Tree:
TableScan
alias: ws2
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (ws_order_number is not null and ws_warehouse_sk is not null) (type: boolean)
+ Select Operator
+ expressions: ws_warehouse_sk (type: int), ws_order_number (type: int)
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: ws_order_number (type: int), ws_warehouse_sk (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
+ Reduce Output Operator
+ key expressions: _col1 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
- Map 8
+ value expressions: _col0 (type: int)
+ Map 16
+ Map Operator Tree:
+ TableScan
+ alias: date_dim
+ Statistics: Num rows: 73049 Data size: 81741831 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 73049 Data size: 292196 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 73049 Data size: 292196 Basic stats: COMPLETE Column stats: COMPLETE
+ Map 19
+ Map Operator Tree:
+ TableScan
+ alias: web_site
+ Statistics: Num rows: 84 Data size: 155408 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 84 Data size: 336 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 84 Data size: 336 Basic stats: COMPLETE Column stats: COMPLETE
+ Map 20
+ Map Operator Tree:
+ TableScan
+ alias: customer_address
+ Statistics: Num rows: 40000000 Data size: 40595195284 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 40000000 Data size: 160000000 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 40000000 Data size: 160000000 Basic stats: COMPLETE Column stats: COMPLETE
+ Map 21
+ Map Operator Tree:
+ TableScan
+ alias: ws1
+ Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: ws_warehouse_sk (type: int), ws_order_number (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int), _col1 (type: int)
+ Map 9
Map Operator Tree:
TableScan
alias: customer_address
@@ -202,7 +247,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 20000000 Data size: 20297597642 Basic stats: COMPLETE Column stats: NONE
- Reducer 11
+ Reducer 12
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -219,22 +264,85 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 7199233 Data size: 662597045 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: boolean)
- Reducer 13
+ Reducer 14
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: int)
+ 1 _col1 (type: int)
+ outputColumnNames: _col0, _col2, _col3
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (_col2 <> _col0) (type: boolean)
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col2 (type: int), _col3 (type: int)
+ outputColumnNames: _col2, _col3
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col2 (type: int), _col3 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 5072854730221289472 Data size: 5072854730221289472 Basic stats: COMPLETE Column stats: NONE
+ Reducer 15
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: int), KEY._col1 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2536427365110644736 Data size: 2536427365110644736 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 2536427365110644736 Data size: 2536427365110644736 Basic stats: COMPLETE Column stats: NONE
+ Reducer 17
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ Inner Join 0 to 2
+ Inner Join 0 to 3
+ keys:
+ 0
+ 1
+ 2
+ 3
+ outputColumnNames: _col3, _col4
+ Statistics: Num rows: 9223372036854775807 Data size: 9223372036854775807 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col4 (type: int), _col3 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 9223372036854775807 Data size: 9223372036854775807 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 9223372036854775807 Data size: 9223372036854775807 Basic stats: COMPLETE Column stats: NONE
+ Reducer 18
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 72001334 Data size: 9790099106 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4611686018427387903 Data size: 4611686018427387904 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), _col0 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 72001334 Data size: 9790099106 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4611686018427387903 Data size: 4611686018427387904 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 72001334 Data size: 9790099106 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4611686018427387903 Data size: 4611686018427387904 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Reducer 2
Local Work:
@@ -256,7 +364,7 @@ STAGE PLANS:
1 _col0 (type: int)
outputColumnNames: _col3, _col4, _col5, _col6
input vertices:
- 1 Map 9
+ 1 Map 10
Statistics: Num rows: 191667562 Data size: 26061245514 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col4 (type: int)
@@ -269,45 +377,58 @@ STAGE PLANS:
Join Operator
condition map:
Left Outer Join 0 to 1
- Inner Join 0 to 2
keys:
0 _col4 (type: int)
1 _col0 (type: int)
- 2 _col1 (type: int)
- outputColumnNames: _col3, _col4, _col5, _col6, _col14, _col15
- residual filter predicates: {(_col3 <> _col15)}
- Statistics: Num rows: 421668645 Data size: 57334741373 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col3, _col4, _col5, _col6, _col14
+ Statistics: Num rows: 210834322 Data size: 28667370686 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col3 (type: int), _col4 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col3 (type: int), _col4 (type: int)
+ Statistics: Num rows: 210834322 Data size: 28667370686 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2)), _col14 (type: boolean)
+ Reducer 4
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col3 (type: int), _col4 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
+ outputColumnNames: _col4, _col5, _col6, _col14
+ Statistics: Num rows: 2790070162094850048 Data size: 2790070162094850048 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col4 (type: int), _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2)), _col14 (type: boolean)
outputColumnNames: _col4, _col5, _col6, _col16
- Statistics: Num rows: 421668645 Data size: 57334741373 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2790070162094850048 Data size: 2790070162094850048 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: _col16 is null (type: boolean)
- Statistics: Num rows: 210834322 Data size: 28667370618 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col4 (type: int), _col5 (type: decimal(7,2)), _col6 (type: decimal(7,2))
outputColumnNames: _col4, _col5, _col6
- Statistics: Num rows: 210834322 Data size: 28667370618 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: sum(_col5), sum(_col6)
keys: _col4 (type: int)
mode: hash
outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 210834322 Data size: 28667370618 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 210834322 Data size: 28667370618 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: decimal(17,2)), _col3 (type: decimal(17,2))
- Reducer 4
+ Reducer 5
Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0), sum(VALUE._col1)
keys: KEY._col0 (type: int)
mode: partial2
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 210834322 Data size: 28667370618 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1395035081047425024 Data size: 1395035081047425024 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(_col0), sum(_col1), sum(_col2)
mode: partial2
@@ -317,7 +438,7 @@ STAGE PLANS:
sort order:
Statistics: Num rows: 1 Data size: 344 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col1 (type: decimal(17,2)), _col2 (type: decimal(17,2))
- Reducer 5
+ Reducer 6
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), sum(VALUE._col1), sum(VALUE._col2)
@@ -334,7 +455,7 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 344 Basic stats: COMPLETE Column stats: NONE
TopN Hash Memory Usage: 0.1
value expressions: _col1 (type: decimal(17,2)), _col2 (type: decimal(17,2))
- Reducer 6
+ Reducer 7
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: bigint), VALUE._col0 (type: decimal(17,2)), VALUE._col1 (type: decimal(17,2))
http://git-wip-us.apache.org/repos/asf/hive/blob/9375cf3c/ql/src/test/results/clientpositive/perf/tez/query14.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/tez/query14.q.out b/ql/src/test/results/clientpositive/perf/tez/query14.q.out
index dfdd53d..b2a45f1 100644
--- a/ql/src/test/results/clientpositive/perf/tez/query14.q.out
+++ b/ql/src/test/results/clientpositive/perf/tez/query14.q.out
@@ -1,6 +1,6 @@
-Warning: Shuffle Join MERGEJOIN[892][tables = [$hdt$_2, $hdt$_3, $hdt$_1]] in Stage 'Reducer 16' is a cross product
-Warning: Shuffle Join MERGEJOIN[891][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 12' is a cross product
Warning: Shuffle Join MERGEJOIN[890][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 5' is a cross product
+Warning: Shuffle Join MERGEJOIN[891][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 12' is a cross product
+Warning: Shuffle Join MERGEJOIN[892][tables = [$hdt$_2, $hdt$_3, $hdt$_1]] in Stage 'Reducer 16' is a cross product
PREHOOK: query: explain
with cross_items as
(select i_item_sk ss_item_sk
http://git-wip-us.apache.org/repos/asf/hive/blob/9375cf3c/ql/src/test/results/clientpositive/perf/tez/query16.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/tez/query16.q.out b/ql/src/test/results/clientpositive/perf/tez/query16.q.out
index b2b990e..a7b710d 100644
--- a/ql/src/test/results/clientpositive/perf/tez/query16.q.out
+++ b/ql/src/test/results/clientpositive/perf/tez/query16.q.out
@@ -1,3 +1,4 @@
+Warning: Shuffle Join MERGEJOIN[113][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_4]] in Stage 'Reducer 18' is a cross product
PREHOOK: query: explain
select
count(distinct cs_order_number) as `order count`
@@ -61,132 +62,180 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 13 <- Map 12 (SIMPLE_EDGE)
-Reducer 15 <- Map 14 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
-Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Map 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 13 (ONE_TO_ONE_EDGE), Reducer 15 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 7 <- Reducer 6 (CUSTOM_SIMPLE_EDGE)
-Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
+Reducer 14 <- Map 13 (SIMPLE_EDGE)
+Reducer 16 <- Map 15 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
+Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
+Reducer 18 <- Map 15 (CUSTOM_SIMPLE_EDGE), Map 20 (CUSTOM_SIMPLE_EDGE), Map 21 (CUSTOM_SIMPLE_EDGE), Map 22 (CUSTOM_SIMPLE_EDGE)
+Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 10 (SIMPLE_EDGE)
+Reducer 3 <- Map 11 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Map 12 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 14 (ONE_TO_ONE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 17 (ONE_TO_ONE_EDGE), Reducer 5 (SIMPLE_EDGE)
+Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+Reducer 8 <- Reducer 7 (CUSTOM_SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
- Reducer 8
- File Output Operator [FS_50]
- Limit [LIM_48] (rows=1 width=344)
+ Reducer 9
+ File Output Operator [FS_74]
+ Limit [LIM_72] (rows=1 width=344)
Number of rows:100
- Select Operator [SEL_47] (rows=1 width=344)
+ Select Operator [SEL_71] (rows=1 width=344)
Output:["_col0","_col1","_col2"]
- <-Reducer 7 [SIMPLE_EDGE]
- SHUFFLE [RS_46]
- Select Operator [SEL_45] (rows=1 width=344)
+ <-Reducer 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_70]
+ Select Operator [SEL_69] (rows=1 width=344)
Output:["_col1","_col2","_col3"]
- Group By Operator [GBY_78] (rows=1 width=344)
+ Group By Operator [GBY_112] (rows=1 width=344)
Output:["_col0","_col1","_col2"],aggregations:["count(VALUE._col0)","sum(VALUE._col1)","sum(VALUE._col2)"]
- <-Reducer 6 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_77]
- Group By Operator [GBY_76] (rows=1 width=344)
+ <-Reducer 7 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_111]
+ Group By Operator [GBY_110] (rows=1 width=344)
Output:["_col0","_col1","_col2"],aggregations:["count(_col0)","sum(_col1)","sum(_col2)"]
- Group By Operator [GBY_75] (rows=421645953 width=135)
+ Group By Operator [GBY_109] (rows=1395035081047425024 width=1)
Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0
- <-Reducer 5 [SIMPLE_EDGE]
- SHUFFLE [RS_74]
+ <-Reducer 6 [SIMPLE_EDGE]
+ SHUFFLE [RS_108]
PartitionCols:_col0
- Group By Operator [GBY_73] (rows=421645953 width=135)
+ Group By Operator [GBY_107] (rows=1395035081047425024 width=1)
Output:["_col0","_col2","_col3"],aggregations:["sum(_col5)","sum(_col6)"],keys:_col4
- Select Operator [SEL_41] (rows=421645953 width=135)
+ Select Operator [SEL_65] (rows=1395035081047425024 width=1)
Output:["_col4","_col5","_col6"]
- Filter Operator [FIL_40] (rows=421645953 width=135)
+ Filter Operator [FIL_64] (rows=1395035081047425024 width=1)
predicate:_col16 is null
- Select Operator [SEL_39] (rows=843291907 width=135)
+ Select Operator [SEL_63] (rows=2790070162094850048 width=1)
Output:["_col4","_col5","_col6","_col16"]
- Merge Join Operator [MERGEJOIN_82] (rows=843291907 width=135)
- Conds:RS_35._col4=RS_36._col0(Left Outer),RS_35._col4=RS_37._col1(Inner),Output:["_col3","_col4","_col5","_col6","_col14","_col15"],residual filter predicates:{(_col3 <> _col15)}
- <-Reducer 13 [ONE_TO_ONE_EDGE]
- FORWARD [RS_36]
- PartitionCols:_col0
- Select Operator [SEL_18] (rows=14399440 width=106)
- Output:["_col0","_col1"]
- Group By Operator [GBY_17] (rows=14399440 width=106)
- Output:["_col0"],keys:KEY._col0
- <-Map 12 [SIMPLE_EDGE]
- SHUFFLE [RS_16]
- PartitionCols:_col0
- Group By Operator [GBY_15] (rows=28798881 width=106)
- Output:["_col0"],keys:cr_order_number
- Filter Operator [FIL_71] (rows=28798881 width=106)
- predicate:cr_order_number is not null
- TableScan [TS_12] (rows=28798881 width=106)
- default@catalog_returns,cr1,Tbl:COMPLETE,Col:NONE,Output:["cr_order_number"]
- <-Reducer 15 [SIMPLE_EDGE]
- SHUFFLE [RS_37]
- PartitionCols:_col1
- Select Operator [SEL_25] (rows=143994918 width=135)
- Output:["_col0","_col1"]
- Group By Operator [GBY_24] (rows=143994918 width=135)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Map 14 [SIMPLE_EDGE]
- SHUFFLE [RS_23]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_22] (rows=287989836 width=135)
- Output:["_col0","_col1"],keys:cs_order_number, cs_warehouse_sk
- Filter Operator [FIL_72] (rows=287989836 width=135)
- predicate:(cs_order_number is not null and cs_warehouse_sk is not null)
- TableScan [TS_19] (rows=287989836 width=135)
- default@catalog_sales,cs2,Tbl:COMPLETE,Col:NONE,Output:["cs_warehouse_sk","cs_order_number"]
- <-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_35]
- PartitionCols:_col4
- Merge Join Operator [MERGEJOIN_81] (rows=383314495 width=135)
- Conds:RS_32._col2=RS_33._col0(Inner),Output:["_col3","_col4","_col5","_col6"]
- <-Map 11 [SIMPLE_EDGE]
- SHUFFLE [RS_33]
+ Merge Join Operator [MERGEJOIN_119] (rows=2790070162094850048 width=1)
+ Conds:RS_60._col3, _col4=RS_61._col0, _col1(Inner),Output:["_col4","_col5","_col6","_col14"]
+ <-Reducer 17 [ONE_TO_ONE_EDGE]
+ FORWARD [RS_61]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_46] (rows=2536427365110644736 width=1)
+ Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
+ <-Reducer 16 [SIMPLE_EDGE]
+ SHUFFLE [RS_45]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_44] (rows=5072854730221289472 width=1)
+ Output:["_col0","_col1"],keys:_col2, _col3
+ Select Operator [SEL_43] (rows=5072854730221289472 width=1)
+ Output:["_col2","_col3"]
+ Filter Operator [FIL_42] (rows=5072854730221289472 width=1)
+ predicate:(_col2 <> _col0)
+ Merge Join Operator [MERGEJOIN_117] (rows=5072854730221289472 width=1)
+ Conds:RS_39._col1=RS_40._col1(Inner),Output:["_col0","_col2","_col3"]
+ <-Map 15 [SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_39]
+ PartitionCols:_col1
+ Select Operator [SEL_20] (rows=287989836 width=135)
+ Output:["_col0","_col1"]
+ TableScan [TS_19] (rows=287989836 width=135)
+ default@catalog_sales,cs2,Tbl:COMPLETE,Col:NONE,Output:["cs_warehouse_sk","cs_order_number"]
+ <-Reducer 19 [SIMPLE_EDGE]
+ SHUFFLE [RS_40]
+ PartitionCols:_col1
+ Select Operator [SEL_38] (rows=4611686018427387903 width=1)
+ Output:["_col0","_col1"]
+ Group By Operator [GBY_37] (rows=4611686018427387903 width=1)
+ Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
+ <-Reducer 18 [SIMPLE_EDGE]
+ SHUFFLE [RS_36]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_35] (rows=9223372036854775807 width=1)
+ Output:["_col0","_col1"],keys:_col4, _col3
+ Merge Join Operator [MERGEJOIN_113] (rows=9223372036854775807 width=1)
+ Conds:(Inner),(Inner),(Inner),Output:["_col3","_col4"]
+ <-Map 15 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_32]
+ Please refer to the previous Select Operator [SEL_20]
+ <-Map 20 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_29]
+ Select Operator [SEL_22] (rows=73049 width=4)
+ TableScan [TS_21] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE
+ <-Map 21 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_30]
+ Select Operator [SEL_24] (rows=60 width=4)
+ TableScan [TS_23] (rows=60 width=2045)
+ default@call_center,call_center,Tbl:COMPLETE,Col:COMPLETE
+ <-Map 22 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_31]
+ Select Operator [SEL_26] (rows=40000000 width=4)
+ TableScan [TS_25] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:COMPLETE
+ <-Reducer 5 [SIMPLE_EDGE]
+ SHUFFLE [RS_60]
+ PartitionCols:_col3, _col4
+ Merge Join Operator [MERGEJOIN_118] (rows=421645953 width=135)
+ Conds:RS_57._col4=RS_58._col0(Left Outer),Output:["_col3","_col4","_col5","_col6","_col14"]
+ <-Reducer 14 [ONE_TO_ONE_EDGE]
+ FORWARD [RS_58]
PartitionCols:_col0
- Select Operator [SEL_11] (rows=30 width=2045)
- Output:["_col0"]
- Filter Operator [FIL_70] (rows=30 width=2045)
- predicate:((cc_county) IN ('Ziebach County', 'Levy County', 'Huron County', 'Franklin Parish', 'Daviess County') and cc_call_center_sk is not null)
- TableScan [TS_9] (rows=60 width=2045)
- default@call_center,call_center,Tbl:COMPLETE,Col:NONE,Output:["cc_call_center_sk","cc_county"]
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_32]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_80] (rows=348467716 width=135)
- Conds:RS_29._col1=RS_30._col0(Inner),Output:["_col2","_col3","_col4","_col5","_col6"]
- <-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_30]
+ Select Operator [SEL_18] (rows=14399440 width=106)
+ Output:["_col0","_col1"]
+ Group By Operator [GBY_17] (rows=14399440 width=106)
+ Output:["_col0"],keys:KEY._col0
+ <-Map 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_16]
+ PartitionCols:_col0
+ Group By Operator [GBY_15] (rows=28798881 width=106)
+ Output:["_col0"],keys:cr_order_number
+ Filter Operator [FIL_104] (rows=28798881 width=106)
+ predicate:cr_order_number is not null
+ TableScan [TS_12] (rows=28798881 width=106)
+ default@catalog_returns,cr1,Tbl:COMPLETE,Col:NONE,Output:["cr_order_number"]
+ <-Reducer 4 [SIMPLE_EDGE]
+ SHUFFLE [RS_57]
+ PartitionCols:_col4
+ Merge Join Operator [MERGEJOIN_116] (rows=383314495 width=135)
+ Conds:RS_54._col2=RS_55._col0(Inner),Output:["_col3","_col4","_col5","_col6"]
+ <-Map 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_55]
PartitionCols:_col0
- Select Operator [SEL_8] (rows=20000000 width=1014)
+ Select Operator [SEL_11] (rows=30 width=2045)
Output:["_col0"]
- Filter Operator [FIL_69] (rows=20000000 width=1014)
- predicate:((ca_state = 'NY') and ca_address_sk is not null)
- TableScan [TS_6] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_29]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_79] (rows=316788826 width=135)
- Conds:RS_26._col0=RS_27._col0(Inner),Output:["_col1","_col2","_col3","_col4","_col5","_col6"]
- <-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_26]
+ Filter Operator [FIL_103] (rows=30 width=2045)
+ predicate:((cc_county) IN ('Ziebach County', 'Levy County', 'Huron County', 'Franklin Parish', 'Daviess County') and cc_call_center_sk is not null)
+ TableScan [TS_9] (rows=60 width=2045)
+ default@call_center,call_center,Tbl:COMPLETE,Col:NONE,Output:["cc_call_center_sk","cc_county"]
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_54]
+ PartitionCols:_col2
+ Merge Join Operator [MERGEJOIN_115] (rows=348467716 width=135)
+ Conds:RS_51._col1=RS_52._col0(Inner),Output:["_col2","_col3","_col4","_col5","_col6"]
+ <-Map 11 [SIMPLE_EDGE]
+ SHUFFLE [RS_52]
PartitionCols:_col0
- Select Operator [SEL_2] (rows=287989836 width=135)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
- Filter Operator [FIL_67] (rows=287989836 width=135)
- predicate:(cs_call_center_sk is not null and cs_ship_addr_sk is not null and cs_ship_date_sk is not null)
- TableScan [TS_0] (rows=287989836 width=135)
- default@catalog_sales,cs1,Tbl:COMPLETE,Col:NONE,Output:["cs_ship_date_sk","cs_ship_addr_sk","cs_call_center_sk","cs_warehouse_sk","cs_order_number","cs_ext_ship_cost","cs_net_profit"]
- <-Map 9 [SIMPLE_EDGE]
- SHUFFLE [RS_27]
- PartitionCols:_col0
- Select Operator [SEL_5] (rows=8116 width=1119)
+ Select Operator [SEL_8] (rows=20000000 width=1014)
Output:["_col0"]
- Filter Operator [FIL_68] (rows=8116 width=1119)
- predicate:(CAST( d_date AS TIMESTAMP) BETWEEN 2001-04-01 00:00:00.0 AND 2001-05-31 01:00:00.0 and d_date_sk is not null)
- TableScan [TS_3] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_date"]
+ Filter Operator [FIL_102] (rows=20000000 width=1014)
+ predicate:((ca_state = 'NY') and ca_address_sk is not null)
+ TableScan [TS_6] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ <-Reducer 2 [SIMPLE_EDGE]
+ SHUFFLE [RS_51]
+ PartitionCols:_col1
+ Merge Join Operator [MERGEJOIN_114] (rows=316788826 width=135)
+ Conds:RS_48._col0=RS_49._col0(Inner),Output:["_col1","_col2","_col3","_col4","_col5","_col6"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_48]
+ PartitionCols:_col0
+ Select Operator [SEL_2] (rows=287989836 width=135)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
+ Filter Operator [FIL_100] (rows=287989836 width=135)
+ predicate:(cs_call_center_sk is not null and cs_ship_addr_sk is not null and cs_ship_date_sk is not null)
+ TableScan [TS_0] (rows=287989836 width=135)
+ default@catalog_sales,cs1,Tbl:COMPLETE,Col:NONE,Output:["cs_ship_date_sk","cs_ship_addr_sk","cs_call_center_sk","cs_warehouse_sk","cs_order_number","cs_ext_ship_cost","cs_net_profit"]
+ <-Map 10 [SIMPLE_EDGE]
+ SHUFFLE [RS_49]
+ PartitionCols:_col0
+ Select Operator [SEL_5] (rows=8116 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_101] (rows=8116 width=1119)
+ predicate:(CAST( d_date AS TIMESTAMP) BETWEEN 2001-04-01 00:00:00.0 AND 2001-05-31 01:00:00.0 and d_date_sk is not null)
+ TableScan [TS_3] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_date"]
http://git-wip-us.apache.org/repos/asf/hive/blob/9375cf3c/ql/src/test/results/clientpositive/perf/tez/query23.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/tez/query23.q.out b/ql/src/test/results/clientpositive/perf/tez/query23.q.out
index 3507278..7112de6 100644
--- a/ql/src/test/results/clientpositive/perf/tez/query23.q.out
+++ b/ql/src/test/results/clientpositive/perf/tez/query23.q.out
@@ -1,5 +1,5 @@
-Warning: Shuffle Join MERGEJOIN[369][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 30' is a cross product
Warning: Shuffle Join MERGEJOIN[367][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 25' is a cross product
+Warning: Shuffle Join MERGEJOIN[369][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 30' is a cross product
PREHOOK: query: explain
with frequent_ss_items as
(select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt