You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by ha...@apache.org on 2017/02/15 01:26:03 UTC
[01/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Repository: hive
Updated Branches:
refs/heads/master b14ef6d6f -> bddf5a7a9
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out b/ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out
index bdfdc34..94210fd 100644
--- a/ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out
+++ b/ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out
@@ -39,95 +39,10 @@ PREHOOK: type: QUERY
POSTHOOK: query: explain select * from src11 where src11.key1 in (select key from src where src11.value1 = value and key > '9')
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-2 depends on stages: Stage-3
- Stage-1 depends on stages: Stage-2
+ Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: src11
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Select Operator
- expressions: value1 (type: string)
- outputColumnNames: value1
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Group By Operator
- keys: value1 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: src
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-1
Map Reduce
Map Operator Tree:
@@ -144,11 +59,25 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ alias: src
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -157,10 +86,10 @@ STAGE PLANS:
0 _col0 (type: string), _col1 (type: string)
1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 200 Data size: 2132 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 200 Data size: 2132 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -177,95 +106,10 @@ PREHOOK: type: QUERY
POSTHOOK: query: explain select * from src a where a.key in (select key from src where a.value = value and key > '9')
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-2 depends on stages: Stage-3
- Stage-1 depends on stages: Stage-2
+ Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: src
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-1
Map Reduce
Map Operator Tree:
@@ -282,11 +126,25 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ alias: src
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -329,46 +187,11 @@ where b.key in
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-4 is a root stage
- Stage-2 depends on stages: Stage-4
- Stage-3 depends on stages: Stage-2
- Stage-1 depends on stages: Stage-3
+ Stage-2 is a root stage
+ Stage-1 depends on stages: Stage-2
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-2
Map Reduce
Map Operator Tree:
@@ -376,65 +199,29 @@ STAGE PLANS:
alias: src
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -462,7 +249,7 @@ STAGE PLANS:
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -506,17 +293,11 @@ where b.p_name not in
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-3 is a root stage
- Stage-4 depends on stages: Stage-3, Stage-6
- Stage-5 depends on stages: Stage-4
- Stage-1 depends on stages: Stage-5
- Stage-2 depends on stages: Stage-1, Stage-10
- Stage-6 is a root stage
- Stage-7 is a root stage
- Stage-8 depends on stages: Stage-7, Stage-11
- Stage-9 depends on stages: Stage-8
- Stage-10 depends on stages: Stage-9, Stage-12
- Stage-11 is a root stage
- Stage-12 is a root stage
+ Stage-4 depends on stages: Stage-3
+ Stage-1 depends on stages: Stage-4
+ Stage-2 depends on stages: Stage-1, Stage-6
+ Stage-5 is a root stage
+ Stage-6 depends on stages: Stage-5
Stage-0 depends on stages: Stage-2
STAGE PLANS:
@@ -526,18 +307,21 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- TopN Hash Memory Usage: 0.1
- value expressions: p_name (type: string)
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ TopN Hash Memory Usage: 0.1
+ value expressions: p_name (type: string)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col1 (type: string), KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col1, _col2, _col5
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
PTF Operator
Function definitions:
Input definition
@@ -558,20 +342,26 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ outputColumnNames: _col1, _col0
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count(), count(_col0)
+ keys: _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-4
Map Reduce
@@ -581,49 +371,7 @@ STAGE PLANS:
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-5
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reduce Operator Tree:
Group By Operator
@@ -631,7 +379,7 @@ STAGE PLANS:
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -660,7 +408,7 @@ STAGE PLANS:
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reduce Operator Tree:
Join Operator
@@ -690,10 +438,10 @@ STAGE PLANS:
value expressions: _col2 (type: int), _col4 (type: bigint), _col5 (type: bigint)
TableScan
Reduce Output Operator
- key expressions: _col3 (type: string), _col1 (type: string)
+ key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
- Map-reduce partition columns: _col3 (type: string), _col1 (type: string)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: boolean)
Reduce Operator Tree:
Join Operator
@@ -701,11 +449,11 @@ STAGE PLANS:
Left Outer Join0 to 1
keys:
0 _col0 (type: string), _col1 (type: string)
- 1 _col3 (type: string), _col1 (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1, _col2, _col4, _col5, _col8
Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col8 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END (type: boolean)
+ predicate: (not CASE WHEN ((_col4 = 0)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (true) ELSE (false) END) (type: boolean)
Statistics: Num rows: 15 Data size: 1903 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col0 (type: string), _col2 (type: int)
@@ -719,57 +467,27 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-6
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_mfgr (type: string)
- outputColumnNames: p_mfgr
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-7
+ Stage: Stage-5
Map Reduce
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- TopN Hash Memory Usage: 0.1
- value expressions: p_name (type: string)
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ TopN Hash Memory Usage: 0.1
+ value expressions: p_name (type: string)
Reduce Operator Tree:
Select Operator
expressions: VALUE._col1 (type: string), KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col1, _col2, _col5
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
PTF Operator
Function definitions:
Input definition
@@ -790,59 +508,27 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
+ expressions: _col1 (type: string), _col2 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-8
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-9
+ Stage: Stage-6
Map Reduce
Map Operator Tree:
TableScan
@@ -850,121 +536,26 @@ STAGE PLANS:
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-10
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string), _col2 (type: boolean)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-11
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_mfgr (type: string)
- outputColumnNames: p_mfgr
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-12
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: p_name (type: string)
- outputColumnNames: p_name
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out b/ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out
index 5ea9a65..a2f59d5 100644
--- a/ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out
+++ b/ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out
@@ -205,18 +205,9 @@ PLAN VECTORIZATION:
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-16 depends on stages: Stage-1
- Stage-12 depends on stages: Stage-16
- Stage-11 depends on stages: Stage-12, Stage-13 , consists of Stage-15, Stage-3
- Stage-15 has a backup stage: Stage-3
- Stage-10 depends on stages: Stage-15
- Stage-3
- Stage-6 is a root stage
- Stage-18 depends on stages: Stage-6
- Stage-8 depends on stages: Stage-18
- Stage-17 depends on stages: Stage-8
- Stage-13 depends on stages: Stage-17
- Stage-0 depends on stages: Stage-10, Stage-3
+ Stage-8 depends on stages: Stage-1
+ Stage-6 depends on stages: Stage-8
+ Stage-0 depends on stages: Stage-6
STAGE PLANS:
Stage: Stage-1
@@ -267,12 +258,15 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-16
+ Stage: Stage-8
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_1:li
Fetch Operator
limit: -1
+ $hdt$_2:lineitem
+ Fetch Operator
+ limit: -1
Alias -> Map Local Operator Tree:
$hdt$_1:li
TableScan
@@ -289,264 +283,32 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col1 (type: int)
-
- Stage: Stage-12
- Map Reduce
- Map Operator Tree:
- TableScan
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col1 (type: int)
- outputColumnNames: _col0, _col1, _col3, _col4
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.SequenceFileInputFormat
- Local Work:
- Map Reduce Local Work
-
- Stage: Stage-11
- Conditional Operator
-
- Stage: Stage-15
- Map Reduce Local Work
- Alias -> Map Local Tables:
- $INTNAME1
- Fetch Operator
- limit: -1
- Alias -> Map Local Operator Tree:
- $INTNAME1
- TableScan
- HashTable Sink Operator
- keys:
- 0 _col1 (type: int), _col4 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
-
- Stage: Stage-10
- Map Reduce
- Map Operator Tree:
- TableScan
- Map Join Operator
- condition map:
- Left Semi Join 0 to 1
- keys:
- 0 _col1 (type: int), _col4 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.SequenceFileInputFormat
- Local Work:
- Map Reduce Local Work
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col1 (type: int), _col4 (type: int)
- sort order: ++
- Map-reduce partition columns: _col1 (type: int), _col4 (type: int)
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int), _col3 (type: int)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- keys:
- 0 _col1 (type: int), _col4 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
-
- Stage: Stage-6
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: l_partkey (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
- Reduce Vectorization:
- enabled: false
- enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true
- enableConditionsNotMet: hive.execution.engine mr IN [tez, spark] IS false
- Reduce Operator Tree:
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-18
- Map Reduce Local Work
- Alias -> Map Local Tables:
- $hdt$_2:$hdt$_3:$hdt$_4:li
- Fetch Operator
- limit: -1
- Alias -> Map Local Operator Tree:
- $hdt$_2:$hdt$_3:$hdt$_4:li
- TableScan
- alias: li
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int), l_linenumber (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- HashTable Sink Operator
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
-
- Stage: Stage-8
- Map Reduce
- Map Operator Tree:
- TableScan
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col2
- Statistics: Num rows: 110 Data size: 13198 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: _col2 (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 110 Data size: 13198 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 110 Data size: 13198 Basic stats: COMPLETE Column stats: NONE
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.SequenceFileInputFormat
- Local Work:
- Map Reduce Local Work
- Reduce Vectorization:
- enabled: false
- enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true
- enableConditionsNotMet: hive.execution.engine mr IN [tez, spark] IS false
- Reduce Operator Tree:
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 55 Data size: 6599 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-17
- Map Reduce Local Work
- Alias -> Map Local Tables:
- $hdt$_2:$hdt$_2:lineitem
- Fetch Operator
- limit: -1
- Alias -> Map Local Operator Tree:
- $hdt$_2:$hdt$_2:lineitem
+ $hdt$_2:lineitem
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (l_shipmode = 'AIR') (type: boolean)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((l_shipmode = 'AIR') and (l_linenumber = l_linenumber)) (type: boolean)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: l_orderkey (type: int), l_linenumber (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- HashTable Sink Operator
- keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ Group By Vectorization:
+ vectorOutput: false
+ native: false
+ projectedOutputColumns: null
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ HashTable Sink Operator
+ keys:
+ 0 _col1 (type: int), _col4 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
- Stage: Stage-13
+ Stage: Stage-6
Map Reduce
Map Operator Tree:
TableScan
@@ -554,29 +316,29 @@ STAGE PLANS:
condition map:
Inner Join 0 to 1
keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
+ 0 _col0 (type: int)
+ 1 _col1 (type: int)
+ outputColumnNames: _col0, _col1, _col3, _col4
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ keys:
+ 0 _col1 (type: int), _col4 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
+ outputColumnNames: _col0, _col3
+ Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col3 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
+ Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Map Vectorization:
enabled: false
enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
[06/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/perf/query69.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/query69.q.out b/ql/src/test/results/clientpositive/perf/query69.q.out
index 71ddd71..38c7fa7 100644
--- a/ql/src/test/results/clientpositive/perf/query69.q.out
+++ b/ql/src/test/results/clientpositive/perf/query69.q.out
@@ -1,12 +1,9 @@
-Warning: Shuffle Join MERGEJOIN[261][tables = [$hdt$_2, $hdt$_3, $hdt$_4]] in Stage 'Reducer 17' is a cross product
-Warning: Shuffle Join MERGEJOIN[262][tables = [$hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 22' is a cross product
-Warning: Shuffle Join MERGEJOIN[273][tables = [$hdt$_3, $hdt$_4, $hdt$_2, $hdt$_1]] in Stage 'Reducer 13' is a cross product
-Warning: Shuffle Join MERGEJOIN[263][tables = [$hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 33' is a cross product
-Warning: Shuffle Join MERGEJOIN[264][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 38' is a cross product
-Warning: Shuffle Join MERGEJOIN[275][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_0]] in Stage 'Reducer 29' is a cross product
-Warning: Shuffle Join MERGEJOIN[265][tables = [$hdt$_2, $hdt$_3, $hdt$_4]] in Stage 'Reducer 49' is a cross product
-Warning: Shuffle Join MERGEJOIN[266][tables = [$hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 54' is a cross product
-Warning: Shuffle Join MERGEJOIN[278][tables = [$hdt$_3, $hdt$_4, $hdt$_2, $hdt$_1]] in Stage 'Reducer 45' is a cross product
+Warning: Shuffle Join MERGEJOIN[177][tables = [$hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 16' is a cross product
+Warning: Shuffle Join MERGEJOIN[185][tables = [$hdt$_2, $hdt$_3, $hdt$_1]] in Stage 'Reducer 12' is a cross product
+Warning: Shuffle Join MERGEJOIN[178][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 26' is a cross product
+Warning: Shuffle Join MERGEJOIN[186][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Stage 'Reducer 22' is a cross product
+Warning: Shuffle Join MERGEJOIN[179][tables = [$hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 36' is a cross product
+Warning: Shuffle Join MERGEJOIN[188][tables = [$hdt$_2, $hdt$_3, $hdt$_1]] in Stage 'Reducer 32' is a cross product
PREHOOK: query: explain select
cd_gender,
cd_marital_status,
@@ -100,34 +97,25 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 11 <- Map 10 (SIMPLE_EDGE), Map 15 (SIMPLE_EDGE)
-Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
-Reducer 13 <- Reducer 12 (CUSTOM_SIMPLE_EDGE), Reducer 23 (CUSTOM_SIMPLE_EDGE)
-Reducer 14 <- Reducer 13 (SIMPLE_EDGE)
-Reducer 17 <- Map 16 (CUSTOM_SIMPLE_EDGE), Map 19 (CUSTOM_SIMPLE_EDGE), Map 20 (CUSTOM_SIMPLE_EDGE)
-Reducer 18 <- Reducer 17 (SIMPLE_EDGE)
+Reducer 11 <- Map 10 (SIMPLE_EDGE), Map 14 (SIMPLE_EDGE)
+Reducer 12 <- Reducer 11 (CUSTOM_SIMPLE_EDGE), Reducer 17 (CUSTOM_SIMPLE_EDGE)
+Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
+Reducer 16 <- Map 15 (CUSTOM_SIMPLE_EDGE), Map 18 (CUSTOM_SIMPLE_EDGE), Map 19 (CUSTOM_SIMPLE_EDGE)
+Reducer 17 <- Reducer 16 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
-Reducer 22 <- Map 21 (CUSTOM_SIMPLE_EDGE), Map 24 (CUSTOM_SIMPLE_EDGE), Map 25 (CUSTOM_SIMPLE_EDGE)
+Reducer 21 <- Map 20 (SIMPLE_EDGE), Map 24 (SIMPLE_EDGE)
+Reducer 22 <- Reducer 21 (CUSTOM_SIMPLE_EDGE), Reducer 27 (CUSTOM_SIMPLE_EDGE)
Reducer 23 <- Reducer 22 (SIMPLE_EDGE)
-Reducer 27 <- Map 26 (SIMPLE_EDGE), Map 31 (SIMPLE_EDGE)
-Reducer 28 <- Reducer 27 (SIMPLE_EDGE), Reducer 34 (SIMPLE_EDGE)
-Reducer 29 <- Reducer 28 (CUSTOM_SIMPLE_EDGE), Reducer 39 (CUSTOM_SIMPLE_EDGE)
+Reducer 26 <- Map 25 (CUSTOM_SIMPLE_EDGE), Map 28 (CUSTOM_SIMPLE_EDGE), Map 29 (CUSTOM_SIMPLE_EDGE)
+Reducer 27 <- Reducer 26 (SIMPLE_EDGE)
Reducer 3 <- Map 9 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 30 <- Reducer 29 (SIMPLE_EDGE)
-Reducer 33 <- Map 32 (CUSTOM_SIMPLE_EDGE), Map 35 (CUSTOM_SIMPLE_EDGE), Map 36 (CUSTOM_SIMPLE_EDGE)
-Reducer 34 <- Reducer 33 (SIMPLE_EDGE)
-Reducer 38 <- Map 37 (CUSTOM_SIMPLE_EDGE), Map 40 (CUSTOM_SIMPLE_EDGE), Map 41 (CUSTOM_SIMPLE_EDGE)
-Reducer 39 <- Reducer 38 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 14 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE), Reducer 30 (SIMPLE_EDGE)
-Reducer 43 <- Map 42 (SIMPLE_EDGE), Map 47 (SIMPLE_EDGE)
-Reducer 44 <- Reducer 43 (SIMPLE_EDGE), Reducer 50 (SIMPLE_EDGE)
-Reducer 45 <- Reducer 44 (CUSTOM_SIMPLE_EDGE), Reducer 55 (CUSTOM_SIMPLE_EDGE)
-Reducer 46 <- Reducer 45 (SIMPLE_EDGE)
-Reducer 49 <- Map 48 (CUSTOM_SIMPLE_EDGE), Map 51 (CUSTOM_SIMPLE_EDGE), Map 52 (CUSTOM_SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE), Reducer 46 (SIMPLE_EDGE)
-Reducer 50 <- Reducer 49 (SIMPLE_EDGE)
-Reducer 54 <- Map 53 (CUSTOM_SIMPLE_EDGE), Map 56 (CUSTOM_SIMPLE_EDGE), Map 57 (CUSTOM_SIMPLE_EDGE)
-Reducer 55 <- Reducer 54 (SIMPLE_EDGE)
+Reducer 31 <- Map 30 (SIMPLE_EDGE), Map 34 (SIMPLE_EDGE)
+Reducer 32 <- Reducer 31 (CUSTOM_SIMPLE_EDGE), Reducer 37 (CUSTOM_SIMPLE_EDGE)
+Reducer 33 <- Reducer 32 (SIMPLE_EDGE)
+Reducer 36 <- Map 35 (CUSTOM_SIMPLE_EDGE), Map 38 (CUSTOM_SIMPLE_EDGE), Map 39 (CUSTOM_SIMPLE_EDGE)
+Reducer 37 <- Reducer 36 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 13 (SIMPLE_EDGE), Reducer 23 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 33 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
@@ -136,360 +124,259 @@ Stage-0
limit:100
Stage-1
Reducer 7
- File Output Operator [FS_184]
- Limit [LIM_183] (rows=100 width=1)
+ File Output Operator [FS_130]
+ Limit [LIM_129] (rows=100 width=1)
Number of rows:100
- Select Operator [SEL_182] (rows=1268213682555322368 width=1)
+ Select Operator [SEL_128] (rows=1268213682555322368 width=1)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7"]
<-Reducer 6 [SIMPLE_EDGE]
- SHUFFLE [RS_181]
- Select Operator [SEL_180] (rows=1268213682555322368 width=1)
+ SHUFFLE [RS_127]
+ Select Operator [SEL_126] (rows=1268213682555322368 width=1)
Output:["_col0","_col1","_col2","_col3","_col4","_col6"]
- Group By Operator [GBY_179] (rows=1268213682555322368 width=1)
+ Group By Operator [GBY_125] (rows=1268213682555322368 width=1)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["count(VALUE._col0)"],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4
<-Reducer 5 [SIMPLE_EDGE]
- SHUFFLE [RS_178]
+ SHUFFLE [RS_124]
PartitionCols:_col0, _col1, _col2, _col3, _col4
- Group By Operator [GBY_177] (rows=2536427365110644736 width=1)
+ Group By Operator [GBY_123] (rows=2536427365110644736 width=1)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["count()"],keys:_col6, _col7, _col8, _col9, _col10
- Select Operator [SEL_176] (rows=2536427365110644736 width=1)
+ Select Operator [SEL_122] (rows=2536427365110644736 width=1)
Output:["_col6","_col7","_col8","_col9","_col10"]
- Filter Operator [FIL_175] (rows=2536427365110644736 width=1)
+ Filter Operator [FIL_121] (rows=2536427365110644736 width=1)
predicate:_col15 is null
- Merge Join Operator [MERGEJOIN_279] (rows=5072854730221289472 width=1)
- Conds:RS_172._col0=RS_173._col0(Left Outer),Output:["_col6","_col7","_col8","_col9","_col10","_col15"]
+ Merge Join Operator [MERGEJOIN_189] (rows=5072854730221289472 width=1)
+ Conds:RS_118._col0=RS_119._col0(Left Outer),Output:["_col6","_col7","_col8","_col9","_col10","_col15"]
+ <-Reducer 33 [SIMPLE_EDGE]
+ SHUFFLE [RS_119]
+ PartitionCols:_col0
+ Select Operator [SEL_117] (rows=4611686018427387903 width=1)
+ Output:["_col0","_col1"]
+ Group By Operator [GBY_116] (rows=4611686018427387903 width=1)
+ Output:["_col0"],keys:KEY._col0
+ <-Reducer 32 [SIMPLE_EDGE]
+ SHUFFLE [RS_115]
+ PartitionCols:_col0
+ Group By Operator [GBY_114] (rows=9223372036854775807 width=1)
+ Output:["_col0"],keys:_col5
+ Merge Join Operator [MERGEJOIN_188] (rows=9223372036854775807 width=1)
+ Conds:(Inner),Output:["_col5"]
+ <-Reducer 31 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_110]
+ Merge Join Operator [MERGEJOIN_184] (rows=158394413 width=135)
+ Conds:RS_107._col0=RS_108._col0(Inner)
+ <-Map 30 [SIMPLE_EDGE]
+ SHUFFLE [RS_107]
+ PartitionCols:_col0
+ Select Operator [SEL_88] (rows=143994918 width=135)
+ Output:["_col0"]
+ Filter Operator [FIL_174] (rows=143994918 width=135)
+ predicate:((cs_ship_customer_sk = cs_ship_customer_sk) and cs_sold_date_sk is not null)
+ TableScan [TS_86] (rows=287989836 width=135)
+ default@catalog_sales,catalog_sales,Tbl:COMPLETE,Col:NONE,Output:["cs_sold_date_sk","cs_ship_customer_sk"]
+ <-Map 34 [SIMPLE_EDGE]
+ SHUFFLE [RS_108]
+ PartitionCols:_col0
+ Select Operator [SEL_91] (rows=4058 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_175] (rows=4058 width=1119)
+ predicate:((d_year = 1999) and d_moy BETWEEN 1 AND 3 and d_date_sk is not null)
+ TableScan [TS_89] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
+ <-Reducer 37 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_111]
+ Group By Operator [GBY_105] (rows=4611686018427387903 width=1)
+ Output:["_col0"],keys:KEY._col0
+ <-Reducer 36 [SIMPLE_EDGE]
+ SHUFFLE [RS_104]
+ PartitionCols:_col0
+ Group By Operator [GBY_103] (rows=9223372036854775807 width=1)
+ Output:["_col0"],keys:_col2
+ Merge Join Operator [MERGEJOIN_179] (rows=9223372036854775807 width=1)
+ Conds:(Inner),(Inner),Output:["_col2"]
+ <-Map 35 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_98]
+ Select Operator [SEL_93] (rows=40000000 width=4)
+ TableScan [TS_92] (rows=40000000 width=1014)
+ default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
+ <-Map 38 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_99]
+ Select Operator [SEL_95] (rows=1861800 width=4)
+ TableScan [TS_94] (rows=1861800 width=385)
+ default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
+ <-Map 39 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_100]
+ Select Operator [SEL_97] (rows=80000000 width=860)
+ Output:["_col0"]
+ TableScan [TS_96] (rows=80000000 width=860)
+ default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
<-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_172]
+ SHUFFLE [RS_118]
PartitionCols:_col0
- Select Operator [SEL_121] (rows=4611686018427387903 width=1)
+ Select Operator [SEL_85] (rows=4611686018427387903 width=1)
Output:["_col0","_col10","_col6","_col7","_col8","_col9"]
- Filter Operator [FIL_120] (rows=4611686018427387903 width=1)
+ Filter Operator [FIL_84] (rows=4611686018427387903 width=1)
predicate:_col13 is null
- Select Operator [SEL_119] (rows=9223372036854775807 width=1)
- Output:["_col0","_col6","_col7","_col8","_col9","_col10","_col13"]
- Merge Join Operator [MERGEJOIN_276] (rows=9223372036854775807 width=1)
- Conds:RS_115._col0=RS_116._col0(Left Outer),RS_115._col0=RS_117._col0(Inner),Output:["_col0","_col6","_col7","_col8","_col9","_col10","_col12"]
- <-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_116]
- PartitionCols:_col0
- Select Operator [SEL_58] (rows=4611686018427387903 width=1)
- Output:["_col0","_col1"]
- Group By Operator [GBY_57] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 13 [SIMPLE_EDGE]
- SHUFFLE [RS_56]
- PartitionCols:_col0
- Group By Operator [GBY_55] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col6
- Merge Join Operator [MERGEJOIN_273] (rows=9223372036854775807 width=1)
- Conds:(Inner),Output:["_col6"]
- <-Reducer 12 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_51]
- Merge Join Operator [MERGEJOIN_272] (rows=5072854730221289472 width=1)
- Conds:RS_48._col1=RS_49._col0(Inner)
- <-Reducer 11 [SIMPLE_EDGE]
- SHUFFLE [RS_48]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_269] (rows=158402938 width=135)
- Conds:RS_45._col0=RS_46._col0(Inner),Output:["_col1"]
- <-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_45]
- PartitionCols:_col0
- Select Operator [SEL_11] (rows=144002668 width=135)
- Output:["_col0","_col1"]
- Filter Operator [FIL_249] (rows=144002668 width=135)
- predicate:ws_sold_date_sk is not null
- TableScan [TS_9] (rows=144002668 width=135)
- default@web_sales,web_sales,Tbl:COMPLETE,Col:NONE,Output:["ws_sold_date_sk","ws_bill_customer_sk"]
- <-Map 15 [SIMPLE_EDGE]
- SHUFFLE [RS_46]
- PartitionCols:_col0
- Select Operator [SEL_14] (rows=4058 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_250] (rows=4058 width=1119)
- predicate:((d_year = 1999) and d_moy BETWEEN 1 AND 3 and d_date_sk is not null)
- TableScan [TS_12] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
- <-Reducer 18 [SIMPLE_EDGE]
- SHUFFLE [RS_49]
- PartitionCols:_col0
- Group By Operator [GBY_28] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 17 [SIMPLE_EDGE]
- SHUFFLE [RS_27]
- PartitionCols:_col0
- Group By Operator [GBY_26] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_261] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 16 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_21]
- Select Operator [SEL_16] (rows=40000000 width=4)
- TableScan [TS_15] (rows=40000000 width=1014)
- default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
- <-Map 19 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_22]
- Select Operator [SEL_18] (rows=1861800 width=4)
- TableScan [TS_17] (rows=1861800 width=385)
- default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
- <-Map 20 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_23]
- Select Operator [SEL_20] (rows=80000000 width=860)
- Output:["_col0"]
- TableScan [TS_19] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
- <-Reducer 23 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_52]
- Group By Operator [GBY_43] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 22 [SIMPLE_EDGE]
- SHUFFLE [RS_42]
- PartitionCols:_col0
- Group By Operator [GBY_41] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_262] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 21 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_36]
- Select Operator [SEL_31] (rows=40000000 width=4)
- TableScan [TS_30] (rows=40000000 width=1014)
- default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
- <-Map 24 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_37]
- Select Operator [SEL_33] (rows=1861800 width=4)
- TableScan [TS_32] (rows=1861800 width=385)
- default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
- <-Map 25 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_38]
- Select Operator [SEL_35] (rows=80000000 width=860)
- Output:["_col0"]
- TableScan [TS_34] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_115]
- PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_268] (rows=96800003 width=860)
- Conds:RS_112._col1=RS_113._col0(Inner),Output:["_col0","_col6","_col7","_col8","_col9","_col10"]
- <-Map 9 [SIMPLE_EDGE]
- SHUFFLE [RS_113]
- PartitionCols:_col0
- Select Operator [SEL_8] (rows=1861800 width=385)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
- Filter Operator [FIL_248] (rows=1861800 width=385)
- predicate:cd_demo_sk is not null
- TableScan [TS_6] (rows=1861800 width=385)
- default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:NONE,Output:["cd_demo_sk","cd_gender","cd_marital_status","cd_education_status","cd_purchase_estimate","cd_credit_rating"]
- <-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_112]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_267] (rows=88000001 width=860)
- Conds:RS_109._col2=RS_110._col0(Inner),Output:["_col0","_col1"]
- <-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_109]
- PartitionCols:_col2
- Select Operator [SEL_2] (rows=80000000 width=860)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_246] (rows=80000000 width=860)
- predicate:(c_current_addr_sk is not null and c_current_cdemo_sk is not null)
- TableScan [TS_0] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_current_cdemo_sk","c_current_addr_sk"]
- <-Map 8 [SIMPLE_EDGE]
- SHUFFLE [RS_110]
- PartitionCols:_col0
- Select Operator [SEL_5] (rows=20000000 width=1014)
- Output:["_col0"]
- Filter Operator [FIL_247] (rows=20000000 width=1014)
- predicate:((ca_state) IN ('CO', 'IL', 'MN') and ca_address_sk is not null)
- TableScan [TS_3] (rows=40000000 width=1014)
- default@customer_address,ca,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 30 [SIMPLE_EDGE]
- SHUFFLE [RS_117]
- PartitionCols:_col0
- Group By Operator [GBY_107] (rows=4611686018427387903 width=1)
+ Merge Join Operator [MERGEJOIN_187] (rows=9223372036854775807 width=1)
+ Conds:RS_79._col0=RS_80._col0(Inner),RS_79._col0=RS_81._col0(Left Outer),Output:["_col0","_col6","_col7","_col8","_col9","_col10","_col13"]
+ <-Reducer 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_80]
+ PartitionCols:_col0
+ Group By Operator [GBY_39] (rows=4611686018427387903 width=1)
+ Output:["_col0"],keys:KEY._col0
+ <-Reducer 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_38]
+ PartitionCols:_col0
+ Group By Operator [GBY_37] (rows=9223372036854775807 width=1)
+ Output:["_col0"],keys:_col5
+ Merge Join Operator [MERGEJOIN_185] (rows=9223372036854775807 width=1)
+ Conds:(Inner),Output:["_col5"]
+ <-Reducer 11 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_33]
+ Merge Join Operator [MERGEJOIN_182] (rows=316797605 width=88)
+ Conds:RS_30._col0=RS_31._col0(Inner)
+ <-Map 10 [SIMPLE_EDGE]
+ SHUFFLE [RS_30]
+ PartitionCols:_col0
+ Select Operator [SEL_11] (rows=287997817 width=88)
+ Output:["_col0"]
+ Filter Operator [FIL_168] (rows=287997817 width=88)
+ predicate:((ss_customer_sk = ss_customer_sk) and ss_sold_date_sk is not null)
+ TableScan [TS_9] (rows=575995635 width=88)
+ default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_customer_sk"]
+ <-Map 14 [SIMPLE_EDGE]
+ SHUFFLE [RS_31]
+ PartitionCols:_col0
+ Select Operator [SEL_14] (rows=4058 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_169] (rows=4058 width=1119)
+ predicate:((d_year = 1999) and d_moy BETWEEN 1 AND 3 and d_date_sk is not null)
+ TableScan [TS_12] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
+ <-Reducer 17 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_34]
+ Group By Operator [GBY_28] (rows=4611686018427387903 width=1)
+ Output:["_col0"],keys:KEY._col0
+ <-Reducer 16 [SIMPLE_EDGE]
+ SHUFFLE [RS_27]
+ PartitionCols:_col0
+ Group By Operator [GBY_26] (rows=9223372036854775807 width=1)
+ Output:["_col0"],keys:_col2
+ Merge Join Operator [MERGEJOIN_177] (rows=9223372036854775807 width=1)
+ Conds:(Inner),(Inner),Output:["_col2"]
+ <-Map 15 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_21]
+ Select Operator [SEL_16] (rows=40000000 width=4)
+ TableScan [TS_15] (rows=40000000 width=1014)
+ default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
+ <-Map 18 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_22]
+ Select Operator [SEL_18] (rows=1861800 width=4)
+ TableScan [TS_17] (rows=1861800 width=385)
+ default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
+ <-Map 19 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_23]
+ Select Operator [SEL_20] (rows=80000000 width=860)
+ Output:["_col0"]
+ TableScan [TS_19] (rows=80000000 width=860)
+ default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
+ <-Reducer 23 [SIMPLE_EDGE]
+ SHUFFLE [RS_81]
+ PartitionCols:_col0
+ Select Operator [SEL_72] (rows=4611686018427387903 width=1)
+ Output:["_col0","_col1"]
+ Group By Operator [GBY_71] (rows=4611686018427387903 width=1)
Output:["_col0"],keys:KEY._col0
- <-Reducer 29 [SIMPLE_EDGE]
- SHUFFLE [RS_106]
+ <-Reducer 22 [SIMPLE_EDGE]
+ SHUFFLE [RS_70]
PartitionCols:_col0
- Group By Operator [GBY_105] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col6
- Merge Join Operator [MERGEJOIN_275] (rows=9223372036854775807 width=1)
- Conds:(Inner),Output:["_col6"]
- <-Reducer 28 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_101]
- Merge Join Operator [MERGEJOIN_274] (rows=5072854730221289472 width=1)
- Conds:RS_98._col1=RS_99._col0(Inner)
- <-Reducer 27 [SIMPLE_EDGE]
- SHUFFLE [RS_98]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_270] (rows=633595212 width=88)
- Conds:RS_95._col0=RS_96._col0(Inner),Output:["_col1"]
- <-Map 26 [SIMPLE_EDGE]
- SHUFFLE [RS_95]
- PartitionCols:_col0
- Select Operator [SEL_61] (rows=575995635 width=88)
- Output:["_col0","_col1"]
- Filter Operator [FIL_253] (rows=575995635 width=88)
- predicate:ss_sold_date_sk is not null
- TableScan [TS_59] (rows=575995635 width=88)
- default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_customer_sk"]
- <-Map 31 [SIMPLE_EDGE]
- SHUFFLE [RS_96]
- PartitionCols:_col0
- Select Operator [SEL_64] (rows=4058 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_254] (rows=4058 width=1119)
- predicate:((d_year = 1999) and d_moy BETWEEN 1 AND 3 and d_date_sk is not null)
- TableScan [TS_62] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
- <-Reducer 34 [SIMPLE_EDGE]
- SHUFFLE [RS_99]
+ Group By Operator [GBY_69] (rows=9223372036854775807 width=1)
+ Output:["_col0"],keys:_col5
+ Merge Join Operator [MERGEJOIN_186] (rows=9223372036854775807 width=1)
+ Conds:(Inner),Output:["_col5"]
+ <-Reducer 21 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_65]
+ Merge Join Operator [MERGEJOIN_183] (rows=79201469 width=135)
+ Conds:RS_62._col0=RS_63._col0(Inner)
+ <-Map 20 [SIMPLE_EDGE]
+ SHUFFLE [RS_62]
+ PartitionCols:_col0
+ Select Operator [SEL_43] (rows=72001334 width=135)
+ Output:["_col0"]
+ Filter Operator [FIL_171] (rows=72001334 width=135)
+ predicate:((ws_bill_customer_sk = ws_bill_customer_sk) and ws_sold_date_sk is not null)
+ TableScan [TS_41] (rows=144002668 width=135)
+ default@web_sales,web_sales,Tbl:COMPLETE,Col:NONE,Output:["ws_sold_date_sk","ws_bill_customer_sk"]
+ <-Map 24 [SIMPLE_EDGE]
+ SHUFFLE [RS_63]
PartitionCols:_col0
- Group By Operator [GBY_78] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 33 [SIMPLE_EDGE]
- SHUFFLE [RS_77]
- PartitionCols:_col0
- Group By Operator [GBY_76] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_263] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 32 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_71]
- Select Operator [SEL_66] (rows=40000000 width=4)
- TableScan [TS_65] (rows=40000000 width=1014)
- default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
- <-Map 35 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_72]
- Select Operator [SEL_68] (rows=1861800 width=4)
- TableScan [TS_67] (rows=1861800 width=385)
- default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
- <-Map 36 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_73]
- Select Operator [SEL_70] (rows=80000000 width=860)
- Output:["_col0"]
- TableScan [TS_69] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
- <-Reducer 39 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_102]
- Group By Operator [GBY_93] (rows=4611686018427387903 width=1)
+ Select Operator [SEL_46] (rows=4058 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_172] (rows=4058 width=1119)
+ predicate:((d_year = 1999) and d_moy BETWEEN 1 AND 3 and d_date_sk is not null)
+ TableScan [TS_44] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
+ <-Reducer 27 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_66]
+ Group By Operator [GBY_60] (rows=4611686018427387903 width=1)
Output:["_col0"],keys:KEY._col0
- <-Reducer 38 [SIMPLE_EDGE]
- SHUFFLE [RS_92]
+ <-Reducer 26 [SIMPLE_EDGE]
+ SHUFFLE [RS_59]
PartitionCols:_col0
- Group By Operator [GBY_91] (rows=9223372036854775807 width=1)
+ Group By Operator [GBY_58] (rows=9223372036854775807 width=1)
Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_264] (rows=9223372036854775807 width=1)
+ Merge Join Operator [MERGEJOIN_178] (rows=9223372036854775807 width=1)
Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 37 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_86]
- Select Operator [SEL_81] (rows=40000000 width=4)
- TableScan [TS_80] (rows=40000000 width=1014)
+ <-Map 25 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_53]
+ Select Operator [SEL_48] (rows=40000000 width=4)
+ TableScan [TS_47] (rows=40000000 width=1014)
default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
- <-Map 40 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_87]
- Select Operator [SEL_83] (rows=1861800 width=4)
- TableScan [TS_82] (rows=1861800 width=385)
+ <-Map 28 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_54]
+ Select Operator [SEL_50] (rows=1861800 width=4)
+ TableScan [TS_49] (rows=1861800 width=385)
default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
- <-Map 41 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_88]
- Select Operator [SEL_85] (rows=80000000 width=860)
+ <-Map 29 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_55]
+ Select Operator [SEL_52] (rows=80000000 width=860)
Output:["_col0"]
- TableScan [TS_84] (rows=80000000 width=860)
+ TableScan [TS_51] (rows=80000000 width=860)
default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
- <-Reducer 46 [SIMPLE_EDGE]
- SHUFFLE [RS_173]
- PartitionCols:_col0
- Select Operator [SEL_171] (rows=4611686018427387903 width=1)
- Output:["_col0","_col1"]
- Group By Operator [GBY_170] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 45 [SIMPLE_EDGE]
- SHUFFLE [RS_169]
- PartitionCols:_col0
- Group By Operator [GBY_168] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col6
- Merge Join Operator [MERGEJOIN_278] (rows=9223372036854775807 width=1)
- Conds:(Inner),Output:["_col6"]
- <-Reducer 44 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_164]
- Merge Join Operator [MERGEJOIN_277] (rows=5072854730221289472 width=1)
- Conds:RS_161._col1=RS_162._col0(Inner)
- <-Reducer 43 [SIMPLE_EDGE]
- SHUFFLE [RS_161]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_271] (rows=316788826 width=135)
- Conds:RS_158._col0=RS_159._col0(Inner),Output:["_col1"]
- <-Map 42 [SIMPLE_EDGE]
- SHUFFLE [RS_158]
- PartitionCols:_col0
- Select Operator [SEL_124] (rows=287989836 width=135)
- Output:["_col0","_col1"]
- Filter Operator [FIL_257] (rows=287989836 width=135)
- predicate:cs_sold_date_sk is not null
- TableScan [TS_122] (rows=287989836 width=135)
- default@catalog_sales,catalog_sales,Tbl:COMPLETE,Col:NONE,Output:["cs_sold_date_sk","cs_ship_customer_sk"]
- <-Map 47 [SIMPLE_EDGE]
- SHUFFLE [RS_159]
- PartitionCols:_col0
- Select Operator [SEL_127] (rows=4058 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_258] (rows=4058 width=1119)
- predicate:((d_year = 1999) and d_moy BETWEEN 1 AND 3 and d_date_sk is not null)
- TableScan [TS_125] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
- <-Reducer 50 [SIMPLE_EDGE]
- SHUFFLE [RS_162]
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_79]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_181] (rows=96800003 width=860)
+ Conds:RS_76._col1=RS_77._col0(Inner),Output:["_col0","_col6","_col7","_col8","_col9","_col10"]
+ <-Map 9 [SIMPLE_EDGE]
+ SHUFFLE [RS_77]
+ PartitionCols:_col0
+ Select Operator [SEL_8] (rows=1861800 width=385)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Filter Operator [FIL_167] (rows=1861800 width=385)
+ predicate:cd_demo_sk is not null
+ TableScan [TS_6] (rows=1861800 width=385)
+ default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:NONE,Output:["cd_demo_sk","cd_gender","cd_marital_status","cd_education_status","cd_purchase_estimate","cd_credit_rating"]
+ <-Reducer 2 [SIMPLE_EDGE]
+ SHUFFLE [RS_76]
+ PartitionCols:_col1
+ Merge Join Operator [MERGEJOIN_180] (rows=88000001 width=860)
+ Conds:RS_73._col2=RS_74._col0(Inner),Output:["_col0","_col1"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_73]
+ PartitionCols:_col2
+ Select Operator [SEL_2] (rows=80000000 width=860)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_165] (rows=80000000 width=860)
+ predicate:(c_current_addr_sk is not null and c_current_cdemo_sk is not null)
+ TableScan [TS_0] (rows=80000000 width=860)
+ default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_current_cdemo_sk","c_current_addr_sk"]
+ <-Map 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_74]
PartitionCols:_col0
- Group By Operator [GBY_141] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 49 [SIMPLE_EDGE]
- SHUFFLE [RS_140]
- PartitionCols:_col0
- Group By Operator [GBY_139] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_265] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 48 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_134]
- Select Operator [SEL_129] (rows=40000000 width=4)
- TableScan [TS_128] (rows=40000000 width=1014)
- default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
- <-Map 51 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_135]
- Select Operator [SEL_131] (rows=1861800 width=4)
- TableScan [TS_130] (rows=1861800 width=385)
- default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
- <-Map 52 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_136]
- Select Operator [SEL_133] (rows=80000000 width=860)
- Output:["_col0"]
- TableScan [TS_132] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
- <-Reducer 55 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_165]
- Group By Operator [GBY_156] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 54 [SIMPLE_EDGE]
- SHUFFLE [RS_155]
- PartitionCols:_col0
- Group By Operator [GBY_154] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_266] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 53 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_149]
- Select Operator [SEL_144] (rows=40000000 width=4)
- TableScan [TS_143] (rows=40000000 width=1014)
- default@customer_address,ca,Tbl:COMPLETE,Col:COMPLETE
- <-Map 56 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_150]
- Select Operator [SEL_146] (rows=1861800 width=4)
- TableScan [TS_145] (rows=1861800 width=385)
- default@customer_demographics,customer_demographics,Tbl:COMPLETE,Col:COMPLETE
- <-Map 57 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_151]
- Select Operator [SEL_148] (rows=80000000 width=860)
- Output:["_col0"]
- TableScan [TS_147] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk"]
+ Select Operator [SEL_5] (rows=20000000 width=1014)
+ Output:["_col0"]
+ Filter Operator [FIL_166] (rows=20000000 width=1014)
+ predicate:((ca_state) IN ('CO', 'IL', 'MN') and ca_address_sk is not null)
+ TableScan [TS_3] (rows=40000000 width=1014)
+ default@customer_address,ca,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
[12/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/subquery_multi.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_multi.q.out b/ql/src/test/results/clientpositive/llap/subquery_multi.q.out
index 6e33d07..d2016b3 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_multi.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_multi.q.out
@@ -1681,11 +1681,13 @@ POSTHOOK: Input: default@tempty
17927 almond aquamarine yellow dodger mint Manufacturer#4 Brand#41 ECONOMY BRUSHED COPPER 7 SM PKG 1844.92 ites. eve
33357 almond azure aquamarine papaya violet Manufacturer#4 Brand#41 STANDARD ANODIZED TIN 12 WRAP CASE 1290.35 reful
78486 almond azure blanched chiffon midnight Manufacturer#5 Brand#52 LARGE BRUSHED BRASS 23 MED BAG 1464.48 hely blith
-Warning: Shuffle Join MERGEJOIN[187][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 18' is a cross product
-Warning: Shuffle Join MERGEJOIN[190][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 23' is a cross product
-PREHOOK: query: explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
+PREHOOK: query: explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type)
+ AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type
+ AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
PREHOOK: type: QUERY
-POSTHOOK: query: explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
+POSTHOOK: query: explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type)
+ AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type
+ AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
@@ -1696,32 +1698,16 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 24 (SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
- Reducer 15 <- Map 14 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
- Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
- Reducer 18 <- Map 17 (CUSTOM_SIMPLE_EDGE), Reducer 21 (CUSTOM_SIMPLE_EDGE)
- Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 21 <- Map 20 (SIMPLE_EDGE)
- Reducer 23 <- Map 22 (CUSTOM_SIMPLE_EDGE), Reducer 26 (CUSTOM_SIMPLE_EDGE)
- Reducer 24 <- Reducer 23 (SIMPLE_EDGE)
- Reducer 26 <- Map 25 (SIMPLE_EDGE)
- Reducer 28 <- Map 27 (SIMPLE_EDGE), Reducer 34 (SIMPLE_EDGE)
- Reducer 29 <- Reducer 28 (SIMPLE_EDGE), Reducer 38 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 13 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
- Reducer 30 <- Reducer 29 (SIMPLE_EDGE)
- Reducer 31 <- Reducer 30 (SIMPLE_EDGE), Reducer 40 (SIMPLE_EDGE)
- Reducer 33 <- Map 32 (SIMPLE_EDGE), Reducer 36 (SIMPLE_EDGE)
- Reducer 34 <- Reducer 33 (SIMPLE_EDGE)
- Reducer 36 <- Map 35 (SIMPLE_EDGE)
- Reducer 38 <- Map 37 (SIMPLE_EDGE)
- Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Reducer 31 (SIMPLE_EDGE)
- Reducer 40 <- Map 39 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 11 <- Map 10 (SIMPLE_EDGE)
+ Reducer 13 <- Map 12 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
+ Reducer 14 <- Reducer 13 (SIMPLE_EDGE)
+ Reducer 16 <- Map 15 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 14 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
+ Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1744,192 +1730,69 @@ STAGE PLANS:
Map 10
Map Operator Tree:
TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string), p_container (type: string)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 14
- Map Operator Tree:
- TableScan
alias: pp
Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 17
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 20
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: p_type
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 22
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 25
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: p_type
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_type (type: string), p_brand (type: string)
+ outputColumnNames: p_type, p_brand
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_type (type: string), p_brand (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 27
+ Map 12
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string), p_container (type: string)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 3744 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_brand (type: string), p_type (type: string), p_container (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 13 Data size: 3744 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
+ Statistics: Num rows: 13 Data size: 3744 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 32
+ Map 15
Map Operator Tree:
TableScan
alias: pp
Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 35
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 37
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 39
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2392 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_brand (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_type (type: string), p_brand (type: string)
+ outputColumnNames: p_type, p_brand
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_type (type: string), p_brand (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Map 5
@@ -1937,38 +1800,65 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_name (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2925 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_type (type: string), p_name (type: string)
+ outputColumnNames: p_type, p_name
+ Statistics: Num rows: 13 Data size: 2925 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_type (type: string), p_name (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1350 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1350 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 8
+ Map 7
Map Operator Tree:
TableScan
alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 3744 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_brand (type: string), p_type (type: string), p_container (type: string)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 13 Data size: 3744 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
+ Statistics: Num rows: 13 Data size: 3744 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: string)
Execution mode: llap
LLAP IO: no inputs
Reducer 11
Execution mode: llap
Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 13
+ Execution mode: llap
+ Reduce Operator Tree:
Merge Join Operator
condition map:
Inner Join 0 to 1
@@ -1977,130 +1867,58 @@ STAGE PLANS:
1 _col1 (type: string), _col0 (type: string)
outputColumnNames: _col1, _col2
Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col2, _col5
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col5 (type: string), _col2 (type: string)
- outputColumnNames: _col5, _col2
+ expressions: _col2 (type: string), _col1 (type: string)
+ outputColumnNames: _col2, _col1
Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: count(), count(_col2)
- keys: _col5 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 13
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0), count(VALUE._col1)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 15
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: string)
+ keys: _col2 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 16
+ Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 14
Execution mode: llap
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: string), _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 1 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: boolean)
+ Reducer 16
+ Execution mode: llap
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: string)
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 18
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0
- Statistics: Num rows: 338 Data size: 35152 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 19
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -2118,104 +1936,6 @@ STAGE PLANS:
Map-reduce partition columns: _col4 (type: string)
Statistics: Num rows: 6 Data size: 1485 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
- Reducer 21
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 23
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0
- Statistics: Num rows: 338 Data size: 35152 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 24
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 26
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 28
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string), _col0 (type: string)
- 1 _col1 (type: string), _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
- Reducer 29
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col2, _col5
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col5 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 3
Execution mode: llap
Reduce Operator Tree:
@@ -2233,109 +1953,6 @@ STAGE PLANS:
Map-reduce partition columns: _col3 (type: string), _col4 (type: string)
Statistics: Num rows: 6 Data size: 1633 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string), _col12 (type: bigint), _col13 (type: bigint)
- Reducer 30
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: boolean)
- Reducer 31
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col3 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col3 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: boolean)
- Reducer 33
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 12 Data size: 2352 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 12 Data size: 2352 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 34
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: string), _col0 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 36
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 38
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 4
Execution mode: llap
Reduce Operator Tree:
@@ -2344,11 +1961,11 @@ STAGE PLANS:
Left Outer Join0 to 1
keys:
0 _col3 (type: string), _col4 (type: string)
- 1 _col3 (type: string), _col1 (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col12, _col13, _col16
Statistics: Num rows: 6 Data size: 1796 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: CASE WHEN ((_col12 = 0)) THEN (true) WHEN (_col12 is null) THEN (true) WHEN (_col16 is not null) THEN (false) WHEN (_col3 is null) THEN (null) WHEN ((_col13 < _col12)) THEN (false) ELSE (true) END (type: boolean)
+ predicate: (not CASE WHEN ((_col12 = 0)) THEN (false) WHEN (_col12 is null) THEN (false) WHEN (_col16 is not null) THEN (true) WHEN (_col3 is null) THEN (null) WHEN ((_col13 < _col12)) THEN (true) ELSE (false) END) (type: boolean)
Statistics: Num rows: 3 Data size: 898 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
@@ -2361,47 +1978,9 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 40
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 6
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 12 Data size: 2700 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 12 Data size: 2700 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 1350 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 6 Data size: 1350 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
@@ -2416,19 +1995,44 @@ STAGE PLANS:
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
Statistics: Num rows: 6 Data size: 1350 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 9
+ Reducer 8
Execution mode: llap
Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: string), _col0 (type: string)
+ 1 _col1 (type: string), _col0 (type: string)
+ outputColumnNames: _col1, _col2
+ Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(_col2)
+ keys: _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Reducer 9
+ Execution mode: llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0), count(VALUE._col1)
+ keys: KEY._col0 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
Stage: Stage-0
Fetch Operator
@@ -2436,14 +2040,16 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[187][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 18' is a cross product
-Warning: Shuffle Join MERGEJOIN[190][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 23' is a cross product
-PREHOOK: query: select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
+PREHOOK: query: select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type)
+ AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type
+ AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
PREHOOK: type: QUERY
PREHOOK: Input: default@part
PREHOOK: Input: default@part_null
#### A masked pattern was here ####
-POSTHOOK: query: select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
+POSTHOOK: query: select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type)
+ AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type
+ AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
POSTHOOK: Input: default@part_null
@@ -2487,13 +2093,9 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
- Reducer 12 <- Map 11 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+ Reducer 3 <- Map 6 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
Reducer 5 <- Map 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE), Reducer 10 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE), Reducer 12 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2516,26 +2118,6 @@ STAGE PLANS:
value expressions: _col0 (type: int), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 11
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: p_name is not null (type: boolean)
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
Map 4
Map Operator Tree:
TableScan
@@ -2561,64 +2143,25 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
- Map Operator Tree:
- TableScan
- alias: part_null
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: p_name is not null (type: boolean)
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: p_name (type: string), p_type (type: string)
+ expressions: p_brand (type: string), p_type (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Reducer 10
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 8 Data size: 1685 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 1685 Basic stats: COMPLETE Column stats: NONE
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -2646,10 +2189,10 @@ STAGE PLANS:
0 _col3 (type: string), _col4 (type: string)
1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 30 Data size: 6165 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 1903 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 30 Data size: 6165 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 15 Data size: 1903 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -2667,52 +2210,6 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 28 Data size: 5605 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 5605 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 5605 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 28 Data size: 5605 Basic stats: COMPLETE Column stats: NONE
- Reducer 9
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -2756,8 +2253,6 @@ POSTHOOK: Input: default@part_null
192697 almond antique blue firebrick mint Manufacturer#5 Brand#52 MEDIUM BURNISHED TIN 31 LG DRUM 1789.69 ickly ir
15103 almond aquamarine dodger light gainsboro Manufacturer#5 Brand#53 ECONOMY BURNISHED STEEL 46 LG PACK 1018.1 packages hinder carefu
155733 almond antique sky peru orange Manufacturer#5 Brand#53 SMALL PLATED BRASS 2 WRAP DRUM 1788.73 furiously. bra
-Warning: Shuffle Join MERGEJOIN[187][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 18' is a cross product
-Warning: Shuffle Join MERGEJOIN[190][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 23' is a cross product
PREHOOK: query: explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type AND part.p_container=part_null.p_container) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type AND part.p_container=part_null.p_container) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type))
@@ -2771,32 +2266,16 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 24 (SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
- Reducer 15 <- Map 14 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
- Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
- Reducer 18 <- Map 17 (CUSTOM_SIMPLE_EDGE), Reducer 21 (CUSTOM_SIMPLE_EDGE)
- Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 21 <- Map 20 (SIMPLE_EDGE)
- Reducer 23 <- Map 22 (CUSTOM_SIMPLE_EDGE), Reducer 26 (CUSTOM_SIMPLE_EDGE)
- Reducer 24 <- Reducer 23 (SIMPLE_EDGE)
- Reducer 26 <- Map 25 (SIMPLE_EDGE)
- Reducer 28 <- Map 27 (SIMPLE_EDGE), Reducer 34 (SIMPLE_EDGE)
- Reducer 29 <- Reducer 28 (SIMPLE_EDGE), Reducer 38 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 13 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
- Reducer 30 <- Reducer 29 (SIMPLE_EDGE)
- Reducer 31 <- Reducer 30 (SIMPLE_EDGE), Reducer 40 (SIMPLE_EDGE)
- Reducer 33 <- Map 32 (SIMPLE_EDGE), Reducer 36 (SIMPLE_EDGE)
- Reducer 34 <- Reducer 33 (SIMPLE_EDGE)
- Reducer 36 <- Map 35 (SIMPLE_EDGE)
- Reducer 38 <- Map 37 (SIMPLE_EDGE)
- Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Reducer 31 (SIMPLE_EDGE)
- Reducer 40 <- Map 39 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 11 <- Map 10 (SIMPLE_EDGE)
+ Reducer 13 <- Map 12 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
+ Reducer 14 <- Reducer 13 (SIMPLE_EDGE)
+ Reducer 16 <- Map 15 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 14 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
+ Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2819,441 +2298,133 @@ STAGE PLANS:
Map 10
Map Operator Tree:
TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string), p_container (type: string)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 14
- Map Operator Tree:
- TableScan
- alias: pp
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 17
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 20
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string), p_container (type: string)
- outputColumnNames: p_type, p_container
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string), p_container (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 22
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 25
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string), p_container (type: string)
- outputColumnNames: p_type, p_container
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string), p_container (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 27
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string), p_container (type: string)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 32
- Map Operator Tree:
- TableScan
alias: pp
Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 35
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 37
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 39
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2392 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_brand (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 8242 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_name (type: string), p_type (type: string), p_container (type: string)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 8242 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col2 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col2 (type: string)
- Statistics: Num rows: 26 Data size: 8242 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string), p_container (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string), _col0 (type: string)
- 1 _col1 (type: string), _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: string)
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col2, _col5
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col5 (type: string), _col2 (type: string)
- outputColumnNames: _col5, _col2
- Statistics: Num rows: 1 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(), count(_col2)
- keys: _col5 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 13
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0), count(VALUE._col1)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 120 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 15
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Select Operator
+ expressions: p_type (type: string), p_brand (type: string)
+ outputColumnNames: p_type, p_brand
Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 16
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: string), _col0 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 18
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0
- Statistics: Num rows: 338 Data size: 35152 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 19
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 28 Data size: 2912 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 2
+ Group By Operator
+ keys: p_type (type: string), p_brand (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col4 (type: string), _col1 (type: string), _col6 (type: string)
- 1 _col1 (type: string), _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 5 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col4 (type: string)
- sort order: +
- Map-reduce partition columns: _col4 (type: string)
- Statistics: Num rows: 5 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
- Reducer 21
+ LLAP IO: no inputs
+ Map 12
+ Map Operator Tree:
+ TableScan
+ alias: part
+ Statistics: Num rows: 26 Data size: 7488 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predi
<TRUNCATED>
[05/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/perf/query81.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/query81.q.out b/ql/src/test/results/clientpositive/perf/query81.q.out
index 8ca0068..25bd68e 100644
--- a/ql/src/test/results/clientpositive/perf/query81.q.out
+++ b/ql/src/test/results/clientpositive/perf/query81.q.out
@@ -1,4 +1,3 @@
-Warning: Shuffle Join MERGEJOIN[154][tables = [$hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 22' is a cross product
PREHOOK: query: explain with customer_total_return as
(select cr_returning_customer_sk as ctr_customer_sk
,ca_state as ctr_state,
@@ -60,249 +59,163 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
-Reducer 14 <- Map 13 (SIMPLE_EDGE), Map 19 (SIMPLE_EDGE)
-Reducer 15 <- Map 20 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
+Reducer 10 <- Reducer 16 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 14 <- Map 13 (SIMPLE_EDGE), Map 17 (SIMPLE_EDGE)
+Reducer 15 <- Map 18 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
-Reducer 17 <- Reducer 16 (SIMPLE_EDGE), Reducer 23 (SIMPLE_EDGE)
-Reducer 18 <- Reducer 17 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
-Reducer 22 <- Map 21 (CUSTOM_SIMPLE_EDGE), Map 30 (CUSTOM_SIMPLE_EDGE), Reducer 27 (CUSTOM_SIMPLE_EDGE)
-Reducer 23 <- Reducer 22 (SIMPLE_EDGE)
-Reducer 25 <- Map 24 (SIMPLE_EDGE), Map 28 (SIMPLE_EDGE)
-Reducer 26 <- Map 29 (SIMPLE_EDGE), Reducer 25 (SIMPLE_EDGE)
-Reducer 27 <- Reducer 26 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 18 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 8 <- Map 11 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
-Reducer 9 <- Map 12 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+Reducer 7 <- Map 11 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
+Reducer 8 <- Map 12 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:100
Stage-1
- Reducer 5
- File Output Operator [FS_102]
- Limit [LIM_101] (rows=100 width=1)
+ Reducer 4
+ File Output Operator [FS_67]
+ Limit [LIM_66] (rows=100 width=860)
Number of rows:100
- Select Operator [SEL_100] (rows=930023387364950016 width=1)
+ Select Operator [SEL_65] (rows=32266667 width=860)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15"]
- <-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_99]
- Select Operator [SEL_98] (rows=930023387364950016 width=1)
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_64]
+ Select Operator [SEL_63] (rows=32266667 width=860)
Output:["_col0","_col1","_col11","_col12","_col13","_col14","_col15","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9"]
- Filter Operator [FIL_97] (rows=930023387364950016 width=1)
+ Filter Operator [FIL_62] (rows=32266667 width=860)
predicate:(_col2 > CASE WHEN (_col22 is null) THEN (null) ELSE (_col21) END)
- Select Operator [SEL_96] (rows=2790070162094850048 width=1)
+ Select Operator [SEL_61] (rows=96800003 width=860)
Output:["_col2","_col4","_col5","_col6","_col7","_col8","_col9","_col11","_col12","_col13","_col14","_col16","_col18","_col19","_col20","_col21","_col22"]
- Merge Join Operator [MERGEJOIN_162] (rows=2790070162094850048 width=1)
- Conds:RS_93._col19=RS_94._col2(Left Outer),Output:["_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17","_col20","_col21","_col22"]
- <-Reducer 18 [SIMPLE_EDGE]
- SHUFFLE [RS_94]
- PartitionCols:_col2
- Select Operator [SEL_86] (rows=2536427365110644736 width=1)
- Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_85] (rows=2536427365110644736 width=1)
- Output:["_col0","_col1"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0
- <-Reducer 17 [SIMPLE_EDGE]
- SHUFFLE [RS_84]
- PartitionCols:_col0
- Group By Operator [GBY_83] (rows=5072854730221289472 width=1)
- Output:["_col0","_col1"],aggregations:["avg(_col2)"],keys:_col3
- Select Operator [SEL_82] (rows=5072854730221289472 width=1)
- Output:["_col3","_col2"]
- Merge Join Operator [MERGEJOIN_161] (rows=5072854730221289472 width=1)
- Conds:RS_79._col1=RS_80._col0(Inner),Output:["_col2","_col3"]
- <-Reducer 16 [SIMPLE_EDGE]
- SHUFFLE [RS_79]
- PartitionCols:_col1
- Select Operator [SEL_45] (rows=22000000 width=1014)
- Output:["_col1","_col2"]
- Group By Operator [GBY_44] (rows=22000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
- <-Reducer 15 [SIMPLE_EDGE]
- SHUFFLE [RS_43]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_42] (rows=44000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col7, _col1
- Select Operator [SEL_41] (rows=44000000 width=1014)
- Output:["_col7","_col1","_col3"]
- Merge Join Operator [MERGEJOIN_159] (rows=44000000 width=1014)
- Conds:RS_38._col2=RS_39._col0(Inner),Output:["_col1","_col3","_col7"]
- <-Map 20 [SIMPLE_EDGE]
- SHUFFLE [RS_39]
- PartitionCols:_col0
- Select Operator [SEL_34] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_148] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_32] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_38]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_158] (rows=31678769 width=106)
- Conds:RS_35._col0=RS_36._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Map 13 [SIMPLE_EDGE]
- SHUFFLE [RS_35]
- PartitionCols:_col0
- Select Operator [SEL_28] (rows=28798881 width=106)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_146] (rows=28798881 width=106)
- predicate:(cr_returned_date_sk is not null and cr_returning_addr_sk is not null)
- TableScan [TS_26] (rows=28798881 width=106)
- default@catalog_returns,catalog_returns,Tbl:COMPLETE,Col:NONE,Output:["cr_returned_date_sk","cr_returning_customer_sk","cr_returning_addr_sk","cr_return_amt_inc_tax"]
- <-Map 19 [SIMPLE_EDGE]
- SHUFFLE [RS_36]
- PartitionCols:_col0
- Select Operator [SEL_31] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_147] (rows=36524 width=1119)
- predicate:((d_year = 1998) and d_date_sk is not null)
- TableScan [TS_29] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Reducer 23 [SIMPLE_EDGE]
- SHUFFLE [RS_80]
+ Merge Join Operator [MERGEJOIN_105] (rows=96800003 width=860)
+ Conds:RS_58._col0=RS_59._col0(Inner),Output:["_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17","_col20","_col21","_col22"]
+ <-Reducer 10 [SIMPLE_EDGE]
+ SHUFFLE [RS_59]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_104] (rows=24200000 width=1014)
+ Conds:RS_51._col1=RS_52._col2(Left Outer),Output:["_col0","_col2","_col3","_col4"]
+ <-Reducer 16 [SIMPLE_EDGE]
+ SHUFFLE [RS_52]
+ PartitionCols:_col2
+ Select Operator [SEL_50] (rows=8711661 width=106)
+ Output:["_col0","_col1","_col2"]
+ Group By Operator [GBY_49] (rows=8711661 width=106)
+ Output:["_col0","_col1"],aggregations:["avg(_col2)"],keys:_col0
+ Select Operator [SEL_45] (rows=17423323 width=106)
+ Output:["_col0","_col2"]
+ Group By Operator [GBY_44] (rows=17423323 width=106)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
+ <-Reducer 15 [SIMPLE_EDGE]
+ SHUFFLE [RS_43]
PartitionCols:_col0
- Group By Operator [GBY_77] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 22 [SIMPLE_EDGE]
- SHUFFLE [RS_76]
+ Group By Operator [GBY_42] (rows=34846646 width=106)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col7, _col1
+ Select Operator [SEL_41] (rows=34846646 width=106)
+ Output:["_col7","_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_103] (rows=34846646 width=106)
+ Conds:RS_38._col2=RS_39._col0(Inner),Output:["_col1","_col3","_col7"]
+ <-Map 18 [SIMPLE_EDGE]
+ SHUFFLE [RS_39]
+ PartitionCols:_col0
+ Select Operator [SEL_34] (rows=20000000 width=1014)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_98] (rows=20000000 width=1014)
+ predicate:((ca_state = ca_state) and ca_address_sk is not null)
+ TableScan [TS_32] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ <-Reducer 14 [SIMPLE_EDGE]
+ SHUFFLE [RS_38]
+ PartitionCols:_col2
+ Merge Join Operator [MERGEJOIN_102] (rows=31678769 width=106)
+ Conds:RS_35._col0=RS_36._col0(Inner),Output:["_col1","_col2","_col3"]
+ <-Map 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_35]
+ PartitionCols:_col0
+ Select Operator [SEL_28] (rows=28798881 width=106)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_96] (rows=28798881 width=106)
+ predicate:(cr_returned_date_sk is not null and cr_returning_addr_sk is not null)
+ TableScan [TS_26] (rows=28798881 width=106)
+ default@catalog_returns,catalog_returns,Tbl:COMPLETE,Col:NONE,Output:["cr_returned_date_sk","cr_returning_customer_sk","cr_returning_addr_sk","cr_return_amt_inc_tax"]
+ <-Map 17 [SIMPLE_EDGE]
+ SHUFFLE [RS_36]
+ PartitionCols:_col0
+ Select Operator [SEL_31] (rows=36524 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_97] (rows=36524 width=1119)
+ predicate:((d_year = 1998) and d_date_sk is not null)
+ TableScan [TS_29] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
+ <-Reducer 9 [SIMPLE_EDGE]
+ SHUFFLE [RS_51]
+ PartitionCols:_col1
+ Select Operator [SEL_25] (rows=22000000 width=1014)
+ Output:["_col0","_col1","_col2"]
+ Group By Operator [GBY_24] (rows=22000000 width=1014)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
+ <-Reducer 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_23]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_22] (rows=44000000 width=1014)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col7, _col1
+ Select Operator [SEL_21] (rows=44000000 width=1014)
+ Output:["_col7","_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_101] (rows=44000000 width=1014)
+ Conds:RS_18._col2=RS_19._col0(Inner),Output:["_col1","_col3","_col7"]
+ <-Map 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_19]
PartitionCols:_col0
- Group By Operator [GBY_75] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_154] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 21 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_70]
- Select Operator [SEL_47] (rows=80000000 width=4)
- TableScan [TS_46] (rows=80000000 width=860)
- default@customer,customer,Tbl:COMPLETE,Col:COMPLETE
- <-Map 30 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_72]
- Select Operator [SEL_69] (rows=40000000 width=4)
- TableScan [TS_68] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:COMPLETE
- <-Reducer 27 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_71]
- Select Operator [SEL_67] (rows=22000000 width=1014)
- Output:["_col1"]
- Group By Operator [GBY_66] (rows=22000000 width=1014)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 26 [SIMPLE_EDGE]
- SHUFFLE [RS_65]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_64] (rows=44000000 width=1014)
- Output:["_col0","_col1"],keys:_col7, _col1
- Select Operator [SEL_63] (rows=44000000 width=1014)
- Output:["_col7","_col1"]
- Merge Join Operator [MERGEJOIN_153] (rows=44000000 width=1014)
- Conds:RS_60._col2=RS_61._col0(Inner),Output:["_col1","_col7"]
- <-Map 29 [SIMPLE_EDGE]
- SHUFFLE [RS_61]
- PartitionCols:_col0
- Select Operator [SEL_56] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_151] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_54] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 25 [SIMPLE_EDGE]
- SHUFFLE [RS_60]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_152] (rows=31678769 width=106)
- Conds:RS_57._col0=RS_58._col0(Inner),Output:["_col1","_col2"]
- <-Map 24 [SIMPLE_EDGE]
- SHUFFLE [RS_57]
- PartitionCols:_col0
- Select Operator [SEL_50] (rows=28798881 width=106)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_149] (rows=28798881 width=106)
- predicate:(cr_returned_date_sk is not null and cr_returning_addr_sk is not null)
- TableScan [TS_48] (rows=28798881 width=106)
- default@catalog_returns,catalog_returns,Tbl:COMPLETE,Col:NONE,Output:["cr_returned_date_sk","cr_returning_customer_sk","cr_returning_addr_sk"]
- <-Map 28 [SIMPLE_EDGE]
- SHUFFLE [RS_58]
- PartitionCols:_col0
- Select Operator [SEL_53] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_150] (rows=36524 width=1119)
- predicate:((d_year = 1998) and d_date_sk is not null)
- TableScan [TS_51] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_93]
- PartitionCols:_col19
- Merge Join Operator [MERGEJOIN_160] (rows=96800003 width=860)
- Conds:RS_90._col0=RS_91._col0(Inner),Output:["_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17","_col19","_col20"]
- <-Reducer 10 [SIMPLE_EDGE]
- SHUFFLE [RS_91]
+ Select Operator [SEL_14] (rows=40000000 width=1014)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_95] (rows=40000000 width=1014)
+ predicate:ca_address_sk is not null
+ TableScan [TS_12] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ <-Reducer 7 [SIMPLE_EDGE]
+ SHUFFLE [RS_18]
+ PartitionCols:_col2
+ Merge Join Operator [MERGEJOIN_100] (rows=31678769 width=106)
+ Conds:RS_15._col0=RS_16._col0(Inner),Output:["_col1","_col2","_col3"]
+ <-Map 11 [SIMPLE_EDGE]
+ SHUFFLE [RS_16]
+ PartitionCols:_col0
+ Select Operator [SEL_11] (rows=36524 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_94] (rows=36524 width=1119)
+ predicate:((d_year = 1998) and d_date_sk is not null)
+ TableScan [TS_9] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
+ <-Map 6 [SIMPLE_EDGE]
+ SHUFFLE [RS_15]
+ PartitionCols:_col0
+ Select Operator [SEL_8] (rows=28798881 width=106)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_93] (rows=28798881 width=106)
+ predicate:(cr_returned_date_sk is not null and cr_returning_addr_sk is not null and cr_returning_customer_sk is not null)
+ TableScan [TS_6] (rows=28798881 width=106)
+ default@catalog_returns,catalog_returns,Tbl:COMPLETE,Col:NONE,Output:["cr_returned_date_sk","cr_returning_customer_sk","cr_returning_addr_sk","cr_return_amt_inc_tax"]
+ <-Reducer 2 [SIMPLE_EDGE]
+ SHUFFLE [RS_58]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_99] (rows=88000001 width=860)
+ Conds:RS_55._col2=RS_56._col0(Inner),Output:["_col0","_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_55]
+ PartitionCols:_col2
+ Select Operator [SEL_2] (rows=80000000 width=860)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Filter Operator [FIL_91] (rows=80000000 width=860)
+ predicate:(c_customer_sk is not null and c_current_addr_sk is not null)
+ TableScan [TS_0] (rows=80000000 width=860)
+ default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_customer_id","c_current_addr_sk","c_salutation","c_first_name","c_last_name"]
+ <-Map 5 [SIMPLE_EDGE]
+ SHUFFLE [RS_56]
PartitionCols:_col0
- Group By Operator [GBY_24] (rows=22000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
- <-Reducer 9 [SIMPLE_EDGE]
- SHUFFLE [RS_23]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_22] (rows=44000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col1, _col7
- Select Operator [SEL_21] (rows=44000000 width=1014)
- Output:["_col1","_col7","_col3"]
- Merge Join Operator [MERGEJOIN_157] (rows=44000000 width=1014)
- Conds:RS_18._col2=RS_19._col0(Inner),Output:["_col1","_col3","_col7"]
- <-Map 12 [SIMPLE_EDGE]
- SHUFFLE [RS_19]
- PartitionCols:_col0
- Select Operator [SEL_14] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_145] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_12] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 8 [SIMPLE_EDGE]
- SHUFFLE [RS_18]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_156] (rows=31678769 width=106)
- Conds:RS_15._col0=RS_16._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Map 11 [SIMPLE_EDGE]
- SHUFFLE [RS_16]
- PartitionCols:_col0
- Select Operator [SEL_11] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_144] (rows=36524 width=1119)
- predicate:((d_year = 1998) and d_date_sk is not null)
- TableScan [TS_9] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Map 7 [SIMPLE_EDGE]
- SHUFFLE [RS_15]
- PartitionCols:_col0
- Select Operator [SEL_8] (rows=28798881 width=106)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_143] (rows=28798881 width=106)
- predicate:(cr_returned_date_sk is not null and cr_returning_addr_sk is not null and cr_returning_customer_sk is not null)
- TableScan [TS_6] (rows=28798881 width=106)
- default@catalog_returns,catalog_returns,Tbl:COMPLETE,Col:NONE,Output:["cr_returned_date_sk","cr_returning_customer_sk","cr_returning_addr_sk","cr_return_amt_inc_tax"]
- <-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_90]
- PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_155] (rows=88000001 width=860)
- Conds:RS_87._col2=RS_88._col0(Inner),Output:["_col0","_col1","_col3","_col4","_col5","_col7","_col8","_col9","_col10","_col11","_col12","_col14","_col15","_col16","_col17"]
- <-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_87]
- PartitionCols:_col2
- Select Operator [SEL_2] (rows=80000000 width=860)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
- Filter Operator [FIL_141] (rows=80000000 width=860)
- predicate:(c_customer_sk is not null and c_current_addr_sk is not null)
- TableScan [TS_0] (rows=80000000 width=860)
- default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_customer_id","c_current_addr_sk","c_salutation","c_first_name","c_last_name"]
- <-Map 6 [SIMPLE_EDGE]
- SHUFFLE [RS_88]
- PartitionCols:_col0
- Select Operator [SEL_5] (rows=20000000 width=1014)
- Output:["_col0","_col1","_col10","_col11","_col2","_col3","_col4","_col5","_col6","_col8","_col9"]
- Filter Operator [FIL_142] (rows=20000000 width=1014)
- predicate:((ca_state = 'IL') and ca_address_sk is not null)
- TableScan [TS_3] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_street_number","ca_street_name","ca_street_type","ca_suite_number","ca_city","ca_county","ca_state","ca_zip","ca_country","ca_gmt_offset","ca_location_type"]
+ Select Operator [SEL_5] (rows=20000000 width=1014)
+ Output:["_col0","_col1","_col10","_col11","_col2","_col3","_col4","_col5","_col6","_col8","_col9"]
+ Filter Operator [FIL_92] (rows=20000000 width=1014)
+ predicate:((ca_state = 'IL') and ca_address_sk is not null)
+ TableScan [TS_3] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_street_number","ca_street_name","ca_street_type","ca_suite_number","ca_city","ca_county","ca_state","ca_zip","ca_country","ca_gmt_offset","ca_location_type"]
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/semijoin5.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/semijoin5.q.out b/ql/src/test/results/clientpositive/semijoin5.q.out
index 07b7470..fd8e372 100644
--- a/ql/src/test/results/clientpositive/semijoin5.q.out
+++ b/ql/src/test/results/clientpositive/semijoin5.q.out
@@ -48,14 +48,11 @@ WHERE (t2.smallint_col_19) IN (SELECT
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1, Stage-8
+ Stage-2 depends on stages: Stage-1, Stage-7
Stage-3 depends on stages: Stage-2
Stage-4 depends on stages: Stage-3
- Stage-9 is a root stage
- Stage-10 depends on stages: Stage-9
- Stage-6 depends on stages: Stage-10
+ Stage-6 is a root stage
Stage-7 depends on stages: Stage-6
- Stage-8 depends on stages: Stage-7
Stage-0 depends on stages: Stage-4
STAGE PLANS:
@@ -236,149 +233,40 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-9
+ Stage: Stage-6
Map Reduce
Map Operator Tree:
TableScan
- alias: t1
+ alias: tt1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
- predicate: (tinyint_col_3 is not null and bigint_col_7 is not null and decimal2016_col_26 is not null) (type: boolean)
+ predicate: decimal2612_col_77 is not null (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
- expressions: tinyint_col_3 (type: tinyint), bigint_col_7 (type: bigint), timestamp_col_9 (type: timestamp), decimal2016_col_26 (type: decimal(20,16))
- outputColumnNames: _col0, _col1, _col2, _col3
+ expressions: decimal2612_col_77 (type: decimal(26,12))
+ outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: tinyint), _col3 (type: decimal(34,16)), _col1 (type: bigint)
- sort order: +++
- Map-reduce partition columns: _col0 (type: tinyint), _col3 (type: decimal(34,16)), _col1 (type: bigint)
+ key expressions: _col0 (type: decimal(26,12))
+ sort order: +
+ Map-reduce partition columns: _col0 (type: decimal(26,12))
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- value expressions: _col2 (type: timestamp)
- TableScan
- alias: t2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Filter Operator
- predicate: (tinyint_col_20 is not null and decimal2709_col_9 is not null and tinyint_col_15 is not null) (type: boolean)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Select Operator
- expressions: decimal2709_col_9 (type: decimal(27,9)), tinyint_col_15 (type: tinyint), tinyint_col_20 (type: tinyint)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Output Operator
- key expressions: _col2 (type: tinyint), _col0 (type: decimal(34,16)), UDFToLong(_col1) (type: bigint)
- sort order: +++
- Map-reduce partition columns: _col2 (type: tinyint), _col0 (type: decimal(34,16)), UDFToLong(_col1) (type: bigint)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: tinyint), _col3 (type: decimal(34,16)), _col1 (type: bigint)
- 1 _col2 (type: tinyint), _col0 (type: decimal(34,16)), UDFToLong(_col1) (type: bigint)
- outputColumnNames: _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Group By Operator
- keys: _col2 (type: timestamp)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-10
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: timestamp)
- sort order: +
- Map-reduce partition columns: _col0 (type: timestamp)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: timestamp)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-6
- Map Reduce
- Map Operator Tree:
TableScan
alias: tt2
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Filter Operator
- predicate: decimal1911_col_16 is not null (type: boolean)
+ predicate: ((timestamp_col_18 = timestamp_col_18) and decimal1911_col_16 is not null) (type: boolean)
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Select Operator
expressions: decimal1911_col_16 (type: decimal(19,11)), timestamp_col_18 (type: timestamp)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
Reduce Output Operator
- key expressions: _col1 (type: timestamp)
- sort order: +
- Map-reduce partition columns: _col1 (type: timestamp)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- value expressions: _col0 (type: decimal(19,11))
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: timestamp)
- sort order: +
- Map-reduce partition columns: _col0 (type: timestamp)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: timestamp)
- 1 _col0 (type: timestamp)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-7
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: decimal(26,12))
- sort order: +
- Map-reduce partition columns: _col0 (type: decimal(26,12))
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- value expressions: _col2 (type: timestamp)
- TableScan
- alias: tt1
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Filter Operator
- predicate: decimal2612_col_77 is not null (type: boolean)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Select Operator
- expressions: decimal2612_col_77 (type: decimal(26,12))
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Output Operator
key expressions: _col0 (type: decimal(26,12))
sort order: +
Map-reduce partition columns: _col0 (type: decimal(26,12))
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ value expressions: _col1 (type: timestamp)
Reduce Operator Tree:
Join Operator
condition map:
@@ -400,7 +288,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-8
+ Stage: Stage-7
Map Reduce
Map Operator Tree:
TableScan
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out b/ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out
index 5313e9b..3467215 100644
--- a/ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out
+++ b/ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out
@@ -95,9 +95,7 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 4), Reducer 4 (PARTITION-LEVEL SORT, 4)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 4), Reducer 6 (PARTITION-LEVEL SORT, 4)
- Reducer 6 <- Map 5 (GROUP, 4)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 4), Map 3 (PARTITION-LEVEL SORT, 4)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -124,37 +122,22 @@ STAGE PLANS:
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (l_shipmode = 'AIR') (type: boolean)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((l_shipmode = 'AIR') and (l_linenumber = l_linenumber)) (type: boolean)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: l_orderkey (type: int), l_linenumber (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: int)
- sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
- Map 5
- Map Operator Tree:
- TableScan
- alias: li
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_linenumber (type: int)
- outputColumnNames: l_linenumber
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: l_linenumber (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -164,54 +147,18 @@ STAGE PLANS:
0 _col0 (type: int), 1 (type: int)
1 _col0 (type: int), _col1 (type: int)
outputColumnNames: _col1, _col2
- Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), _col2 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_exists.q.out b/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
index bc25efe..1901dba 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
@@ -24,9 +24,7 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Map 3 (PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -49,36 +47,22 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (value > 'val_9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((value = value) and (key = key) and (value > 'val_9')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: key, value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -96,42 +80,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col3 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -274,9 +222,7 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Map 3 (PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -299,34 +245,23 @@ STAGE PLANS:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
+ Filter Operator
+ predicate: (value = value) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: value (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -336,50 +271,14 @@ STAGE PLANS:
0 _col1 (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
[13/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/subquery_in.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_in.q.out b/ql/src/test/results/clientpositive/llap/subquery_in.q.out
index 155a901..78752a0 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_in.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_in.q.out
@@ -139,9 +139,7 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -166,35 +164,22 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -215,44 +200,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -493,12 +440,10 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
Reducer 4 <- Map 3 (SIMPLE_EDGE)
- Reducer 5 <- Reducer 4 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+ Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -523,29 +468,15 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- TopN Hash Memory Usage: 0.1
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
+ TopN Hash Memory Usage: 0.1
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -576,7 +507,7 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col2, _col5
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
PTF Operator
Function definitions:
Input definition
@@ -597,56 +528,35 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 2960 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string), _col5 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: min(_col1)
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: int)
Reducer 5
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: min(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string), _col1 (type: int)
mode: hash
@@ -657,7 +567,7 @@ STAGE PLANS:
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
+ Reducer 6
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -674,19 +584,6 @@ STAGE PLANS:
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 9
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -745,10 +642,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -773,35 +668,18 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 6
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -825,55 +703,21 @@ STAGE PLANS:
Reducer 4
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -1121,7 +965,6 @@ POSTHOOK: Input: default@lineitem
#### A masked pattern was here ####
108570 8571
4297 1798
-Warning: Shuffle Join MERGEJOIN[60][tables = [$hdt$_3, $hdt$_4]] in Stage 'Reducer 11' is a cross product
PREHOOK: query: explain select sum(l_extendedprice) from lineitem, part where p_partkey = l_partkey and l_quantity IN (select avg(l_quantity) from lineitem where l_partkey = p_partkey)
PREHOOK: type: QUERY
POSTHOOK: query: explain select sum(l_extendedprice) from lineitem, part where p_partkey = l_partkey and l_quantity IN (select avg(l_quantity) from lineitem where l_partkey = p_partkey)
@@ -1135,14 +978,11 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (CUSTOM_SIMPLE_EDGE), Map 13 (CUSTOM_SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (CUSTOM_SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE), Reducer 12 (SIMPLE_EDGE)
+ Reducer 7 <- Map 6 (SIMPLE_EDGE)
Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
- Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1165,33 +1005,6 @@ STAGE PLANS:
value expressions: _col1 (type: double), _col2 (type: double)
Execution mode: llap
LLAP IO: no inputs
- Map 10
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 100 Data size: 400 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 100 Data size: 400 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 13
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
Map 5
Map Operator Tree:
TableScan
@@ -1216,53 +1029,24 @@ STAGE PLANS:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 1200 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: l_partkey (type: int), l_quantity (type: double)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 100 Data size: 1200 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 100 Data size: 1200 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: double)
+ Filter Operator
+ predicate: (l_partkey = l_partkey) (type: boolean)
+ Statistics: Num rows: 50 Data size: 600 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: avg(l_quantity)
+ keys: l_partkey (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 25 Data size: 2100 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 25 Data size: 2100 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: struct<count:bigint,sum:double,input:double>)
Execution mode: llap
LLAP IO: no inputs
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col1
- Statistics: Num rows: 2600 Data size: 10400 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col1 (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 23 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 23 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 23 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 23 Data size: 92 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 2
+ Reducer 2
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
@@ -1317,62 +1101,35 @@ STAGE PLANS:
Reducer 7
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 23 Data size: 276 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: int), _col1 (type: double)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 23 Data size: 276 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: avg(_col1)
- keys: _col2 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 11 Data size: 924 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 11 Data size: 924 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: struct<count:bigint,sum:double,input:double>)
- Reducer 8
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: avg(VALUE._col0)
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 11 Data size: 132 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 25 Data size: 300 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col1 (type: double), _col0 (type: int)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 5 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 12 Data size: 144 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: double), _col1 (type: int)
sort order: ++
Map-reduce partition columns: _col0 (type: double), _col1 (type: int)
- Statistics: Num rows: 5 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 9
+ Statistics: Num rows: 12 Data size: 144 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 8
Execution mode: llap
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: double), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 5 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 12 Data size: 144 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: double), _col1 (type: int)
sort order: ++
Map-reduce partition columns: _col0 (type: double), _col1 (type: int)
- Statistics: Num rows: 5 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 12 Data size: 144 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -1380,7 +1137,6 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[60][tables = [$hdt$_3, $hdt$_4]] in Stage 'Reducer 11' is a cross product
PREHOOK: query: select sum(l_extendedprice) from lineitem, part where p_partkey = l_partkey and l_quantity IN (select avg(l_quantity) from lineitem where l_partkey = p_partkey)
PREHOOK: type: QUERY
PREHOOK: Input: default@lineitem
@@ -1405,10 +1161,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1436,33 +1190,19 @@ STAGE PLANS:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 6
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_type (type: string), p_size (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -1470,10 +1210,10 @@ STAGE PLANS:
Reduce Operator Tree:
Merge Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col4 (type: string), _col5 (type: int)
- 1 _col0 (type: string), _col1 (type: int)
+ 1 _col1 (type: string), _col0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 1 Data size: 619 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
@@ -1486,31 +1226,6 @@ STAGE PLANS:
Reducer 4
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
@@ -1518,27 +1233,18 @@ STAGE PLANS:
Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string), _col1 (type: int)
- mode: hash
+ mode: complete
outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 324 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 3 Data size: 324 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -1868,11 +1574,9 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
- Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 5 <- Map 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1896,33 +1600,23 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 7
- Map Operator Tree:
- TableScan
- alias: e
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_name = p_name) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_name (type: string), p_partkey (type: int)
+ outputColumnNames: p_name, p_partkey
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_name (type: string), p_partkey (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -1962,31 +1656,6 @@ STAGE PLANS:
Reducer 5
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: int)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
@@ -2001,19 +1670,6 @@ STAGE PLANS:
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 8
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -2205,10 +1861,9 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2221,9 +1876,9 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 26 Data size: 16094 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col1 (type: string), _col5 (type: int), _col5 (type: int)
- sort order: +++
- Map-reduce partition columns: _col1 (type: string), _col5 (type: int), _col5 (type: int)
+ key expressions: _col1 (type: string), _col5 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col5 (type: int)
Statistics: Num rows: 26 Data size: 16094 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: int), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
@@ -2245,7 +1900,7 @@ STAGE PLANS:
value expressions: _col1 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 6
+ Map 5
Map Operator Tree:
TableScan
alias: part
@@ -2267,10 +1922,10 @@ STAGE PLANS:
Reduce Operator Tree:
Merge Join Operator
condition map:
- Inner Join 0 to 1
+ Left Semi Join 0 to 1
keys:
- 0 _col1 (type: string), _col5 (type: int), _col5 (type: int)
- 1 _col0 (type: string), _col2 (type: int), _col1 (type: int)
+ 0 _col1 (type: string), _col5 (type: int)
+ 1 _col0 (type: string), _col1 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 1 Data size: 619 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
@@ -2291,34 +1946,21 @@ STAGE PLANS:
1 _col0 (type: int), (_col0 + 121150) (type: int)
outputColumnNames: _col1, _col3
Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col1 (type: string), _col3 (type: int)
- mode: hash
+ Select Operator
+ expressions: _col1 (type: string), _col3 (type: int)
outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: int)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: int), _col1 (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col2 (type: int), _col1 (type: int)
- sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col2 (type: int), _col1 (type: int)
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 1 Data size: 125 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 6
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -2362,8 +2004,7 @@ STAGE PLANS:
#### A masked pattern was here ####
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2388,33 +2029,19 @@ STAGE PLANS:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 3354 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3354 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col2 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col2 (type: int)
- Statistics: Num rows: 26 Data size: 3354 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 208 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_partkey (type: int), p_size (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: ((p_size = p_size) and (p_partkey = p_partkey)) (type: boolean)
+ Statistics: Num rows: 6 Data size: 774 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_partkey (type: int), p_name (type: string), p_size (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 3 Data size: 387 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string), _col2 (type: int)
+ sort order: +++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: string), _col2 (type: int)
+ Statistics: Num rows: 3 Data size: 387 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -2422,10 +2049,10 @@ STAGE PLANS:
Reduce Operator Tree:
Merge Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col0 (type: int), _col1 (type: string), _col5 (type: int)
- 1 _col0 (type: int), _col1 (type: string), _col2 (type: int)
+ 1 _col1 (type: int), _col0 (type: string), _col2 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 1 Data size: 619 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
@@ -2438,41 +2065,20 @@ STAGE PLANS:
Reducer 4
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int), _col2 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
- outputColumnNames: _col1, _col3, _col4
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: KEY._col0 (type: int), KEY._col1 (type: string), KEY._col2 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 3 Data size: 387 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col3 (type: int), _col1 (type: string), _col4 (type: int)
+ expressions: _col1 (type: string), _col0 (type: int), _col2 (type: int)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: string), _col2 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: string), _col2 (type: int)
- sort order: +++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: string), _col2 (type: int)
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int), KEY._col1 (type: int)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 3 Data size: 387 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: int), _col0 (type: string), _col2 (type: int)
+ sort order: +++
+ Map-reduce partition columns: _col1 (type: int), _col0 (type: string), _col2 (type: int)
+ Statistics: Num rows: 3 Data size: 387 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -2528,8 +2134,7 @@ STAGE PLANS:
#### A masked pattern was here ####
Edges:
Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2554,33 +2159,23 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2392 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_brand (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_brand = p_brand) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_type (type: string), p_brand (type: string)
+ outputColumnNames: p_type, p_brand
+ Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_type (type: string), p_brand (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -2588,7 +2183,7 @@ STAGE PLANS:
Reduce Operator Tree:
Merge Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col1 (type: string), _col2 (type: string)
1 _col0 (type: string), _col1 (type: string)
@@ -2604,41 +2199,16 @@ STAGE PLANS:
Reducer 4
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5096 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 13 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 12 Data size: 1104 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 1176 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -3095,13 +2665,11 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
- Reducer 4 <- Reducer 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -3120,7 +2688,27 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 10
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: sc
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (key = key) (type: boolean)
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
+ Execution mode: llap
+ LLAP IO: no inputs
+ Map 7
Map Operator Tree:
TableScan
alias: s1
@@ -3144,70 +2732,6 @@ STAGE PLANS:
value expressions: _col1 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: sc
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
- Map Operator Tree:
- TableScan
- alias: src
- Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: key (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: bigint)
- outputColumnNames: _col1
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: _col1 is not null (type: boolean)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: bigint)
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: bigint)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: bigint)
- sort order: +
- Map-reduce partition columns: _col0 (type: bigint)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -3218,7 +2742,7 @@ STAGE PLANS:
0 _col0 (type: string), _col1 (type: string)
1 _col1 (type: string), _col0 (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 2 Data size: 174 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 87 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
keys: _col0 (type: string)
@@ -3270,58 +2794,50 @@ STAGE PLANS:
Reducer 6
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col1 (type: string), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: string)
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 9
+ Statistics: Num rows: 125 Data size: 22250 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 8
Execution mode: llap
Reduce Operator Tree:
Group By Operator
+ aggregations: count(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: bigint)
+ outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: _col1 is not null (type: boolean)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: bigint)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: bigint)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: bigint)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: bigint)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -3816,10 +3332,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -3845,35 +3359,22 @@ STAGE PLANS:
alias: part
Statistics: Num rows: 26 Data size: 5798 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (p_size < 10) (type: boolean)
- Statistics: Num rows: 8 Data size: 1784 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((p_size < 10) and (p_mfgr = p_mfgr)) (type: boolean)
+ Statistics: Num rows: 4 Data size: 892 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: p_mfgr (type: string), p_name (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 6
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 4 Data size: 876 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 438 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 2 Data size: 438 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -3910,44 +3411,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 4 Data size: 876 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 4 Data size: 876 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -3986,10 +3449,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 5 <- Map 8 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -4012,58 +3473,41 @@ STAGE PLANS:
Map 3
Map Operator Tree:
TableScan
- alias: pp
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ alias: p
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: p_type is not null (type: boolean)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: p_type (type: string), p_size (type: int)
+ expressions: p_name (type: string), p_type (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col1 (type: int)
+ key expressions: _col1 (type: string)
sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 6
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_size (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
+ Map 5
Map Operator Tree:
TableScan
- alias: p
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
+ alias: pp
+ Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: p_type is not null (type: boolean)
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((p_size = p_size) and p_type is not null) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: p_name (type: string), p_type (type: string)
+ expressions: p_type (type: string), p_size (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col1 (type: string)
+ key expressions: _col0 (type: string)
sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: int)
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -4091,54 +3535,24 @@ STAGE PLANS:
condition map:
Inner Join 0 to 1
keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 18 Data size: 1944 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 18 Data size: 1944 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: int)
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 16 Data size: 2000 Basic stats
<TRUNCATED>
[14/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
index 21fd10c..53839a3 100644
--- a/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
+++ b/ql/src/test/results/clientpositive/llap/explainuser_1.q.out
@@ -1803,71 +1803,50 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
-Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 7 <- Map 6 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 4 <- Map 3 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 2 llap
- File Output Operator [FS_29]
- Select Operator [SEL_28] (rows=1 width=178)
+ File Output Operator [FS_19]
+ Select Operator [SEL_18] (rows=1 width=178)
Output:["_col0","_col1"]
- Filter Operator [FIL_27] (rows=1 width=182)
+ Filter Operator [FIL_17] (rows=1 width=182)
predicate:_col3 is null
- Merge Join Operator [MERGEJOIN_37] (rows=500 width=182)
- Conds:RS_24._col1=RS_25._col0(Left Outer),Output:["_col0","_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_22] (rows=500 width=182)
+ Conds:RS_14._col1=RS_15._col0(Left Outer),Output:["_col0","_col1","_col3"]
<-Map 1 [SIMPLE_EDGE] llap
- SHUFFLE [RS_24]
+ SHUFFLE [RS_14]
PartitionCols:_col1
Select Operator [SEL_1] (rows=500 width=178)
Output:["_col0","_col1"]
TableScan [TS_0] (rows=500 width=178)
default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 5 [SIMPLE_EDGE] llap
- SHUFFLE [RS_25]
+ <-Reducer 4 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_15]
PartitionCols:_col0
- Select Operator [SEL_23] (rows=56 width=95)
+ Select Operator [SEL_13] (rows=36 width=95)
Output:["_col0","_col1"]
- Group By Operator [GBY_22] (rows=56 width=91)
+ Group By Operator [GBY_12] (rows=36 width=91)
Output:["_col0"],keys:_col1
- Select Operator [SEL_18] (rows=83 width=178)
+ Select Operator [SEL_8] (rows=41 width=178)
Output:["_col1"]
- Group By Operator [GBY_17] (rows=83 width=178)
+ Group By Operator [GBY_7] (rows=41 width=178)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 4 [SIMPLE_EDGE] llap
- SHUFFLE [RS_16]
+ <-Map 3 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_6]
PartitionCols:_col0
- Group By Operator [GBY_15] (rows=83 width=178)
- Output:["_col0","_col1"],keys:_col2, _col0
- Select Operator [SEL_14] (rows=166 width=178)
- Output:["_col2","_col0"]
- Merge Join Operator [MERGEJOIN_36] (rows=166 width=178)
- Conds:RS_11._col1=RS_12._col0(Inner),Output:["_col0","_col2"]
- <-Map 3 [SIMPLE_EDGE] llap
- SHUFFLE [RS_11]
- PartitionCols:_col1
- Select Operator [SEL_4] (rows=166 width=178)
- Output:["_col0","_col1"]
- Filter Operator [FIL_34] (rows=166 width=178)
- predicate:(value > 'val_2')
- TableScan [TS_2] (rows=500 width=178)
- default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 7 [SIMPLE_EDGE] llap
- SHUFFLE [RS_12]
- PartitionCols:_col0
- Group By Operator [GBY_9] (rows=214 width=91)
- Output:["_col0"],keys:KEY._col0
- <-Map 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_8]
- PartitionCols:_col0
- Group By Operator [GBY_7] (rows=214 width=91)
- Output:["_col0"],keys:value
- TableScan [TS_5] (rows=500 width=91)
- default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["value"]
+ Group By Operator [GBY_5] (rows=41 width=178)
+ Output:["_col0","_col1"],keys:value, key
+ Select Operator [SEL_4] (rows=83 width=178)
+ Output:["value","key"]
+ Filter Operator [FIL_21] (rows=83 width=178)
+ predicate:((value = value) and (value > 'val_2'))
+ TableScan [TS_2] (rows=500 width=178)
+ default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
PREHOOK: query: explain select *
from src_cbo b
@@ -1891,25 +1870,23 @@ Plan optimized by CBO.
Vertex dependency in root stage
Reducer 2 <- Map 1 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
-Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 8 <- Map 7 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+Reducer 5 <- Map 4 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 3 llap
- File Output Operator [FS_33]
- Select Operator [SEL_32] (rows=1 width=178)
+ File Output Operator [FS_18]
+ Select Operator [SEL_17] (rows=1 width=178)
Output:["_col0","_col1"]
- Filter Operator [FIL_31] (rows=1 width=182)
+ Filter Operator [FIL_16] (rows=1 width=182)
predicate:_col4 is null
- Merge Join Operator [MERGEJOIN_41] (rows=250 width=182)
- Conds:RS_28._col0, _col1=RS_29._col0, _col1(Left Outer),Output:["_col0","_col1","_col4"]
+ Merge Join Operator [MERGEJOIN_21] (rows=250 width=182)
+ Conds:RS_13._col0, _col1=RS_14._col0, _col1(Left Outer),Output:["_col0","_col1","_col4"]
<-Reducer 2 [SIMPLE_EDGE] llap
- SHUFFLE [RS_28]
+ SHUFFLE [RS_13]
PartitionCols:_col0, _col1
Group By Operator [GBY_4] (rows=250 width=178)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
@@ -1922,43 +1899,22 @@ Stage-0
Output:["key","value"]
TableScan [TS_0] (rows=500 width=178)
default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_29]
+ <-Reducer 5 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_14]
PartitionCols:_col0, _col1
- Select Operator [SEL_27] (rows=1 width=182)
+ Select Operator [SEL_12] (rows=20 width=182)
Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_26] (rows=1 width=178)
+ Group By Operator [GBY_11] (rows=20 width=178)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 5 [SIMPLE_EDGE] llap
- SHUFFLE [RS_25]
+ <-Map 4 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_10]
PartitionCols:_col0, _col1
- Group By Operator [GBY_24] (rows=1 width=178)
- Output:["_col0","_col1"],keys:_col2, _col3
- Merge Join Operator [MERGEJOIN_40] (rows=1 width=178)
- Conds:RS_20._col0, _col1=RS_21._col0, _col1(Inner),Output:["_col2","_col3"]
- <-Map 4 [SIMPLE_EDGE] llap
- SHUFFLE [RS_20]
- PartitionCols:_col0, _col1
- Select Operator [SEL_8] (rows=166 width=178)
- Output:["_col0","_col1"]
- Filter Operator [FIL_38] (rows=166 width=178)
- predicate:(value > 'val_12')
- TableScan [TS_6] (rows=500 width=178)
- default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 8 [SIMPLE_EDGE] llap
- SHUFFLE [RS_21]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_18] (rows=250 width=178)
- Output:["_col0","_col1"],keys:_col0, _col1
- Group By Operator [GBY_13] (rows=250 width=178)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Map 7 [SIMPLE_EDGE] llap
- SHUFFLE [RS_12]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_11] (rows=250 width=178)
- Output:["_col0","_col1"],keys:key, value
- TableScan [TS_9] (rows=500 width=178)
- default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
+ Group By Operator [GBY_9] (rows=20 width=178)
+ Output:["_col0","_col1"],keys:key, value
+ Filter Operator [FIL_20] (rows=41 width=178)
+ predicate:((value = value) and (key = key) and (value > 'val_12'))
+ TableScan [TS_6] (rows=500 width=178)
+ default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
PREHOOK: query: create view cv1 as
select *
@@ -1991,55 +1947,34 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
-Reducer 6 <- Map 5 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 2 llap
- File Output Operator [FS_21]
- Merge Join Operator [MERGEJOIN_31] (rows=1 width=178)
- Conds:RS_17._col0, _col1=RS_18._col0, _col1(Left Semi),Output:["_col0","_col1"]
+ File Output Operator [FS_11]
+ Merge Join Operator [MERGEJOIN_16] (rows=1 width=178)
+ Conds:RS_7._col0, _col1=RS_8._col0, _col1(Left Semi),Output:["_col0","_col1"]
<-Map 1 [SIMPLE_EDGE] llap
- SHUFFLE [RS_17]
+ SHUFFLE [RS_7]
PartitionCols:_col0, _col1
Select Operator [SEL_1] (rows=500 width=178)
Output:["_col0","_col1"]
TableScan [TS_0] (rows=500 width=178)
default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"],properties:{"insideView":"TRUE"}
- <-Reducer 4 [SIMPLE_EDGE] llap
- SHUFFLE [RS_18]
+ <-Map 3 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_8]
PartitionCols:_col0, _col1
- Group By Operator [GBY_16] (rows=1 width=178)
+ Group By Operator [GBY_6] (rows=20 width=178)
Output:["_col0","_col1"],keys:_col0, _col1
- Select Operator [SEL_14] (rows=1 width=178)
+ Select Operator [SEL_4] (rows=41 width=178)
Output:["_col0","_col1"]
- Merge Join Operator [MERGEJOIN_30] (rows=1 width=178)
- Conds:RS_11._col0, _col1=RS_12._col0, _col1(Inner),Output:["_col2","_col3"]
- <-Map 3 [SIMPLE_EDGE] llap
- SHUFFLE [RS_11]
- PartitionCols:_col0, _col1
- Select Operator [SEL_4] (rows=166 width=178)
- Output:["_col0","_col1"]
- Filter Operator [FIL_28] (rows=166 width=178)
- predicate:(value > 'val_9')
- TableScan [TS_2] (rows=500 width=178)
- default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_12]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_9] (rows=250 width=178)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Map 5 [SIMPLE_EDGE] llap
- SHUFFLE [RS_8]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_7] (rows=250 width=178)
- Output:["_col0","_col1"],keys:key, value
- TableScan [TS_5] (rows=500 width=178)
- default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"],properties:{"insideView":"TRUE"}
+ Filter Operator [FIL_15] (rows=41 width=178)
+ predicate:((value = value) and (key = key) and (value > 'val_9'))
+ TableScan [TS_2] (rows=500 width=178)
+ default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
PREHOOK: query: explain select *
from (select *
@@ -2062,55 +1997,34 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
-Reducer 6 <- Map 5 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 2 llap
- File Output Operator [FS_21]
- Merge Join Operator [MERGEJOIN_31] (rows=1 width=178)
- Conds:RS_17._col0, _col1=RS_18._col0, _col1(Left Semi),Output:["_col0","_col1"]
+ File Output Operator [FS_11]
+ Merge Join Operator [MERGEJOIN_16] (rows=1 width=178)
+ Conds:RS_7._col0, _col1=RS_8._col0, _col1(Left Semi),Output:["_col0","_col1"]
<-Map 1 [SIMPLE_EDGE] llap
- SHUFFLE [RS_17]
+ SHUFFLE [RS_7]
PartitionCols:_col0, _col1
Select Operator [SEL_1] (rows=500 width=178)
Output:["_col0","_col1"]
TableScan [TS_0] (rows=500 width=178)
default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 4 [SIMPLE_EDGE] llap
- SHUFFLE [RS_18]
+ <-Map 3 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_8]
PartitionCols:_col0, _col1
- Group By Operator [GBY_16] (rows=1 width=178)
+ Group By Operator [GBY_6] (rows=20 width=178)
Output:["_col0","_col1"],keys:_col0, _col1
- Select Operator [SEL_14] (rows=1 width=178)
+ Select Operator [SEL_4] (rows=41 width=178)
Output:["_col0","_col1"]
- Merge Join Operator [MERGEJOIN_30] (rows=1 width=178)
- Conds:RS_11._col0, _col1=RS_12._col0, _col1(Inner),Output:["_col2","_col3"]
- <-Map 3 [SIMPLE_EDGE] llap
- SHUFFLE [RS_11]
- PartitionCols:_col0, _col1
- Select Operator [SEL_4] (rows=166 width=178)
- Output:["_col0","_col1"]
- Filter Operator [FIL_28] (rows=166 width=178)
- predicate:(value > 'val_9')
- TableScan [TS_2] (rows=500 width=178)
- default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
- <-Reducer 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_12]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_9] (rows=250 width=178)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Map 5 [SIMPLE_EDGE] llap
- SHUFFLE [RS_8]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_7] (rows=250 width=178)
- Output:["_col0","_col1"],keys:key, value
- TableScan [TS_5] (rows=500 width=178)
- default@src_cbo,b,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
+ Filter Operator [FIL_15] (rows=41 width=178)
+ predicate:((value = value) and (key = key) and (value > 'val_9'))
+ TableScan [TS_2] (rows=500 width=178)
+ default@src_cbo,a,Tbl:COMPLETE,Col:COMPLETE,Output:["key","value"]
PREHOOK: query: explain select *
from src_cbo
@@ -2167,40 +2081,47 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 10 <- Map 12 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
-Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Map 5 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
-Reducer 7 <- Map 6 (SIMPLE_EDGE), Reducer 11 (SIMPLE_EDGE)
-Reducer 9 <- Map 8 (SIMPLE_EDGE)
+Reducer 4 <- Map 6 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 4 llap
- File Output Operator [FS_44]
- Select Operator [SEL_43] (rows=5 width=8)
+ File Output Operator [FS_22]
+ Select Operator [SEL_21] (rows=5 width=8)
Output:["_col0","_col1"]
- Merge Join Operator [MERGEJOIN_65] (rows=5 width=8)
- Conds:RS_40._col1, _col4=RS_41._col0, _col1(Left Semi),Output:["_col0","_col3"]
+ Merge Join Operator [MERGEJOIN_32] (rows=5 width=8)
+ Conds:RS_18._col1, _col4=RS_19._col0, _col1(Left Semi),Output:["_col0","_col3"]
+ <-Map 6 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_19]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_17] (rows=2 width=8)
+ Output:["_col0","_col1"],keys:_col0, _col1
+ Select Operator [SEL_12] (rows=7 width=8)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_30] (rows=7 width=96)
+ predicate:((l_shipmode = 'AIR') and (l_linenumber = l_linenumber))
+ TableScan [TS_10] (rows=100 width=96)
+ default@lineitem,lineitem,Tbl:COMPLETE,Col:COMPLETE,Output:["l_orderkey","l_linenumber","l_shipmode"]
<-Reducer 3 [SIMPLE_EDGE] llap
- SHUFFLE [RS_40]
+ SHUFFLE [RS_18]
PartitionCols:_col1, _col4
- Merge Join Operator [MERGEJOIN_62] (rows=5 width=16)
- Conds:RS_35._col0=RS_36._col1(Inner),Output:["_col0","_col1","_col3","_col4"]
+ Merge Join Operator [MERGEJOIN_31] (rows=5 width=16)
+ Conds:RS_13._col0=RS_14._col1(Inner),Output:["_col0","_col1","_col3","_col4"]
<-Map 5 [SIMPLE_EDGE] llap
- SHUFFLE [RS_36]
+ SHUFFLE [RS_14]
PartitionCols:_col1
Select Operator [SEL_9] (rows=17 width=16)
Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_58] (rows=17 width=16)
+ Filter Operator [FIL_29] (rows=17 width=16)
predicate:((l_linenumber = 1) and l_partkey is not null)
TableScan [TS_7] (rows=100 width=16)
default@lineitem,li,Tbl:COMPLETE,Col:COMPLETE,Output:["l_orderkey","l_partkey","l_suppkey","l_linenumber"]
<-Reducer 2 [SIMPLE_EDGE] llap
- SHUFFLE [RS_35]
+ SHUFFLE [RS_13]
PartitionCols:_col0
Group By Operator [GBY_5] (rows=50 width=4)
Output:["_col0"],keys:KEY._col0
@@ -2209,63 +2130,10 @@ Stage-0
PartitionCols:_col0
Group By Operator [GBY_3] (rows=50 width=4)
Output:["_col0"],keys:l_partkey
- Filter Operator [FIL_57] (rows=100 width=4)
+ Filter Operator [FIL_28] (rows=100 width=4)
predicate:l_partkey is not null
TableScan [TS_0] (rows=100 width=4)
default@lineitem,lineitem,Tbl:COMPLETE,Col:COMPLETE,Output:["l_partkey"]
- <-Reducer 7 [SIMPLE_EDGE] llap
- SHUFFLE [RS_41]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_39] (rows=4 width=8)
- Output:["_col0","_col1"],keys:_col0, _col1
- Select Operator [SEL_34] (rows=14 width=8)
- Output:["_col0","_col1"]
- Merge Join Operator [MERGEJOIN_64] (rows=14 width=8)
- Conds:RS_31._col1=RS_32._col0(Inner),Output:["_col0","_col3"]
- <-Map 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_31]
- PartitionCols:_col1
- Select Operator [SEL_12] (rows=14 width=95)
- Output:["_col0","_col1"]
- Filter Operator [FIL_59] (rows=14 width=96)
- predicate:(l_shipmode = 'AIR')
- TableScan [TS_10] (rows=100 width=96)
- default@lineitem,lineitem,Tbl:COMPLETE,Col:COMPLETE,Output:["l_orderkey","l_linenumber","l_shipmode"]
- <-Reducer 11 [SIMPLE_EDGE] llap
- SHUFFLE [RS_32]
- PartitionCols:_col0
- Group By Operator [GBY_29] (rows=3 width=4)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 10 [SIMPLE_EDGE] llap
- SHUFFLE [RS_28]
- PartitionCols:_col0
- Group By Operator [GBY_27] (rows=3 width=4)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_63] (rows=34 width=4)
- Conds:RS_23._col0=RS_24._col0(Inner),Output:["_col2"]
- <-Map 12 [SIMPLE_EDGE] llap
- SHUFFLE [RS_24]
- PartitionCols:_col0
- Select Operator [SEL_22] (rows=100 width=8)
- Output:["_col0","_col1"]
- Filter Operator [FIL_61] (rows=100 width=8)
- predicate:l_partkey is not null
- TableScan [TS_20] (rows=100 width=8)
- default@lineitem,li,Tbl:COMPLETE,Col:COMPLETE,Output:["l_partkey","l_linenumber"]
- <-Reducer 9 [SIMPLE_EDGE] llap
- SHUFFLE [RS_23]
- PartitionCols:_col0
- Group By Operator [GBY_18] (rows=50 width=4)
- Output:["_col0"],keys:KEY._col0
- <-Map 8 [SIMPLE_EDGE] llap
- SHUFFLE [RS_17]
- PartitionCols:_col0
- Group By Operator [GBY_16] (rows=50 width=4)
- Output:["_col0"],keys:l_partkey
- Filter Operator [FIL_60] (rows=100 width=4)
- predicate:l_partkey is not null
- TableScan [TS_13] (rows=100 width=4)
- default@lineitem,lineitem,Tbl:COMPLETE,Col:COMPLETE,Output:["l_partkey"]
PREHOOK: query: explain select key, value, count(*)
from src_cbo b
@@ -2516,128 +2384,71 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 10 <- Map 9 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
-Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
-Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
-Reducer 14 <- Map 13 (SIMPLE_EDGE)
-Reducer 16 <- Map 15 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 12 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
-Reducer 8 <- Map 7 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+Reducer 5 <- Map 4 (SIMPLE_EDGE)
+Reducer 7 <- Map 6 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 3 llap
- File Output Operator [FS_54]
- Select Operator [SEL_53] (rows=13 width=223)
+ File Output Operator [FS_26]
+ Select Operator [SEL_25] (rows=13 width=223)
Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_52] (rows=13 width=243)
- predicate:CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col8 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END
- Merge Join Operator [MERGEJOIN_76] (rows=26 width=243)
- Conds:RS_49._col0, _col1=RS_50._col3, _col1(Left Outer),Output:["_col0","_col1","_col2","_col4","_col5","_col8"]
- <-Reducer 12 [SIMPLE_EDGE] llap
- SHUFFLE [RS_50]
- PartitionCols:_col3, _col1
- Merge Join Operator [MERGEJOIN_75] (rows=2 width=223)
- Conds:RS_42._col0=RS_43._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Reducer 11 [SIMPLE_EDGE] llap
- SHUFFLE [RS_42]
- PartitionCols:_col0
- Select Operator [SEL_35] (rows=4 width=223)
- Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_34] (rows=4 width=219)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 10 [SIMPLE_EDGE] llap
- SHUFFLE [RS_33]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_32] (rows=4 width=219)
- Output:["_col0","_col1"],keys:_col1, _col2
- Merge Join Operator [MERGEJOIN_74] (rows=8 width=219)
- Conds:RS_28._col0=RS_29._col0(Inner),Output:["_col1","_col2"]
- <-Map 9 [SIMPLE_EDGE] llap
- SHUFFLE [RS_28]
- PartitionCols:_col0
- Select Operator [SEL_21] (rows=8 width=219)
- Output:["_col0","_col1"]
- Filter Operator [FIL_69] (rows=8 width=223)
- predicate:(p_size < 10)
- TableScan [TS_19] (rows=26 width=223)
- default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_name","p_mfgr","p_size"]
- <-Reducer 14 [SIMPLE_EDGE] llap
- SHUFFLE [RS_29]
- PartitionCols:_col0
- Group By Operator [GBY_26] (rows=5 width=98)
- Output:["_col0"],keys:KEY._col0
- <-Map 13 [SIMPLE_EDGE] llap
- SHUFFLE [RS_25]
- PartitionCols:_col0
- Group By Operator [GBY_24] (rows=5 width=98)
- Output:["_col0"],keys:p_mfgr
- TableScan [TS_22] (rows=26 width=98)
- default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr"]
- <-Reducer 16 [SIMPLE_EDGE] llap
- SHUFFLE [RS_43]
- PartitionCols:_col0
- Group By Operator [GBY_40] (rows=13 width=121)
- Output:["_col0"],keys:KEY._col0
- <-Map 15 [SIMPLE_EDGE] llap
- SHUFFLE [RS_39]
- PartitionCols:_col0
- Group By Operator [GBY_38] (rows=13 width=121)
- Output:["_col0"],keys:p_name
- TableScan [TS_36] (rows=26 width=121)
- default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_name"]
+ Filter Operator [FIL_24] (rows=13 width=243)
+ predicate:(not CASE WHEN ((_col4 = 0)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (true) ELSE (false) END)
+ Merge Join Operator [MERGEJOIN_32] (rows=26 width=243)
+ Conds:RS_21._col0, _col1=RS_22._col0, _col1(Left Outer),Output:["_col0","_col1","_col2","_col4","_col5","_col8"]
<-Reducer 2 [SIMPLE_EDGE] llap
- SHUFFLE [RS_49]
+ SHUFFLE [RS_21]
PartitionCols:_col0, _col1
- Merge Join Operator [MERGEJOIN_73] (rows=26 width=239)
- Conds:RS_46._col1=RS_47._col0(Left Outer),Output:["_col0","_col1","_col2","_col4","_col5"]
+ Merge Join Operator [MERGEJOIN_31] (rows=26 width=239)
+ Conds:RS_18._col1=RS_19._col0(Left Outer),Output:["_col0","_col1","_col2","_col4","_col5"]
<-Map 1 [SIMPLE_EDGE] llap
- SHUFFLE [RS_46]
+ SHUFFLE [RS_18]
PartitionCols:_col1
Select Operator [SEL_1] (rows=26 width=223)
Output:["_col0","_col1","_col2"]
TableScan [TS_0] (rows=26 width=223)
default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_name","p_mfgr","p_size"]
- <-Reducer 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_47]
+ <-Reducer 5 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_19]
PartitionCols:_col0
- Group By Operator [GBY_17] (rows=2 width=114)
+ Group By Operator [GBY_7] (rows=1 width=114)
Output:["_col0","_col1","_col2"],aggregations:["count(VALUE._col0)","count(VALUE._col1)"],keys:KEY._col0
- <-Reducer 5 [SIMPLE_EDGE] llap
- SHUFFLE [RS_16]
+ <-Map 4 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_6]
PartitionCols:_col0
- Group By Operator [GBY_15] (rows=2 width=114)
- Output:["_col0","_col1","_col2"],aggregations:["count()","count(_col1)"],keys:_col2
- Select Operator [SEL_14] (rows=8 width=219)
- Output:["_col2","_col1"]
- Merge Join Operator [MERGEJOIN_72] (rows=8 width=219)
- Conds:RS_11._col0=RS_12._col0(Inner),Output:["_col1","_col2"]
- <-Map 4 [SIMPLE_EDGE] llap
- SHUFFLE [RS_11]
- PartitionCols:_col0
- Select Operator [SEL_4] (rows=8 width=219)
- Output:["_col0","_col1"]
- Filter Operator [FIL_67] (rows=8 width=223)
- predicate:(p_size < 10)
- TableScan [TS_2] (rows=26 width=223)
- default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_name","p_mfgr","p_size"]
- <-Reducer 8 [SIMPLE_EDGE] llap
- SHUFFLE [RS_12]
- PartitionCols:_col0
- Group By Operator [GBY_9] (rows=5 width=98)
- Output:["_col0"],keys:KEY._col0
- <-Map 7 [SIMPLE_EDGE] llap
- SHUFFLE [RS_8]
- PartitionCols:_col0
- Group By Operator [GBY_7] (rows=5 width=98)
- Output:["_col0"],keys:p_mfgr
- TableScan [TS_5] (rows=26 width=98)
- default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr"]
+ Group By Operator [GBY_5] (rows=1 width=114)
+ Output:["_col0","_col1","_col2"],aggregations:["count()","count(p_name)"],keys:p_mfgr
+ Select Operator [SEL_4] (rows=4 width=223)
+ Output:["p_mfgr","p_name"]
+ Filter Operator [FIL_29] (rows=4 width=223)
+ predicate:((p_size < 10) and (p_mfgr = p_mfgr))
+ TableScan [TS_2] (rows=26 width=223)
+ default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_name","p_mfgr","p_size"]
+ <-Reducer 7 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_22]
+ PartitionCols:_col0, _col1
+ Select Operator [SEL_17] (rows=1 width=223)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_16] (rows=1 width=219)
+ predicate:(_col0 = _col0)
+ Group By Operator [GBY_14] (rows=2 width=219)
+ Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
+ <-Map 6 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_13]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_12] (rows=2 width=219)
+ Output:["_col0","_col1"],keys:p_name, p_mfgr
+ Select Operator [SEL_11] (rows=4 width=223)
+ Output:["p_name","p_mfgr"]
+ Filter Operator [FIL_30] (rows=4 width=223)
+ predicate:((p_size < 10) and (p_mfgr = p_mfgr))
+ TableScan [TS_9] (rows=26 width=223)
+ default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_name","p_mfgr","p_size"]
PREHOOK: query: explain select p_name, p_size
from
@@ -2752,118 +2563,65 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 11 <- Map 10 (SIMPLE_EDGE)
-Reducer 12 <- Reducer 11 (SIMPLE_EDGE)
-Reducer 14 <- Map 13 (SIMPLE_EDGE)
-Reducer 15 <- Reducer 14 (SIMPLE_EDGE), Reducer 20 (SIMPLE_EDGE)
-Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
-Reducer 17 <- Reducer 16 (SIMPLE_EDGE), Reducer 22 (SIMPLE_EDGE)
-Reducer 19 <- Map 18 (SIMPLE_EDGE)
+Reducer 10 <- Map 9 (SIMPLE_EDGE)
+Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
-Reducer 20 <- Reducer 19 (SIMPLE_EDGE)
-Reducer 22 <- Map 21 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 17 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
Reducer 7 <- Map 6 (SIMPLE_EDGE)
-Reducer 8 <- Reducer 12 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
-Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
+Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 5 llap
- File Output Operator [FS_83]
- Select Operator [SEL_82] (rows=2 width=106)
+ File Output Operator [FS_42]
+ Select Operator [SEL_41] (rows=3 width=106)
Output:["_col0","_col1"]
<-Reducer 4 [SIMPLE_EDGE] llap
- SHUFFLE [RS_81]
- Select Operator [SEL_80] (rows=2 width=106)
+ SHUFFLE [RS_40]
+ Select Operator [SEL_39] (rows=3 width=106)
Output:["_col0","_col1"]
- Filter Operator [FIL_79] (rows=2 width=126)
- predicate:CASE WHEN ((_col3 = 0)) THEN (true) WHEN (_col3 is null) THEN (true) WHEN (_col7 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (false) ELSE (true) END
- Merge Join Operator [MERGEJOIN_108] (rows=5 width=126)
- Conds:RS_76._col0, _col1=RS_77._col3, _col1(Left Outer),Output:["_col0","_col1","_col3","_col4","_col7"]
- <-Reducer 17 [SIMPLE_EDGE] llap
- SHUFFLE [RS_77]
- PartitionCols:_col3, _col1
- Merge Join Operator [MERGEJOIN_107] (rows=1 width=110)
- Conds:RS_69._col0=RS_70._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Reducer 16 [SIMPLE_EDGE] llap
- SHUFFLE [RS_69]
- PartitionCols:_col0
- Select Operator [SEL_57] (rows=1 width=110)
- Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_56] (rows=1 width=106)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 15 [SIMPLE_EDGE] llap
- SHUFFLE [RS_55]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_54] (rows=1 width=106)
- Output:["_col0","_col1"],keys:_col0, _col3
- Merge Join Operator [MERGEJOIN_106] (rows=1 width=106)
- Conds:RS_50._col1=RS_51._col0(Inner),Output:["_col0","_col3"]
- <-Reducer 14 [SIMPLE_EDGE] llap
- SHUFFLE [RS_50]
- PartitionCols:_col1
- Select Operator [SEL_38] (rows=1 width=114)
- Output:["_col0","_col1"]
- Filter Operator [FIL_98] (rows=1 width=114)
- predicate:((_col2 - _col1) > 600.0)
- Group By Operator [GBY_36] (rows=5 width=114)
- Output:["_col0","_col1","_col2"],aggregations:["min(VALUE._col0)","max(VALUE._col1)"],keys:KEY._col0
- <-Map 13 [SIMPLE_EDGE] llap
- SHUFFLE [RS_35]
- PartitionCols:_col0
- Group By Operator [GBY_34] (rows=5 width=114)
- Output:["_col0","_col1","_col2"],aggregations:["min(p_retailprice)","max(p_retailprice)"],keys:p_mfgr
- TableScan [TS_32] (rows=26 width=106)
- default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
- <-Reducer 20 [SIMPLE_EDGE] llap
- SHUFFLE [RS_51]
- PartitionCols:_col0
- Group By Operator [GBY_48] (rows=2 width=8)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 19 [SIMPLE_EDGE] llap
- SHUFFLE [RS_47]
- PartitionCols:_col0
- Group By Operator [GBY_46] (rows=2 width=8)
- Output:["_col0"],keys:_col1
- Select Operator [SEL_103] (rows=5 width=8)
- Output:["_col1"]
- Group By Operator [GBY_43] (rows=5 width=106)
- Output:["_col0","_col1"],aggregations:["min(VALUE._col0)"],keys:KEY._col0
- <-Map 18 [SIMPLE_EDGE] llap
- SHUFFLE [RS_42]
- PartitionCols:_col0
- Group By Operator [GBY_41] (rows=5 width=106)
- Output:["_col0","_col1"],aggregations:["min(p_retailprice)"],keys:p_mfgr
- Select Operator [SEL_40] (rows=26 width=106)
- Output:["p_mfgr","p_retailprice"]
- TableScan [TS_39] (rows=26 width=106)
- default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
- <-Reducer 22 [SIMPLE_EDGE] llap
- SHUFFLE [RS_70]
- PartitionCols:_col0
- Group By Operator [GBY_67] (rows=5 width=98)
- Output:["_col0"],keys:_col0
- Group By Operator [GBY_62] (rows=5 width=98)
- Output:["_col0"],keys:KEY._col0
- <-Map 21 [SIMPLE_EDGE] llap
- SHUFFLE [RS_61]
- PartitionCols:_col0
- Group By Operator [GBY_60] (rows=5 width=98)
- Output:["_col0"],keys:p_mfgr
- TableScan [TS_58] (rows=26 width=98)
- default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr"]
+ Filter Operator [FIL_38] (rows=3 width=126)
+ predicate:(not CASE WHEN ((_col3 = 0)) THEN (false) WHEN (_col3 is null) THEN (false) WHEN (_col7 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (true) ELSE (false) END)
+ Merge Join Operator [MERGEJOIN_49] (rows=5 width=126)
+ Conds:RS_35._col0, _col1=RS_36._col0, _col1(Left Outer),Output:["_col0","_col1","_col3","_col4","_col7"]
+ <-Reducer 11 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_36]
+ PartitionCols:_col0, _col1
+ Select Operator [SEL_31] (rows=1 width=110)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_30] (rows=1 width=106)
+ predicate:(_col0 = _col0)
+ Group By Operator [GBY_28] (rows=1 width=106)
+ Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
+ <-Reducer 10 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_27]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_26] (rows=1 width=106)
+ Output:["_col0","_col1"],keys:_col0, _col1
+ Select Operator [SEL_24] (rows=1 width=114)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_46] (rows=1 width=114)
+ predicate:((_col1 = _col1) and ((_col2 - _col1) > 600.0))
+ Group By Operator [GBY_22] (rows=5 width=114)
+ Output:["_col0","_col1","_col2"],aggregations:["min(VALUE._col0)","max(VALUE._col1)"],keys:KEY._col0
+ <-Map 9 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_21]
+ PartitionCols:_col0
+ Group By Operator [GBY_20] (rows=5 width=114)
+ Output:["_col0","_col1","_col2"],aggregations:["min(p_retailprice)","max(p_retailprice)"],keys:p_mfgr
+ TableScan [TS_18] (rows=26 width=106)
+ default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
<-Reducer 3 [SIMPLE_EDGE] llap
- SHUFFLE [RS_76]
+ SHUFFLE [RS_35]
PartitionCols:_col0, _col1
- Merge Join Operator [MERGEJOIN_105] (rows=5 width=122)
- Conds:RS_73._col1=RS_74._col0(Left Outer),Output:["_col0","_col1","_col3","_col4"]
+ Merge Join Operator [MERGEJOIN_48] (rows=5 width=122)
+ Conds:RS_32._col1=RS_33._col0(Left Outer),Output:["_col0","_col1","_col3","_col4"]
<-Reducer 2 [SIMPLE_EDGE] llap
- SHUFFLE [RS_73]
+ SHUFFLE [RS_32]
PartitionCols:_col1
Group By Operator [GBY_4] (rows=5 width=106)
Output:["_col0","_col1"],aggregations:["min(VALUE._col0)"],keys:KEY._col0
@@ -2876,61 +2634,31 @@ Stage-0
Output:["p_mfgr","p_retailprice"]
TableScan [TS_0] (rows=26 width=106)
default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
- <-Reducer 9 [SIMPLE_EDGE] llap
- SHUFFLE [RS_74]
+ <-Reducer 8 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_33]
PartitionCols:_col0
- Group By Operator [GBY_30] (rows=1 width=24)
+ Group By Operator [GBY_16] (rows=1 width=24)
Output:["_col0","_col1","_col2"],aggregations:["count(VALUE._col0)","count(VALUE._col1)"],keys:KEY._col0
- <-Reducer 8 [SIMPLE_EDGE] llap
- SHUFFLE [RS_29]
+ <-Reducer 7 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_15]
PartitionCols:_col0
- Group By Operator [GBY_28] (rows=1 width=24)
- Output:["_col0","_col1","_col2"],aggregations:["count()","count(_col0)"],keys:_col3
- Select Operator [SEL_27] (rows=1 width=106)
- Output:["_col3","_col0"]
- Merge Join Operator [MERGEJOIN_104] (rows=1 width=106)
- Conds:RS_24._col1=RS_25._col0(Inner),Output:["_col0","_col3"]
- <-Reducer 12 [SIMPLE_EDGE] llap
- SHUFFLE [RS_25]
- PartitionCols:_col0
- Group By Operator [GBY_22] (rows=2 width=8)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 11 [SIMPLE_EDGE] llap
- SHUFFLE [RS_21]
- PartitionCols:_col0
- Group By Operator [GBY_20] (rows=2 width=8)
- Output:["_col0"],keys:_col1
- Select Operator [SEL_102] (rows=5 width=8)
- Output:["_col1"]
- Group By Operator [GBY_17] (rows=5 width=106)
- Output:["_col0","_col1"],aggregations:["min(VALUE._col0)"],keys:KEY._col0
- <-Map 10 [SIMPLE_EDGE] llap
- SHUFFLE [RS_16]
- PartitionCols:_col0
- Group By Operator [GBY_15] (rows=5 width=106)
- Output:["_col0","_col1"],aggregations:["min(p_retailprice)"],keys:p_mfgr
- Select Operator [SEL_14] (rows=26 width=106)
- Output:["p_mfgr","p_retailprice"]
- TableScan [TS_13] (rows=26 width=106)
- default@part,b,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
- <-Reducer 7 [SIMPLE_EDGE] llap
- SHUFFLE [RS_24]
- PartitionCols:_col1
- Select Operator [SEL_12] (rows=1 width=114)
- Output:["_col0","_col1"]
- Filter Operator [FIL_96] (rows=1 width=114)
- predicate:((_col2 - _col1) > 600.0)
- Group By Operator [GBY_10] (rows=5 width=114)
- Output:["_col0","_col1","_col2"],aggregations:["min(VALUE._col0)","max(VALUE._col1)"],keys:KEY._col0
- <-Map 6 [SIMPLE_EDGE] llap
- SHUFFLE [RS_9]
- PartitionCols:_col0
- Group By Operator [GBY_8] (rows=5 width=114)
- Output:["_col0","_col1","_col2"],aggregations:["min(p_retailprice)","max(p_retailprice)"],keys:p_mfgr
- Select Operator [SEL_7] (rows=26 width=106)
- Output:["p_mfgr","p_retailprice"]
- TableScan [TS_6] (rows=26 width=106)
- default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
+ Group By Operator [GBY_14] (rows=1 width=24)
+ Output:["_col0","_col1","_col2"],aggregations:["count()","count(_col0)"],keys:_col1
+ Select Operator [SEL_12] (rows=1 width=114)
+ Output:["_col1","_col0"]
+ Filter Operator [FIL_45] (rows=1 width=114)
+ predicate:((_col1 = _col1) and ((_col2 - _col1) > 600.0))
+ Group By Operator [GBY_10] (rows=5 width=114)
+ Output:["_col0","_col1","_col2"],aggregations:["min(VALUE._col0)","max(VALUE._col1)"],keys:KEY._col0
+ <-Map 6 [SIMPLE_EDGE] llap
+ SHUFFLE [RS_9]
+ PartitionCols:_col0
+ Group By Operator [GBY_8] (rows=5 width=114)
+ Output:["_col0","_col1","_col2"],aggregations:["min(p_retailprice)","max(p_retailprice)"],keys:p_mfgr
+ Select Operator [SEL_7] (rows=26 width=106)
+ Output:["p_mfgr","p_retailprice"]
+ TableScan [TS_6] (rows=26 width=106)
+ default@part,part,Tbl:COMPLETE,Col:COMPLETE,Output:["p_mfgr","p_retailprice"]
PREHOOK: query: explain select count(c_int) over(), sum(c_float) over(), max(c_int) over(), min(c_int) over(), row_number() over(), rank() over(), dense_rank() over(), percent_rank() over(), lead(c_int, 2, c_int) over(), lag(c_float, 2, c_float) over() from cbo_t1
PREHOOK: type: QUERY
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_exists.q.out b/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
index 23d68ab..9e0a1d7 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_exists.q.out
@@ -25,9 +25,7 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -52,34 +50,22 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value > 'val_9') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((value = value) and (key = key) and (value > 'val_9')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -100,44 +86,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col3 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -281,9 +229,7 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -308,32 +254,23 @@ STAGE PLANS:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (value = value) (type: boolean)
+ Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: value (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 250 Data size: 22750 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 107 Data size: 9737 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 107 Data size: 9737 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -346,52 +283,14 @@ STAGE PLANS:
0 _col1 (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col1 (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 45500 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 214 Data size: 19474 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
[02/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_notexists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_notexists.q.out b/ql/src/test/results/clientpositive/subquery_notexists.q.out
index b916d2f..268f659 100644
--- a/ql/src/test/results/clientpositive/subquery_notexists.q.out
+++ b/ql/src/test/results/clientpositive/subquery_notexists.q.out
@@ -17,110 +17,40 @@ where not exists
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-4 is a root stage
- Stage-2 depends on stages: Stage-4
- Stage-3 depends on stages: Stage-2
- Stage-1 depends on stages: Stage-3
+ Stage-2 is a root stage
+ Stage-1 depends on stages: Stage-2
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-4
+ Stage: Stage-2
Map Reduce
Map Operator Tree:
TableScan
- alias: b
+ alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: key, value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((value = value) and (key = key) and (value > 'val_2')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: key (type: string), value (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (value > 'val_2') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -148,7 +78,7 @@ STAGE PLANS:
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: boolean)
Reduce Operator Tree:
Join Operator
@@ -338,46 +268,11 @@ where not exists
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-4 is a root stage
- Stage-2 depends on stages: Stage-4
- Stage-3 depends on stages: Stage-2
- Stage-1 depends on stages: Stage-3
+ Stage-2 is a root stage
+ Stage-1 depends on stages: Stage-2
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-2
Map Reduce
Map Operator Tree:
@@ -385,77 +280,41 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (value > 'val_2') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((value = value) and (value > 'val_2')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ expressions: value (type: string), key (type: string)
+ outputColumnNames: value, key
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: value (type: string), key (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col1 (type: string)
mode: complete
outputColumnNames: _col0
- Statistics: Num rows: 68 Data size: 722 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 68 Data size: 722 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -484,7 +343,7 @@ STAGE PLANS:
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 68 Data size: 722 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: boolean)
Reduce Operator Tree:
Join Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_notexists_having.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_notexists_having.q.out b/ql/src/test/results/clientpositive/subquery_notexists_having.q.out
index 0ca743d..d78a552 100644
--- a/ql/src/test/results/clientpositive/subquery_notexists_having.q.out
+++ b/ql/src/test/results/clientpositive/subquery_notexists_having.q.out
@@ -20,10 +20,8 @@ having not exists
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1, Stage-4
- Stage-5 is a root stage
- Stage-3 depends on stages: Stage-5
- Stage-4 depends on stages: Stage-3
+ Stage-2 depends on stages: Stage-1, Stage-3
+ Stage-3 is a root stage
Stage-0 depends on stages: Stage-2
STAGE PLANS:
@@ -74,7 +72,7 @@ STAGE PLANS:
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: boolean)
Reduce Operator Tree:
Join Operator
@@ -100,44 +98,6 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-5
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: key, value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: complete
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-3
Map Reduce
Map Operator Tree:
@@ -145,63 +105,28 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (value > 'val_12') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
+ predicate: ((value = value) and (key = key) and (value > 'val_12')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -273,10 +198,8 @@ having not exists
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1, Stage-4
- Stage-5 is a root stage
- Stage-3 depends on stages: Stage-5
- Stage-4 depends on stages: Stage-3
+ Stage-2 depends on stages: Stage-1, Stage-3
+ Stage-3 is a root stage
Stage-0 depends on stages: Stage-2
STAGE PLANS:
@@ -332,7 +255,7 @@ STAGE PLANS:
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 45 Data size: 479 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: boolean)
Reduce Operator Tree:
Join Operator
@@ -358,48 +281,6 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-5
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string), key (type: string)
- outputColumnNames: value, key
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string), key (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: string)
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-3
Map Reduce
Map Operator Tree:
@@ -407,77 +288,41 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (value > 'val_12') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((value = value) and (value > 'val_12')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ expressions: value (type: string), key (type: string)
+ outputColumnNames: value, key
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: value (type: string), key (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col1
- Statistics: Num rows: 91 Data size: 969 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col1 (type: string)
mode: complete
outputColumnNames: _col0
- Statistics: Num rows: 45 Data size: 479 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 45 Data size: 479 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 20 Data size: 212 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_notin_having.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_notin_having.q.out b/ql/src/test/results/clientpositive/subquery_notin_having.q.out
index 644e70e..375a48a 100644
--- a/ql/src/test/results/clientpositive/subquery_notin_having.q.out
+++ b/ql/src/test/results/clientpositive/subquery_notin_having.q.out
@@ -227,20 +227,12 @@ having b.p_mfgr not in
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1, Stage-6
- Stage-3 depends on stages: Stage-2, Stage-12
+ Stage-2 depends on stages: Stage-1, Stage-5
+ Stage-3 depends on stages: Stage-2, Stage-7
Stage-4 is a root stage
- Stage-5 depends on stages: Stage-4, Stage-8
- Stage-6 depends on stages: Stage-5
- Stage-7 is a root stage
- Stage-8 depends on stages: Stage-7
- Stage-9 is a root stage
- Stage-10 depends on stages: Stage-9, Stage-14
- Stage-11 depends on stages: Stage-10
- Stage-12 depends on stages: Stage-11, Stage-15
- Stage-13 is a root stage
- Stage-14 depends on stages: Stage-13
- Stage-15 is a root stage
+ Stage-5 depends on stages: Stage-4
+ Stage-6 is a root stage
+ Stage-7 depends on stages: Stage-6
Stage-0 depends on stages: Stage-3
STAGE PLANS:
@@ -295,7 +287,7 @@ STAGE PLANS:
key expressions: _col0 (type: double)
sort order: +
Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 3 Data size: 399 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reduce Operator Tree:
Join Operator
@@ -325,10 +317,10 @@ STAGE PLANS:
value expressions: _col3 (type: bigint), _col4 (type: bigint)
TableScan
Reduce Output Operator
- key expressions: _col3 (type: string), _col1 (type: double)
+ key expressions: _col0 (type: string), _col1 (type: double)
sort order: ++
- Map-reduce partition columns: _col3 (type: string), _col1 (type: double)
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: double)
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: boolean)
Reduce Operator Tree:
Join Operator
@@ -336,19 +328,19 @@ STAGE PLANS:
Left Outer Join0 to 1
keys:
0 _col0 (type: string), _col1 (type: double)
- 1 _col3 (type: string), _col1 (type: double)
+ 1 _col0 (type: string), _col1 (type: double)
outputColumnNames: _col0, _col1, _col3, _col4, _col7
Statistics: Num rows: 15 Data size: 1903 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: CASE WHEN ((_col3 = 0)) THEN (true) WHEN (_col3 is null) THEN (true) WHEN (_col7 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (false) ELSE (true) END (type: boolean)
- Statistics: Num rows: 7 Data size: 888 Basic stats: COMPLETE Column stats: NONE
+ predicate: (not CASE WHEN ((_col3 = 0)) THEN (false) WHEN (_col3 is null) THEN (false) WHEN (_col7 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (true) ELSE (false) END) (type: boolean)
+ Statistics: Num rows: 8 Data size: 1014 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: double)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 7 Data size: 888 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 8 Data size: 1014 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 7 Data size: 888 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 8 Data size: 1014 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -384,70 +376,34 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((_col2 - _col1) > 600.0) (type: boolean)
- Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((_col1 = _col1) and ((_col2 - _col1) > 600.0)) (type: boolean)
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: string), _col1 (type: double)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ expressions: _col1 (type: double), _col0 (type: string)
+ outputColumnNames: _col1, _col0
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count(), count(_col0)
+ keys: _col1 (type: double)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-5
Map Reduce
Map Operator Tree:
TableScan
Reduce Output Operator
- key expressions: _col1 (type: double)
- sort order: +
- Map-reduce partition columns: _col1 (type: double)
- Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: double)
- sort order: +
- Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: double)
- 1 _col0 (type: double)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col3 (type: double), _col0 (type: string)
- outputColumnNames: _col3, _col0
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col0)
- keys: _col3 (type: double)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-6
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
key expressions: _col0 (type: double)
sort order: +
Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reduce Operator Tree:
Group By Operator
@@ -455,7 +411,7 @@ STAGE PLANS:
keys: KEY._col0 (type: double)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 3 Data size: 399 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -463,74 +419,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-7
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_mfgr (type: string), p_retailprice (type: double)
- outputColumnNames: p_mfgr, p_retailprice
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(p_retailprice)
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: double)
- Reduce Operator Tree:
- Group By Operator
- aggregations: min(VALUE._col0)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: double)
- outputColumnNames: _col1
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: double)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-8
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: double)
- sort order: +
- Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: double)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-9
+ Stage: Stage-6
Map Reduce
Map Operator Tree:
TableScan
@@ -560,57 +449,25 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((_col2 - _col1) > 600.0) (type: boolean)
- Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((_col1 = _col1) and ((_col2 - _col1) > 600.0)) (type: boolean)
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: double)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-10
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col1 (type: double)
- sort order: +
- Map-reduce partition columns: _col1 (type: double)
- Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: double)
- sort order: +
- Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: double)
- 1 _col0 (type: double)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col3 (type: double)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: double)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-11
+ Stage: Stage-7
Map Reduce
Map Operator Tree:
TableScan
@@ -618,94 +475,20 @@ STAGE PLANS:
key expressions: _col0 (type: string), _col1 (type: double)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: double)
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: double)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 399 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: double), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 3 Data size: 399 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-12
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 3 Data size: 399 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: double), _col2 (type: boolean)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 6 Data size: 798 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-13
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: p_mfgr (type: string), p_retailprice (type: double)
- outputColumnNames: p_mfgr, p_retailprice
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(p_retailprice)
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: double)
- Reduce Operator Tree:
- Group By Operator
- aggregations: min(VALUE._col0)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: double)
- outputColumnNames: _col1
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: double)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ expressions: _col0 (type: string), _col1 (type: double), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -713,66 +496,6 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-14
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: double)
- sort order: +
- Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: double)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-15
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_mfgr (type: string)
- outputColumnNames: p_mfgr
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-0
Fetch Operator
limit: -1
@@ -1336,17 +1059,12 @@ POSTHOOK: query: explain SELECT c1 FROM t1 group by c1 having c1 NOT IN (SELECT
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1, Stage-5
- Stage-3 depends on stages: Stage-2, Stage-9
+ Stage-2 depends on stages: Stage-1, Stage-3, Stage-5
+ Stage-3 is a root stage
+ Stage-4 is a root stage
+ Stage-5 depends on stages: Stage-4, Stage-6
Stage-6 is a root stage
- Stage-4 depends on stages: Stage-6
- Stage-5 depends on stages: Stage-4
- Stage-10 is a root stage
- Stage-7 depends on stages: Stage-10
- Stage-8 depends on stages: Stage-7
- Stage-9 depends on stages: Stage-8, Stage-11
- Stage-11 is a root stage
- Stage-0 depends on stages: Stage-3
+ Stage-0 depends on stages: Stage-2
STAGE PLANS:
Stage: Stage-1
@@ -1398,154 +1116,60 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 2 Data size: 171 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int), _col0 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col0 (type: int)
- Statistics: Num rows: 2 Data size: 171 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: bigint), _col3 (type: bigint)
TableScan
Reduce Output Operator
- key expressions: _col3 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col3 (type: int), _col1 (type: int)
+ key expressions: _col3 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col3 (type: int)
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: boolean)
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
+ Left Outer Join0 to 2
keys:
- 0 _col0 (type: int), _col0 (type: int)
- 1 _col3 (type: int), _col1 (type: int)
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ 2 _col3 (type: int)
outputColumnNames: _col0, _col2, _col3, _col6
- Statistics: Num rows: 2 Data size: 188 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 343 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: CASE WHEN ((_col2 = 0)) THEN (true) WHEN (_col2 is null) THEN (true) WHEN (_col6 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col3 < _col2)) THEN (false) ELSE (true) END (type: boolean)
- Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: NONE
+ predicate: (not CASE WHEN ((_col2 = 0)) THEN (false) WHEN (_col2 is null) THEN (false) WHEN (_col6 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col3 < _col2)) THEN (true) ELSE (false) END) (type: boolean)
+ Statistics: Num rows: 2 Data size: 171 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: int)
outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 171 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 94 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 171 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-6
+ Stage: Stage-3
Map Reduce
Map Operator Tree:
TableScan
- alias: t1
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: c1 (type: int)
- outputColumnNames: c1
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
+ alias: t2
+ Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (c1 = c1) (type: boolean)
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Group By Operator
+ aggregations: count(), count(c1)
keys: c1 (type: int)
mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 2 Data size: 156 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 78 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: t2
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: c1 (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 78 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col0)
- keys: _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-5
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(VALUE._col1)
@@ -1560,104 +1184,34 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-10
+ Stage: Stage-4
Map Reduce
Map Operator Tree:
TableScan
- alias: t1
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: c1 (type: int)
- outputColumnNames: c1
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
+ alias: t2
+ Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (c1 = c1) (type: boolean)
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: c1 (type: int)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 4 Data size: 313 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 2 Data size: 156 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int)
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 78 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-7
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: t2
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: c1 (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 78 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-8
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 3 Data size: 6 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int), KEY._col1 (type: int)
- mode: mergepartial
- outputColumnNames: _col0, _col1
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: int), _col1 (type: int), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
+ expressions: _col0 (type: int), true (type: boolean)
+ outputColumnNames: _col0, _col2
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
@@ -1666,7 +1220,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-9
+ Stage: Stage-5
Map Reduce
Map Operator Tree:
TableScan
@@ -1675,7 +1229,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int), _col2 (type: boolean)
+ value expressions: _col2 (type: boolean)
TableScan
Reduce Output Operator
key expressions: _col0 (type: int)
@@ -1689,7 +1243,7 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
+ outputColumnNames: _col2, _col3
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
@@ -1698,7 +1252,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-11
+ Stage: Stage-6
Map Reduce
Map Operator Tree:
TableScan
[07/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/perf/query6.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/query6.q.out b/ql/src/test/results/clientpositive/perf/query6.q.out
index 094459e..ca17206 100644
--- a/ql/src/test/results/clientpositive/perf/query6.q.out
+++ b/ql/src/test/results/clientpositive/perf/query6.q.out
@@ -1,7 +1,4 @@
-Warning: Shuffle Join MERGEJOIN[196][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
-Warning: Shuffle Join MERGEJOIN[193][tables = [$hdt$_5, $hdt$_6]] in Stage 'Reducer 21' is a cross product
-Warning: Shuffle Join MERGEJOIN[194][tables = [$hdt$_5, $hdt$_6, $hdt$_7]] in Stage 'Reducer 22' is a cross product
-Warning: Shuffle Join MERGEJOIN[195][tables = [$hdt$_5, $hdt$_6, $hdt$_7, $hdt$_8, $hdt$_9, $hdt$_10, $hdt$_11]] in Stage 'Reducer 23' is a cross product
+Warning: Shuffle Join MERGEJOIN[111][tables = [$hdt$_5, $hdt$_6]] in Stage 'Reducer 16' is a cross product
PREHOOK: query: explain select a.ca_state state, count(*) cnt
from customer_address a
,customer c
@@ -53,24 +50,16 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 10 <- Reducer 9 (CUSTOM_SIMPLE_EDGE)
-Reducer 12 <- Map 11 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
-Reducer 14 <- Map 13 (SIMPLE_EDGE)
-Reducer 16 <- Map 15 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
-Reducer 18 <- Map 17 (SIMPLE_EDGE), Reducer 24 (SIMPLE_EDGE)
-Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE), Reducer 10 (CUSTOM_SIMPLE_EDGE)
-Reducer 21 <- Map 20 (CUSTOM_SIMPLE_EDGE), Reducer 27 (CUSTOM_SIMPLE_EDGE)
-Reducer 22 <- Reducer 21 (CUSTOM_SIMPLE_EDGE), Reducer 29 (CUSTOM_SIMPLE_EDGE)
-Reducer 23 <- Map 30 (CUSTOM_SIMPLE_EDGE), Map 31 (CUSTOM_SIMPLE_EDGE), Map 32 (CUSTOM_SIMPLE_EDGE), Map 33 (CUSTOM_SIMPLE_EDGE), Reducer 22 (CUSTOM_SIMPLE_EDGE)
-Reducer 24 <- Reducer 23 (SIMPLE_EDGE)
-Reducer 26 <- Map 25 (SIMPLE_EDGE)
-Reducer 27 <- Reducer 26 (CUSTOM_SIMPLE_EDGE)
-Reducer 29 <- Map 28 (SIMPLE_EDGE)
-Reducer 3 <- Reducer 12 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 35 <- Map 34 (SIMPLE_EDGE), Map 36 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 16 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 35 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 13 (SIMPLE_EDGE)
+Reducer 15 <- Map 14 (SIMPLE_EDGE)
+Reducer 16 <- Reducer 15 (CUSTOM_SIMPLE_EDGE), Reducer 20 (CUSTOM_SIMPLE_EDGE)
+Reducer 17 <- Map 21 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
+Reducer 19 <- Map 18 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 20 <- Reducer 19 (CUSTOM_SIMPLE_EDGE)
+Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 12 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 17 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
Reducer 9 <- Map 8 (SIMPLE_EDGE)
@@ -80,253 +69,163 @@ Stage-0
limit:100
Stage-1
Reducer 7
- File Output Operator [FS_129]
- Limit [LIM_128] (rows=100 width=1)
+ File Output Operator [FS_77]
+ Limit [LIM_76] (rows=100 width=88)
Number of rows:100
- Select Operator [SEL_127] (rows=187554724582209678 width=1)
+ Select Operator [SEL_75] (rows=42591679 width=88)
Output:["_col0","_col1"]
<-Reducer 6 [SIMPLE_EDGE]
- SHUFFLE [RS_126]
- Filter Operator [FIL_124] (rows=187554724582209678 width=1)
+ SHUFFLE [RS_74]
+ Filter Operator [FIL_72] (rows=42591679 width=88)
predicate:(_col1 >= 10)
- Group By Operator [GBY_123] (rows=562664173746629034 width=1)
+ Group By Operator [GBY_71] (rows=127775039 width=88)
Output:["_col0","_col1"],aggregations:["count(VALUE._col0)"],keys:KEY._col0
<-Reducer 5 [SIMPLE_EDGE]
- SHUFFLE [RS_122]
+ SHUFFLE [RS_70]
PartitionCols:_col0
- Group By Operator [GBY_121] (rows=1125328347493258069 width=1)
+ Group By Operator [GBY_69] (rows=255550079 width=88)
Output:["_col0","_col1"],aggregations:["count()"],keys:_col1
- Select Operator [SEL_120] (rows=1125328347493258069 width=1)
+ Select Operator [SEL_68] (rows=255550079 width=88)
Output:["_col1"]
- Filter Operator [FIL_119] (rows=1125328347493258069 width=1)
+ Filter Operator [FIL_67] (rows=255550079 width=88)
predicate:(_col10 > (1.2 * CASE WHEN (_col15 is null) THEN (null) ELSE (_col14) END))
- Select Operator [SEL_118] (rows=3375985042479774208 width=1)
+ Select Operator [SEL_66] (rows=766650239 width=88)
Output:["_col1","_col10","_col14","_col15"]
- Merge Join Operator [MERGEJOIN_203] (rows=3375985042479774208 width=1)
- Conds:RS_115._col2=RS_116._col0(Inner),Output:["_col8","_col10","_col11","_col16"]
- <-Reducer 35 [SIMPLE_EDGE]
- SHUFFLE [RS_116]
+ Merge Join Operator [MERGEJOIN_114] (rows=766650239 width=88)
+ Conds:RS_63._col6=RS_64._col0(Inner),Output:["_col1","_col3","_col4","_col12"]
+ <-Reducer 17 [SIMPLE_EDGE]
+ SHUFFLE [RS_64]
PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_200] (rows=88000001 width=860)
- Conds:RS_102._col1=RS_103._col0(Inner),Output:["_col0","_col3"]
- <-Map 34 [SIMPLE_EDGE]
- SHUFFLE [RS_102]
- PartitionCols:_col1
- Select Operator [SEL_98] (rows=80000000 width=860)
- Output:["_col0","_col1"]
- Filter Operator [FIL_191] (rows=80000000 width=860)
- predicate:(c_current_addr_sk is not null and c_customer_sk is not null)
- TableScan [TS_96] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_current_addr_sk"]
- <-Map 36 [SIMPLE_EDGE]
- SHUFFLE [RS_103]
- PartitionCols:_col0
- Select Operator [SEL_101] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_192] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_99] (rows=40000000 width=1014)
- default@customer_address,a,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_115]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_202] (rows=3069077244824791552 width=1)
- Conds:RS_112._col1=RS_113._col0(Inner),Output:["_col2","_col8","_col10","_col11"]
- <-Reducer 16 [SIMPLE_EDGE]
- SHUFFLE [RS_113]
- PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_199] (rows=2790070162094850048 width=1)
- Conds:RS_92._col2=RS_93._col2(Left Outer),Output:["_col0","_col1","_col3","_col4"]
- <-Map 15 [SIMPLE_EDGE]
- SHUFFLE [RS_92]
- PartitionCols:_col2
- Select Operator [SEL_33] (rows=462000 width=1436)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_186] (rows=462000 width=1436)
- predicate:i_item_sk is not null
- TableScan [TS_31] (rows=462000 width=1436)
- default@item,i,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_current_price","i_category"]
- <-Reducer 19 [SIMPLE_EDGE]
- SHUFFLE [RS_93]
- PartitionCols:_col2
- Select Operator [SEL_91] (rows=2536427365110644736 width=1)
- Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_90] (rows=2536427365110644736 width=1)
- Output:["_col0","_col1"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0
- <-Reducer 18 [SIMPLE_EDGE]
- SHUFFLE [RS_89]
+ Select Operator [SEL_53] (rows=80353 width=1119)
+ Output:["_col0"]
+ Merge Join Operator [MERGEJOIN_112] (rows=80353 width=1119)
+ Conds:RS_50._col0=RS_51._col1(Inner),Output:["_col2"]
+ <-Map 21 [SIMPLE_EDGE]
+ SHUFFLE [RS_51]
+ PartitionCols:_col1
+ Select Operator [SEL_46] (rows=73049 width=1119)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_107] (rows=73049 width=1119)
+ predicate:(d_date_sk is not null and d_month_seq is not null)
+ TableScan [TS_44] (rows=73049 width=1119)
+ default@date_dim,d,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_month_seq"]
+ <-Reducer 16 [SIMPLE_EDGE]
+ SHUFFLE [RS_50]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_111] (rows=9131 width=1128)
+ Conds:(Inner),Output:["_col0"]
+ <-Reducer 15 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_47]
+ Group By Operator [GBY_28] (rows=9131 width=1119)
+ Output:["_col0"],keys:KEY._col0
+ <-Map 14 [SIMPLE_EDGE]
+ SHUFFLE [RS_27]
PartitionCols:_col0
- Group By Operator [GBY_88] (rows=5072854730221289472 width=1)
- Output:["_col0","_col1"],aggregations:["avg(_col0)"],keys:_col2
- Select Operator [SEL_87] (rows=5072854730221289472 width=1)
- Output:["_col2","_col0"]
- Merge Join Operator [MERGEJOIN_198] (rows=5072854730221289472 width=1)
- Conds:RS_84._col1=RS_85._col0(Inner),Output:["_col0","_col2"]
- <-Map 17 [SIMPLE_EDGE]
- SHUFFLE [RS_84]
- PartitionCols:_col1
- Select Operator [SEL_35] (rows=462000 width=1436)
- Output:["_col0","_col1"]
- TableScan [TS_34] (rows=462000 width=1436)
- default@item,j,Tbl:COMPLETE,Col:NONE,Output:["i_current_price","i_category"]
- <-Reducer 24 [SIMPLE_EDGE]
- SHUFFLE [RS_85]
- PartitionCols:_col0
- Group By Operator [GBY_82] (rows=4611686018427387903 width=1)
+ Group By Operator [GBY_26] (rows=18262 width=1119)
+ Output:["_col0"],keys:d_month_seq
+ Select Operator [SEL_25] (rows=18262 width=1119)
+ Output:["d_month_seq"]
+ Filter Operator [FIL_105] (rows=18262 width=1119)
+ predicate:((d_year = 2000) and (d_moy = 2) and d_month_seq is not null)
+ TableScan [TS_23] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_month_seq","d_year","d_moy"]
+ <-Reducer 20 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_48]
+ Select Operator [SEL_43] (rows=1 width=8)
+ Filter Operator [FIL_42] (rows=1 width=8)
+ predicate:(sq_count_check(_col0) <= 1)
+ Group By Operator [GBY_40] (rows=1 width=8)
+ Output:["_col0"],aggregations:["count(VALUE._col0)"]
+ <-Reducer 19 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_39]
+ Group By Operator [GBY_38] (rows=1 width=8)
+ Output:["_col0"],aggregations:["count()"]
+ Select Operator [SEL_36] (rows=9131 width=1119)
+ Group By Operator [GBY_35] (rows=9131 width=1119)
Output:["_col0"],keys:KEY._col0
- <-Reducer 23 [SIMPLE_EDGE]
- SHUFFLE [RS_81]
+ <-Map 18 [SIMPLE_EDGE]
+ SHUFFLE [RS_34]
PartitionCols:_col0
- Group By Operator [GBY_80] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col3
- Merge Join Operator [MERGEJOIN_195] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),(Inner),(Inner),Output:["_col3"]
- <-Map 30 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_74]
- Select Operator [SEL_60] (rows=462000 width=1436)
- Output:["_col0"]
- TableScan [TS_59] (rows=462000 width=1436)
- default@item,i,Tbl:COMPLETE,Col:NONE,Output:["i_category"]
- <-Map 31 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_75]
- Select Operator [SEL_62] (rows=40000000 width=4)
- TableScan [TS_61] (rows=40000000 width=1014)
- default@customer_address,a,Tbl:COMPLETE,Col:COMPLETE
- <-Map 32 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_76]
- Select Operator [SEL_64] (rows=80000000 width=4)
- TableScan [TS_63] (rows=80000000 width=860)
- default@customer,c,Tbl:COMPLETE,Col:COMPLETE
- <-Map 33 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_77]
- Select Operator [SEL_66] (rows=575995635 width=4)
- TableScan [TS_65] (rows=575995635 width=88)
- default@store_sales,s,Tbl:COMPLETE,Col:COMPLETE
- <-Reducer 22 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_73]
- Merge Join Operator [MERGEJOIN_194] (rows=667010419 width=1133)
- Conds:(Left Outer)
- <-Reducer 21 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_70]
- Merge Join Operator [MERGEJOIN_193] (rows=73049 width=13)
- Conds:(Inner)
- <-Map 20 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_67]
- Select Operator [SEL_37] (rows=73049 width=4)
- TableScan [TS_36] (rows=73049 width=1119)
- default@date_dim,d,Tbl:COMPLETE,Col:COMPLETE
- <-Reducer 27 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_68]
- Select Operator [SEL_51] (rows=1 width=8)
- Filter Operator [FIL_50] (rows=1 width=8)
- predicate:(sq_count_check(_col0) <= 1)
- Group By Operator [GBY_48] (rows=1 width=8)
- Output:["_col0"],aggregations:["count(VALUE._col0)"]
- <-Reducer 26 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_47]
- Group By Operator [GBY_46] (rows=1 width=8)
- Output:["_col0"],aggregations:["count()"]
- Select Operator [SEL_44] (rows=9131 width=1119)
- Group By Operator [GBY_43] (rows=9131 width=1119)
- Output:["_col0"],keys:KEY._col0
- <-Map 25 [SIMPLE_EDGE]
- SHUFFLE [RS_42]
- PartitionCols:_col0
- Group By Operator [GBY_41] (rows=18262 width=1119)
- Output:["_col0"],keys:d_month_seq
- Select Operator [SEL_40] (rows=18262 width=1119)
- Output:["d_month_seq"]
- Filter Operator [FIL_188] (rows=18262 width=1119)
- predicate:((d_year = 2000) and (d_moy = 2))
- TableScan [TS_38] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_month_seq","d_year","d_moy"]
- <-Reducer 29 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_71]
- Select Operator [SEL_58] (rows=9131 width=1119)
- Group By Operator [GBY_57] (rows=9131 width=1119)
- Output:["_col0"],keys:KEY._col0
- <-Map 28 [SIMPLE_EDGE]
- SHUFFLE [RS_56]
- PartitionCols:_col0
- Group By Operator [GBY_55] (rows=18262 width=1119)
- Output:["_col0"],keys:d_month_seq
- Select Operator [SEL_54] (rows=18262 width=1119)
- Output:["d_month_seq"]
- Filter Operator [FIL_189] (rows=18262 width=1119)
- predicate:((d_year = 2000) and (d_moy = 2))
- TableScan [TS_52] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_month_seq","d_year","d_moy"]
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_112]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_201] (rows=633595212 width=97)
- Conds:RS_109._col0=RS_110._col0(Inner),Output:["_col1","_col2"]
- <-Reducer 12 [SIMPLE_EDGE]
- SHUFFLE [RS_110]
+ Group By Operator [GBY_33] (rows=18262 width=1119)
+ Output:["_col0"],keys:d_month_seq
+ Select Operator [SEL_32] (rows=18262 width=1119)
+ Output:["d_month_seq"]
+ Filter Operator [FIL_106] (rows=18262 width=1119)
+ predicate:((d_year = 2000) and (d_moy = 2))
+ TableScan [TS_30] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_month_seq","d_year","d_moy"]
+ <-Reducer 4 [SIMPLE_EDGE]
+ SHUFFLE [RS_63]
+ PartitionCols:_col6
+ Merge Join Operator [MERGEJOIN_113] (rows=696954748 width=88)
+ Conds:RS_60._col8=RS_61._col0(Inner),Output:["_col1","_col3","_col4","_col6","_col12"]
+ <-Reducer 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_61]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_110] (rows=88000001 width=860)
+ Conds:RS_19._col1=RS_20._col0(Inner),Output:["_col0","_col3"]
+ <-Map 11 [SIMPLE_EDGE]
+ SHUFFLE [RS_19]
+ PartitionCols:_col1
+ Select Operator [SEL_15] (rows=80000000 width=860)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_103] (rows=80000000 width=860)
+ predicate:(c_current_addr_sk is not null and c_customer_sk is not null)
+ TableScan [TS_13] (rows=80000000 width=860)
+ default@customer,c,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_current_addr_sk"]
+ <-Map 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_20]
PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_197] (rows=80353 width=1119)
- Conds:RS_27._col1=RS_28._col0(Inner),Output:["_col0"]
- <-Map 11 [SIMPLE_EDGE]
- SHUFFLE [RS_27]
- PartitionCols:_col1
- Select Operator [SEL_19] (rows=73049 width=1119)
- Output:["_col0","_col1"]
- Filter Operator [FIL_184] (rows=73049 width=1119)
- predicate:(d_date_sk is not null and d_month_seq is not null)
- TableScan [TS_17] (rows=73049 width=1119)
- default@date_dim,d,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_month_seq"]
- <-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_28]
- PartitionCols:_col0
- Group By Operator [GBY_25] (rows=9131 width=1119)
- Output:["_col0"],keys:KEY._col0
- <-Map 13 [SIMPLE_EDGE]
- SHUFFLE [RS_24]
- PartitionCols:_col0
- Group By Operator [GBY_23] (rows=18262 width=1119)
- Output:["_col0"],keys:d_month_seq
- Select Operator [SEL_22] (rows=18262 width=1119)
- Output:["d_month_seq"]
- Filter Operator [FIL_185] (rows=18262 width=1119)
- predicate:((d_year = 2000) and (d_moy = 2) and d_month_seq is not null)
- TableScan [TS_20] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_month_seq","d_year","d_moy"]
+ Select Operator [SEL_18] (rows=40000000 width=1014)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_104] (rows=40000000 width=1014)
+ predicate:ca_address_sk is not null
+ TableScan [TS_16] (rows=40000000 width=1014)
+ default@customer_address,a,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_60]
+ PartitionCols:_col8
+ Merge Join Operator [MERGEJOIN_109] (rows=633595212 width=88)
+ Conds:RS_57._col0=RS_58._col1(Inner),Output:["_col1","_col3","_col4","_col6","_col8"]
+ <-Map 10 [SIMPLE_EDGE]
+ SHUFFLE [RS_58]
+ PartitionCols:_col1
+ Select Operator [SEL_12] (rows=575995635 width=88)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_102] (rows=575995635 width=88)
+ predicate:(ss_customer_sk is not null and ss_sold_date_sk is not null and ss_item_sk is not null)
+ TableScan [TS_10] (rows=575995635 width=88)
+ default@store_sales,s,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_item_sk","ss_customer_sk"]
<-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_109]
+ SHUFFLE [RS_57]
PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_196] (rows=575995635 width=97)
- Conds:(Inner),Output:["_col0","_col1","_col2"]
- <-Map 1 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_106]
- Select Operator [SEL_2] (rows=575995635 width=88)
+ Merge Join Operator [MERGEJOIN_108] (rows=508200 width=1436)
+ Conds:RS_54._col2=RS_55._col2(Left Outer),Output:["_col0","_col1","_col3","_col4"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_54]
+ PartitionCols:_col2
+ Select Operator [SEL_2] (rows=462000 width=1436)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_100] (rows=462000 width=1436)
+ predicate:i_item_sk is not null
+ TableScan [TS_0] (rows=462000 width=1436)
+ default@item,i,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_current_price","i_category"]
+ <-Reducer 9 [SIMPLE_EDGE]
+ SHUFFLE [RS_55]
+ PartitionCols:_col2
+ Select Operator [SEL_9] (rows=115500 width=1436)
Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_182] (rows=575995635 width=88)
- predicate:(ss_customer_sk is not null and ss_sold_date_sk is not null and ss_item_sk is not null)
- TableScan [TS_0] (rows=575995635 width=88)
- default@store_sales,s,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_item_sk","ss_customer_sk"]
- <-Reducer 10 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_107]
- Select Operator [SEL_16] (rows=1 width=8)
- Filter Operator [FIL_15] (rows=1 width=8)
- predicate:(sq_count_check(_col0) <= 1)
- Group By Operator [GBY_13] (rows=1 width=8)
- Output:["_col0"],aggregations:["count(VALUE._col0)"]
- <-Reducer 9 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_12]
- Group By Operator [GBY_11] (rows=1 width=8)
- Output:["_col0"],aggregations:["count()"]
- Select Operator [SEL_9] (rows=9131 width=1119)
- Group By Operator [GBY_8] (rows=9131 width=1119)
- Output:["_col0"],keys:KEY._col0
- <-Map 8 [SIMPLE_EDGE]
- SHUFFLE [RS_7]
- PartitionCols:_col0
- Group By Operator [GBY_6] (rows=18262 width=1119)
- Output:["_col0"],keys:d_month_seq
- Select Operator [SEL_5] (rows=18262 width=1119)
- Output:["d_month_seq"]
- Filter Operator [FIL_183] (rows=18262 width=1119)
- predicate:((d_year = 2000) and (d_moy = 2))
- TableScan [TS_3] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_month_seq","d_year","d_moy"]
+ Group By Operator [GBY_8] (rows=115500 width=1436)
+ Output:["_col0","_col1"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0
+ <-Map 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_7]
+ PartitionCols:_col0
+ Group By Operator [GBY_6] (rows=231000 width=1436)
+ Output:["_col0","_col1"],aggregations:["avg(i_current_price)"],keys:i_category
+ Select Operator [SEL_5] (rows=231000 width=1436)
+ Output:["i_category","i_current_price"]
+ Filter Operator [FIL_101] (rows=231000 width=1436)
+ predicate:(i_category = i_category)
+ TableScan [TS_3] (rows=462000 width=1436)
+ default@item,j,Tbl:COMPLETE,Col:NONE,Output:["i_current_price","i_category"]
[11/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_notin.q.out b/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
index 8530a35..f1efb64 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_notin.q.out
@@ -324,18 +324,12 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
- Reducer 14 <- Reducer 13 (SIMPLE_EDGE), Reducer 18 (SIMPLE_EDGE)
- Reducer 16 <- Map 15 (SIMPLE_EDGE)
- Reducer 18 <- Map 17 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 14 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
Reducer 5 <- Map 4 (SIMPLE_EDGE)
- Reducer 6 <- Reducer 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -355,208 +349,40 @@ STAGE PLANS:
value expressions: _col0 (type: string), _col2 (type: int)
Execution mode: llap
LLAP IO: no inputs
- Map 10
+ Map 4
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 5798 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 5798 Basic stats: COMPLETE Column stats: COMPLETE
- TopN Hash Memory Usage: 0.1
- value expressions: p_name (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 15
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 17
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2899 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 2899 Basic stats: COMPLETE Column stats: COMPLETE
+ TopN Hash Memory Usage: 0.1
+ value expressions: p_name (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 4
+ Map 7
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 5798 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 5798 Basic stats: COMPLETE Column stats: COMPLETE
- TopN Hash Memory Usage: 0.1
- value expressions: p_name (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 2899 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 2899 Basic stats: COMPLETE Column stats: COMPLETE
+ TopN Hash Memory Usage: 0.1
+ value expressions: p_name (type: string)
Execution mode: llap
LLAP IO: no inputs
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: VALUE._col1 (type: string), KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
- outputColumnNames: _col1, _col2, _col5
- Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
- PTF Operator
- Function definitions:
- Input definition
- input alias: ptf_0
- output shape: _col1: string, _col2: string, _col5: int
- type: WINDOWING
- Windowing table definition
- input alias: ptf_1
- name: windowingtablefunction
- order by: _col5 ASC NULLS FIRST
- partition by: _col2
- raw input shape:
- window functions:
- window function definition
- alias: rank_window_0
- arguments: _col5
- name: rank
- window function: GenericUDAFRankEvaluator
- window frame: PRECEDING(MAX)~FOLLOWING(MAX)
- isPivotResult: true
- Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 3928 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col1 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 4 Data size: 876 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 4 Data size: 876 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 13
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 4 Data size: 876 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 4 Data size: 892 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 4 Data size: 892 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: boolean)
- Reducer 14
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 2 Data size: 446 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col3 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col3 (type: string), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 446 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: boolean)
- Reducer 16
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 18
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -582,11 +408,11 @@ STAGE PLANS:
Left Outer Join0 to 1
keys:
0 _col0 (type: string), _col1 (type: string)
- 1 _col3 (type: string), _col1 (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1, _col2, _col4, _col5, _col8
Statistics: Num rows: 26 Data size: 6318 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col8 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END (type: boolean)
+ predicate: (not CASE WHEN ((_col4 = 0)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (true) ELSE (false) END) (type: boolean)
Statistics: Num rows: 13 Data size: 3159 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col1 (type: string), _col0 (type: string), _col2 (type: int)
@@ -605,7 +431,7 @@ STAGE PLANS:
Select Operator
expressions: VALUE._col1 (type: string), KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col1, _col2, _col5
- Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 6383 Basic stats: COMPLETE Column stats: COMPLETE
PTF Operator
Function definitions:
Input definition
@@ -626,75 +452,107 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 12766 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 6383 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 3928 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 4 Data size: 1964 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
+ outputColumnNames: _col1, _col0
+ Statistics: Num rows: 4 Data size: 1964 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(_col0)
+ keys: _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reducer 6
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 8 Data size: 1752 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(), count(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(VALUE._col1)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ Reducer 8
+ Execution mode: llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: VALUE._col1 (type: string), KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
+ outputColumnNames: _col1, _col2, _col5
+ Statistics: Num rows: 13 Data size: 6383 Basic stats: COMPLETE Column stats: COMPLETE
+ PTF Operator
+ Function definitions:
+ Input definition
+ input alias: ptf_0
+ output shape: _col1: string, _col2: string, _col5: int
+ type: WINDOWING
+ Windowing table definition
+ input alias: ptf_1
+ name: windowingtablefunction
+ order by: _col5 ASC NULLS FIRST
+ partition by: _col2
+ raw input shape:
+ window functions:
+ window function definition
+ alias: rank_window_0
+ arguments: _col5
+ name: rank
+ window function: GenericUDAFRankEvaluator
+ window frame: PRECEDING(MAX)~FOLLOWING(MAX)
+ isPivotResult: true
+ Statistics: Num rows: 13 Data size: 6383 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (rank_window_0 <= 2) (type: boolean)
+ Statistics: Num rows: 4 Data size: 1964 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: string), _col2 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 4 Data size: 1964 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 438 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 2 Data size: 438 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 9
Execution mode: llap
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 438 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 1 Data size: 219 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 223 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 1 Data size: 223 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: boolean)
Stage: Stage-0
Fetch Operator
@@ -1080,25 +938,17 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 10 <- Map 9 (SIMPLE_EDGE)
+ Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
Reducer 12 <- Map 11 (SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
+ Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
Reducer 14 <- Reducer 13 (SIMPLE_EDGE)
- Reducer 16 <- Map 15 (SIMPLE_EDGE)
- Reducer 18 <- Map 17 (SIMPLE_EDGE)
- Reducer 19 <- Reducer 18 (SIMPLE_EDGE), Reducer 24 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 20 <- Reducer 19 (SIMPLE_EDGE)
- Reducer 21 <- Reducer 20 (SIMPLE_EDGE)
- Reducer 22 <- Reducer 21 (SIMPLE_EDGE), Reducer 26 (SIMPLE_EDGE)
- Reducer 24 <- Map 23 (SIMPLE_EDGE)
- Reducer 26 <- Map 25 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 14 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
- Reducer 4 <- Reducer 22 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 14 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
Reducer 6 <- Map 5 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 10 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
-#### A masked pattern was here ####
+ Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (SIMPLE_EDGE)
+#### A masked pattern was here ####
Vertices:
Map 1
Map Operator Tree:
@@ -1122,128 +972,77 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- TopN Hash Memory Usage: 0.1
- Execution mode: llap
- LLAP IO: no inputs
- Map 15
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
+ TopN Hash Memory Usage: 0.1
Execution mode: llap
LLAP IO: no inputs
- Map 17
+ Map 5
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- TopN Hash Memory Usage: 0.1
- Execution mode: llap
- LLAP IO: no inputs
- Map 23
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 25
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_size (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
+ TopN Hash Memory Usage: 0.1
Execution mode: llap
LLAP IO: no inputs
- Map 5
+ Map 8
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 2652 Basic stats: COMPLETE Column stats: COMPLETE
- TopN Hash Memory Usage: 0.1
- Execution mode: llap
- LLAP IO: no inputs
- Map 9
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 2548 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1326 Basic stats: COMPLETE Column stats: COMPLETE
+ TopN Hash Memory Usage: 0.1
Execution mode: llap
LLAP IO: no inputs
Reducer 10
Execution mode: llap
Reduce Operator Tree:
Group By Operator
+ aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(_col1)
+ keys: _col0 (type: string)
+ mode: complete
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
Reducer 12
Execution mode: llap
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col2, _col5
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
PTF Operator
Function definitions:
Input definition
@@ -1264,149 +1063,70 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 2960 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string), _col5 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: min(_col1)
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: int)
Reducer 13
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: min(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
- Reducer 14
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- aggregations: count(), count(_col1)
- keys: _col0 (type: string)
- mode: complete
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: COMPLETE
+ keys: _col0 (type: string), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 228 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 16
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 14
Execution mode: llap
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 18
- Execution mode: llap
- Reduce Operator Tree:
- Select Operator
- expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
- outputColumnNames: _col2, _col5
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
- PTF Operator
- Function definitions:
- Input definition
- input alias: ptf_0
- output shape: _col2: string, _col5: int
- type: WINDOWING
- Windowing table definition
- input alias: ptf_1
- name: windowingtablefunction
- order by: _col5 ASC NULLS FIRST
- partition by: _col2
- raw input shape:
- window functions:
- window function definition
- alias: rank_window_0
- arguments: _col5
- name: rank
- window function: GenericUDAFRankEvaluator
- window frame: PRECEDING(MAX)~FOLLOWING(MAX)
- isPivotResult: true
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 2960 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col2 (type: string), _col5 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: _col0 (type: int), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
- Reducer 19
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 8 Data size: 816 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: min(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
+ key expressions: _col1 (type: string), _col0 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: boolean)
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -1431,92 +1151,12 @@ STAGE PLANS:
Map-reduce partition columns: _col1 (type: string)
Statistics: Num rows: 8 Data size: 1848 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: string), _col2 (type: int)
- Reducer 20
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: min(VALUE._col0)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 204 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col1 (type: int), _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 21
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: boolean)
- Reducer 22
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col3 (type: int)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col3 (type: int)
- Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: boolean)
- Reducer 24
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 5 Data size: 490 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 26
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
+ Left Outer Join0 to 1
keys:
0 _col1 (type: string)
1 _col0 (type: string)
@@ -1536,11 +1176,11 @@ STAGE PLANS:
Left Outer Join0 to 1
keys:
0 _col1 (type: string), _col2 (type: int)
- 1 _col1 (type: string), _col3 (type: int)
+ 1 _col1 (type: string), _col0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col6, _col7, _col10
Statistics: Num rows: 8 Data size: 1944 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: CASE WHEN ((_col6 = 0)) THEN (true) WHEN (_col6 is null) THEN (true) WHEN (_col10 is not null) THEN (false) WHEN (_col2 is null) THEN (null) WHEN ((_col7 < _col6)) THEN (false) ELSE (true) END (type: boolean)
+ predicate: (not CASE WHEN ((_col6 = 0)) THEN (false) WHEN (_col6 is null) THEN (false) WHEN (_col10 is not null) THEN (true) WHEN (_col2 is null) THEN (null) WHEN ((_col7 < _col6)) THEN (true) ELSE (false) END) (type: boolean)
Statistics: Num rows: 4 Data size: 972 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col1 (type: string), _col0 (type: string), _col2 (type: int)
@@ -1559,7 +1199,7 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col2, _col5
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
PTF Operator
Function definitions:
Input definition
@@ -1580,60 +1220,91 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 9620 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 2960 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 8 Data size: 784 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 784 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 98 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 98 Basic stats: COMPLETE Column stats: COMPLETE
Reducer 7
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col2
- Statistics: Num rows: 8 Data size: 784 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 2 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 8
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 2 Data size: 196 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 98 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
aggregations: count()
keys: _col0 (type: string)
mode: complete
outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 212 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 2 Data size: 212 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 106 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: bigint)
+ Reducer 9
+ Execution mode: llap
+ Reduce Operator Tree:
+ Select Operator
+ expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
+ outputColumnNames: _col2, _col5
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
+ PTF Operator
+ Function definitions:
+ Input definition
+ input alias: ptf_0
+ output shape: _col2: string, _col5: int
+ type: WINDOWING
+ Windowing table definition
+ input alias: ptf_1
+ name: windowingtablefunction
+ order by: _col5 ASC NULLS FIRST
+ partition by: _col2
+ raw input shape:
+ window functions:
+ window function definition
+ alias: rank_window_0
+ arguments: _col5
+ name: rank
+ window function: GenericUDAFRankEvaluator
+ window frame: PRECEDING(MAX)~FOLLOWING(MAX)
+ isPivotResult: true
+ Statistics: Num rows: 13 Data size: 4810 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (rank_window_0 <= 2) (type: boolean)
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col2 (type: string), _col5 (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 4 Data size: 1480 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: min(_col1)
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 102 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: int)
Stage: Stage-0
Fetch Operator
@@ -2076,16 +1747,10 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 10 <- Map 9 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
- Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE), Reducer 16 (SIMPLE_EDGE)
- Reducer 14 <- Map 13 (SIMPLE_EDGE)
- Reducer 16 <- Map 15 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 12 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
- Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+ Reducer 5 <- Map 4 (SIMPLE_EDGE)
+ Reducer 7 <- Map 6 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2108,203 +1773,82 @@ STAGE PLANS:
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 13
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 15
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_size (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
Map 4
Map Operator Tree:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 7
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_type (type: string), (p_size * p_size) (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(_col1)
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 6 Data size: 720 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 6 Data size: 720 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
- Map 9
+ Map 6
Map Operator Tree:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: (p_size * p_size) (type: int), p_type (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: _col1 (type: string), _col0 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Reducer 10
+ Reducer 2
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join0 to 1
keys:
- 0 _col0 (type: string)
+ 0 _col4 (type: string)
1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: (_col1 * _col1) (type: int), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
- Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 6 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 6 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: boolean)
- Reducer 12
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col10, _col11
+ Statistics: Num rows: 26 Data size: 16510 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col4 (type: string), _col5 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col4 (type: string), _col5 (type: int)
+ Statistics: Num rows: 26 Data size: 16510 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col6 (type: string), _col7 (type: double), _col8 (type: string), _col10 (type: bigint), _col11 (type: bigint)
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 4 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string), _col3 (type: int)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col3 (type: int)
- Statistics: Num rows: 4 Data size: 448 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: boolean)
- Reducer 14
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 16
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 2
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col4 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col10, _col11
- Statistics: Num rows: 26 Data size: 16510 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col4 (type: string), _col5 (type: int)
- sort order: ++
- Map-reduce partition columns: _col4 (type: string), _col5 (type: int)
- Statistics: Num rows: 26 Data size: 16510 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col6 (type: string), _col7 (type: double), _col8 (type: string), _col10 (type: bigint), _col11 (type: bigint)
- Reducer 3
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
+ Left Outer Join0 to 1
keys:
0 _col4 (type: string), _col5 (type: int)
- 1 _col1 (type: string), _col3 (type: int)
+ 1 _col1 (type: string), _col0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col10, _col11, _col14
Statistics: Num rows: 26 Data size: 16614 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
@@ -2324,33 +1868,6 @@ STAGE PLANS:
Reducer 5
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), (_col1 * _col1) (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(), count(_col1)
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 6 Data size: 720 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 6 Data size: 720 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0), count(VALUE._col1)
keys: KEY._col0 (type: string)
@@ -2363,19 +1880,31 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 6 Data size: 720 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: bigint), _col2 (type: bigint)
- Reducer 8
+ Reducer 7
Execution mode: llap
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 3 Data size: 336 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 3 Data size: 336 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
+ Statistics: Num rows: 3 Data size: 336 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: boolean)
Stage: Stage-0
Fetch Operator
@@ -2900,17 +2429,13 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (SIMPLE_EDGE), Reducer 15 (SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
- Reducer 15 <- Map 14 (SIMPLE_EDGE)
- Reducer 17 <- Map 16 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 13 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+ Reducer 11 <- Map 10 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
Reducer 4 <- Reducer 3 (CUSTOM_SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 9 <- Reducer 11 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2933,40 +2458,6 @@ STAGE PLANS:
Map 10
Map Operator Tree:
TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 14
- Map Operator Tree:
- TableScan
- alias: e
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 16
- Map Operator Tree:
- TableScan
alias: e
Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
@@ -2986,107 +2477,50 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int)
+ Filter Operator
+ predicate: (p_name = p_name) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_name (type: string), p_partkey (type: int)
+ outputColumnNames: p_name, p_partkey
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(p_partkey)
+ keys: p_name (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 6 Data size: 822 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 6 Data size: 822 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: bigint), _col2 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
- Map 8
+ Map 7
Map Operator Tree:
TableScan
- alias: e
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
+ alias: part
+ Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_name = p_name) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_partkey (type: int), p_name (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Reducer 11
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: int), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
- Statistics: Num rows:
<TRUNCATED>
[09/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/subquery_views.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_views.q.out b/ql/src/test/results/clientpositive/llap/subquery_views.q.out
index d3fdec0..bfd56e6 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_views.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_views.q.out
@@ -124,27 +124,17 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (SIMPLE_EDGE), Reducer 15 (SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
- Reducer 15 <- Map 14 (SIMPLE_EDGE)
- Reducer 17 <- Map 16 (SIMPLE_EDGE)
- Reducer 19 <- Map 18 (SIMPLE_EDGE), Reducer 23 (SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 20 <- Reducer 19 (SIMPLE_EDGE), Reducer 29 (SIMPLE_EDGE)
- Reducer 22 <- Map 21 (SIMPLE_EDGE), Reducer 25 (SIMPLE_EDGE)
- Reducer 23 <- Reducer 22 (SIMPLE_EDGE)
- Reducer 25 <- Map 24 (SIMPLE_EDGE)
- Reducer 27 <- Map 26 (SIMPLE_EDGE), Reducer 31 (SIMPLE_EDGE)
- Reducer 28 <- Reducer 27 (SIMPLE_EDGE)
- Reducer 29 <- Reducer 28 (SIMPLE_EDGE), Reducer 33 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 13 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
- Reducer 31 <- Map 30 (SIMPLE_EDGE)
- Reducer 33 <- Map 32 (SIMPLE_EDGE)
- Reducer 4 <- Reducer 20 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 10 <- Map 9 (SIMPLE_EDGE)
+ Reducer 12 <- Map 11 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE), Reducer 17 (SIMPLE_EDGE)
+ Reducer 14 <- Map 13 (SIMPLE_EDGE)
+ Reducer 16 <- Map 15 (SIMPLE_EDGE)
+ Reducer 17 <- Reducer 16 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
+ Reducer 19 <- Map 18 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 12 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
+ Reducer 5 <- Map 4 (SIMPLE_EDGE)
+ Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 8 <- Reducer 10 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -168,64 +158,7 @@ STAGE PLANS:
Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 10
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: (value > 'val_11') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 14
- Map Operator Tree:
- TableScan
- alias: b
- properties:
- insideView TRUE
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 16
- Map Operator Tree:
- TableScan
- alias: b
- properties:
- insideView TRUE
- Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: key (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 18
+ Map 11
Map Operator Tree:
TableScan
alias: b
@@ -246,83 +179,49 @@ STAGE PLANS:
Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 21
+ Map 13
Map Operator Tree:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value > 'val_11') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((value = value) and (key = key) and (value > 'val_11')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(key)
+ keys: key (type: string), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: bigint), _col3 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
- Map 24
- Map Operator Tree:
- TableScan
- alias: b
- properties:
- insideView TRUE
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 26
+ Map 15
Map Operator Tree:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value > 'val_11') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
+ predicate: ((value = value) and (key = key) and (value > 'val_11')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 30
- Map Operator Tree:
- TableScan
- alias: b
- properties:
- insideView TRUE
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 32
+ Map 18
Map Operator Tree:
TableScan
alias: b
@@ -341,114 +240,68 @@ STAGE PLANS:
Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 5
+ Map 4
Map Operator Tree:
TableScan
alias: a
Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (value > 'val_11') (type: boolean)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: key (type: string), value (type: string)
+ predicate: ((value = value) and (key = key) and (value > 'val_11')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(), count(key)
+ keys: key (type: string), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: bigint), _col3 (type: bigint)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Map 6
+ Map Operator Tree:
+ TableScan
+ alias: a
+ Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: ((value = value) and (key = key) and (value > 'val_11')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 7298 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 29548 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Map 8
+ Map 9
Map Operator Tree:
TableScan
alias: b
properties:
insideView TRUE
- Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 500 Data size: 43500 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
- keys: key (type: string), value (type: string)
+ keys: key (type: string)
mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col0
+ Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string)
- sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: string), _col3 (type: boolean)
- Reducer 13
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2, _col3, _col4
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col4 (type: string), _col2 (type: string), _col1 (type: string)
- sort order: +++
- Map-reduce partition columns: _col4 (type: string), _col2 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col3 (type: boolean)
- Reducer 15
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 17
+ Reducer 10
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -461,58 +314,26 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 19
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1, _col4, _col5
- Statistics: Num rows: 166 Data size: 32204 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col0 (type: string)
- sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 166 Data size: 32204 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col4 (type: bigint), _col5 (type: bigint)
- Reducer 2
+ Reducer 12
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
condition map:
Left Outer Join0 to 1
+ Left Outer Join0 to 2
keys:
0 _col0 (type: string), _col1 (type: string)
1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1, _col4, _col5
- Statistics: Num rows: 166 Data size: 32204 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col0 (type: string)
- sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 166 Data size: 32204 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col4 (type: bigint), _col5 (type: bigint)
- Reducer 20
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string), _col0 (type: string)
- 1 _col4 (type: string), _col2 (type: string), _col1 (type: string)
+ 2 _col4 (type: string), _col2 (type: string)
outputColumnNames: _col0, _col4, _col5, _col9
- Statistics: Num rows: 166 Data size: 17762 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 107 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col9 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END (type: boolean)
- Statistics: Num rows: 83 Data size: 8881 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 107 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 83 Data size: 7221 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 87 Basic stats: COMPLETE Column stats: COMPLETE
Group By Operator
keys: _col0 (type: string)
mode: hash
@@ -523,34 +344,7 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 1 Data size: 87 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 22
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col3 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col3, _col0
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(), count(_col0)
- keys: _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: bigint), _col3 (type: bigint)
- Reducer 23
+ Reducer 14
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -558,66 +352,32 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: bigint), _col3 (type: bigint)
- Reducer 25
+ Reducer 16
Execution mode: llap
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 27
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string)
- sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string)
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 28
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string), KEY._col2 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
+ expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col2, _col3
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string), _col2 (type: string), _col3 (type: boolean)
- Reducer 29
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: string), _col3 (type: boolean)
+ Reducer 17
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
@@ -626,65 +386,54 @@ STAGE PLANS:
keys:
0 _col0 (type: string)
1 _col0 (type: string)
- outputColumnNames: _col1, _col2, _col3, _col4
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
+ outputColumnNames: _col2, _col3, _col4
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col4 (type: string), _col2 (type: string), _col1 (type: string)
- sort order: +++
- Map-reduce partition columns: _col4 (type: string), _col2 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 269 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: _col4 (type: string), _col2 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col4 (type: string), _col2 (type: string)
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col3 (type: boolean)
- Reducer 3
+ Reducer 19
+ Execution mode: llap
+ Reduce Operator Tree:
+ Group By Operator
+ keys: KEY._col0 (type: string)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
+ Reducer 2
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
condition map:
Left Outer Join0 to 1
+ Left Outer Join0 to 2
keys:
- 0 _col0 (type: string), _col1 (type: string), _col0 (type: string)
- 1 _col4 (type: string), _col2 (type: string), _col1 (type: string)
+ 0 _col0 (type: string), _col1 (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+ 2 _col4 (type: string), _col2 (type: string)
outputColumnNames: _col0, _col1, _col4, _col5, _col9
- Statistics: Num rows: 166 Data size: 32868 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 198 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col9 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END (type: boolean)
- Statistics: Num rows: 83 Data size: 16434 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 198 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col1 (type: string)
- Reducer 31
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 33
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 205 Data size: 17835 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 4
+ Reducer 3
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
@@ -694,42 +443,15 @@ STAGE PLANS:
0 _col0 (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
compressed: false
- Statistics: Num rows: 83 Data size: 14774 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col2, _col3
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col3 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col3, _col0
- Statistics: Num rows: 1 Data size: 265 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(), count(_col0)
- keys: _col2 (type: string), _col3 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: bigint), _col3 (type: bigint)
- Reducer 7
+ Reducer 5
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -737,26 +459,48 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 194 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3880 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col2 (type: bigint), _col3 (type: bigint)
- Reducer 9
+ Reducer 7
Execution mode: llap
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 20 Data size: 3560 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col2, _col3
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: string), _col3 (type: boolean)
+ Reducer 8
+ Execution mode: llap
+ Reduce Operator Tree:
+ Merge Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col2, _col3, _col4
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
+ key expressions: _col4 (type: string), _col2 (type: string)
sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 44500 Basic stats: COMPLETE Column stats: COMPLETE
+ Map-reduce partition columns: _col4 (type: string), _col2 (type: string)
+ Statistics: Num rows: 20 Data size: 3640 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col3 (type: boolean)
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out b/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
index 7cbd4f6..48deecf 100644
--- a/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
+++ b/ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out
@@ -233,10 +233,7 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Map 7 <- Reducer 6 (BROADCAST_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (BROADCAST_EDGE), Reducer 8 (BROADCAST_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE)
- Reducer 8 <- Map 4 (BROADCAST_EDGE), Map 7 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 3 (BROADCAST_EDGE), Map 4 (BROADCAST_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -297,88 +294,26 @@ STAGE PLANS:
alias: lineitem
Statistics: Num rows: 100 Data size: 9600 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: (l_shipmode = 'AIR') (type: boolean)
- Statistics: Num rows: 14 Data size: 1344 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((l_shipmode = 'AIR') and (l_linenumber = l_linenumber)) (type: boolean)
+ Statistics: Num rows: 7 Data size: 672 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: l_orderkey (type: int), l_linenumber (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1330 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: int)
- sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 14 Data size: 1330 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
- Map 5
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 400 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 100 Data size: 400 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: l_partkey (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
- Map 7
- Map Operator Tree:
- TableScan
- alias: li
- Statistics: Num rows: 100 Data size: 800 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 100 Data size: 800 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: l_partkey (type: int), l_linenumber (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 100 Data size: 800 Basic stats: COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col2
- input vertices:
- 0 Reducer 6
- Statistics: Num rows: 34 Data size: 136 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: _col2 (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 7 Data size: 56 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ Group By Vectorization:
+ vectorOutput: false
+ native: false
+ projectedOutputColumns: null
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 2 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
Map Vectorization:
@@ -432,7 +367,7 @@ STAGE PLANS:
nativeConditionsMet: hive.mapjoin.optimized.hashtable IS true, hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Small table vectorizes IS true, Optimized Table and Supports Key Types IS true
outputColumnNames: _col0, _col3
input vertices:
- 1 Reducer 8
+ 1 Map 4
Statistics: Num rows: 5 Data size: 40 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col0 (type: int), _col3 (type: int)
@@ -452,99 +387,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 6
- Execution mode: vectorized, llap
- Reduce Vectorization:
- enabled: true
- enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
- groupByVectorOutput: true
- allNative: false
- usesVectorUDFAdaptor: false
- vectorized: true
- Reduce Operator Tree:
- Group By Operator
- Group By Vectorization:
- className: VectorGroupByOperator
- vectorOutput: true
- keyExpressions: col 0
- native: false
- projectedOutputColumns: []
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Reduce Sink Vectorization:
- className: VectorReduceSinkLongOperator
- native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- Statistics: Num rows: 50 Data size: 200 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 8
- Execution mode: vectorized, llap
- Reduce Vectorization:
- enabled: true
- enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true
- groupByVectorOutput: true
- allNative: false
- usesVectorUDFAdaptor: false
- vectorized: true
- Reduce Operator Tree:
- Group By Operator
- Group By Vectorization:
- className: VectorGroupByOperator
- vectorOutput: true
- keyExpressions: col 0
- native: false
- projectedOutputColumns: []
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 3 Data size: 12 Basic stats: COMPLETE Column stats: COMPLETE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- Map Join Vectorization:
- className: VectorMapJoinInnerLongOperator
- native: true
- nativeConditionsMet: hive.mapjoin.optimized.hashtable IS true, hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Small table vectorizes IS true, Optimized Table and Supports Key Types IS true
- outputColumnNames: _col0, _col3
- input vertices:
- 0 Map 4
- Statistics: Num rows: 14 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Select Vectorization:
- className: VectorSelectOperator
- native: true
- projectedOutputColumns: [1, 0]
- Statistics: Num rows: 14 Data size: 112 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- Group By Vectorization:
- className: VectorGroupByOperator
- vectorOutput: true
- keyExpressions: col 1, col 0
- native: false
- projectedOutputColumns: []
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 4 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Reduce Sink Vectorization:
- className: VectorReduceSinkMultiKeyOperator
- native: true
- nativeConditionsMet: hive.vectorized.execution.reducesink.new.enabled IS true, hive.execution.engine tez IN [tez, spark] IS true, Not ACID UPDATE or DELETE IS true, No buckets IS true, No TopN IS true, Uniform Hash IS true, No DISTINCT columns IS true, BinarySortableSerDe for keys IS true, LazyBinarySerDe for values IS true
- Statistics: Num rows: 4 Data size: 32 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/perf/query1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/query1.q.out b/ql/src/test/results/clientpositive/perf/query1.q.out
index a47548f..09278e3 100644
--- a/ql/src/test/results/clientpositive/perf/query1.q.out
+++ b/ql/src/test/results/clientpositive/perf/query1.q.out
@@ -1,4 +1,3 @@
-Warning: Shuffle Join MERGEJOIN[119][tables = [$hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 18' is a cross product
PREHOOK: query: explain with customer_total_return as
(select sr_customer_sk as ctr_customer_sk
,sr_store_sk as ctr_store_sk
@@ -48,19 +47,13 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 16 (SIMPLE_EDGE)
+Reducer 12 <- Map 11 (SIMPLE_EDGE), Map 14 (SIMPLE_EDGE)
Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
-Reducer 14 <- Reducer 13 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
-Reducer 15 <- Reducer 14 (SIMPLE_EDGE)
-Reducer 18 <- Map 17 (CUSTOM_SIMPLE_EDGE), Map 20 (CUSTOM_SIMPLE_EDGE), Reducer 23 (CUSTOM_SIMPLE_EDGE)
-Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 8 (SIMPLE_EDGE)
-Reducer 22 <- Map 21 (SIMPLE_EDGE), Map 24 (SIMPLE_EDGE)
-Reducer 23 <- Reducer 22 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Map 9 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
Reducer 5 <- Map 10 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 15 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 13 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
Stage-0
@@ -68,152 +61,87 @@ Stage-0
limit:100
Stage-1
Reducer 7
- File Output Operator [FS_84]
- Limit [LIM_83] (rows=100 width=2)
+ File Output Operator [FS_54]
+ Limit [LIM_53] (rows=100 width=860)
Number of rows:100
- Select Operator [SEL_82] (rows=435393727038609152 width=2)
+ Select Operator [SEL_52] (rows=32266667 width=860)
Output:["_col0"]
<-Reducer 6 [SIMPLE_EDGE]
- SHUFFLE [RS_81]
- Select Operator [SEL_80] (rows=435393727038609152 width=2)
+ SHUFFLE [RS_51]
+ Select Operator [SEL_50] (rows=32266667 width=860)
Output:["_col0"]
- Filter Operator [FIL_79] (rows=435393727038609152 width=2)
+ Filter Operator [FIL_49] (rows=32266667 width=860)
predicate:(_col2 > CASE WHEN (_col8 is null) THEN (null) ELSE (_col7) END)
- Merge Join Operator [MERGEJOIN_125] (rows=1306181181115827456 width=2)
- Conds:RS_75._col1=RS_76._col2(Left Outer),Output:["_col2","_col6","_col7","_col8"]
- <-Reducer 15 [SIMPLE_EDGE]
- SHUFFLE [RS_76]
+ Merge Join Operator [MERGEJOIN_78] (rows=96800003 width=860)
+ Conds:RS_45._col1=RS_46._col2(Left Outer),Output:["_col2","_col6","_col7","_col8"]
+ <-Reducer 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_46]
PartitionCols:_col2
- Select Operator [SEL_68] (rows=1187437411641012224 width=2)
+ Select Operator [SEL_38] (rows=7918783 width=77)
Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_67] (rows=1187437411641012224 width=2)
- Output:["_col0","_col1"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0
- <-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_66]
- PartitionCols:_col0
- Group By Operator [GBY_65] (rows=2374874823282024448 width=2)
- Output:["_col0","_col1"],aggregations:["avg(_col2)"],keys:_col3
- Select Operator [SEL_64] (rows=2374874823282024448 width=2)
- Output:["_col3","_col2"]
- Merge Join Operator [MERGEJOIN_124] (rows=2374874823282024448 width=2)
- Conds:RS_61._col1=RS_62._col0(Inner),Output:["_col2","_col3"]
- <-Reducer 13 [SIMPLE_EDGE]
- SHUFFLE [RS_61]
- PartitionCols:_col1
- Select Operator [SEL_33] (rows=31675133 width=77)
- Output:["_col1","_col2"]
- Group By Operator [GBY_32] (rows=31675133 width=77)
- Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
- <-Reducer 12 [SIMPLE_EDGE]
- SHUFFLE [RS_31]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_30] (rows=63350266 width=77)
- Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col2, _col1
- Select Operator [SEL_29] (rows=63350266 width=77)
- Output:["_col2","_col1","_col3"]
- Merge Join Operator [MERGEJOIN_123] (rows=63350266 width=77)
- Conds:RS_26._col0=RS_27._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Map 11 [SIMPLE_EDGE]
- SHUFFLE [RS_26]
- PartitionCols:_col0
- Select Operator [SEL_22] (rows=57591150 width=77)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_114] (rows=57591150 width=77)
- predicate:sr_returned_date_sk is not null
- TableScan [TS_20] (rows=57591150 width=77)
- default@store_returns,store_returns,Tbl:COMPLETE,Col:NONE,Output:["sr_returned_date_sk","sr_customer_sk","sr_store_sk","sr_fee"]
- <-Map 16 [SIMPLE_EDGE]
- SHUFFLE [RS_27]
- PartitionCols:_col0
- Select Operator [SEL_25] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_115] (rows=36524 width=1119)
- predicate:((d_year = 2000) and d_date_sk is not null)
- TableScan [TS_23] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Reducer 19 [SIMPLE_EDGE]
- SHUFFLE [RS_62]
- PartitionCols:_col0
- Group By Operator [GBY_59] (rows=2158977065280000000 width=2)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 18 [SIMPLE_EDGE]
- SHUFFLE [RS_58]
+ Group By Operator [GBY_37] (rows=7918783 width=77)
+ Output:["_col0","_col1"],aggregations:["avg(_col2)"],keys:_col1
+ Select Operator [SEL_33] (rows=15837566 width=77)
+ Output:["_col1","_col2"]
+ Group By Operator [GBY_32] (rows=15837566 width=77)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
+ <-Reducer 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_31]
+ PartitionCols:_col0
+ Group By Operator [GBY_30] (rows=31675133 width=77)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col2, _col1
+ Select Operator [SEL_29] (rows=31675133 width=77)
+ Output:["_col2","_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_77] (rows=31675133 width=77)
+ Conds:RS_26._col0=RS_27._col0(Inner),Output:["_col1","_col2","_col3"]
+ <-Map 11 [SIMPLE_EDGE]
+ SHUFFLE [RS_26]
PartitionCols:_col0
- Group By Operator [GBY_57] (rows=4317954130560000000 width=2)
- Output:["_col0"],keys:_col3
- Merge Join Operator [MERGEJOIN_119] (rows=4317954130560000000 width=2)
- Conds:(Inner),(Inner),Output:["_col3"]
- <-Map 17 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_52]
- Select Operator [SEL_35] (rows=80000000 width=4)
- TableScan [TS_34] (rows=80000000 width=860)
- default@customer,customer,Tbl:COMPLETE,Col:COMPLETE
- <-Map 20 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_53]
- Select Operator [SEL_37] (rows=1704 width=4)
- TableScan [TS_36] (rows=1704 width=1910)
- default@store,store,Tbl:COMPLETE,Col:COMPLETE
- <-Reducer 23 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_54]
- Select Operator [SEL_51] (rows=31675133 width=77)
- Output:["_col1"]
- Group By Operator [GBY_50] (rows=31675133 width=77)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 22 [SIMPLE_EDGE]
- SHUFFLE [RS_49]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_48] (rows=63350266 width=77)
- Output:["_col0","_col1"],keys:_col1, _col2
- Merge Join Operator [MERGEJOIN_118] (rows=63350266 width=77)
- Conds:RS_44._col0=RS_45._col0(Inner),Output:["_col1","_col2"]
- <-Map 21 [SIMPLE_EDGE]
- SHUFFLE [RS_44]
- PartitionCols:_col0
- Select Operator [SEL_40] (rows=57591150 width=77)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_116] (rows=57591150 width=77)
- predicate:sr_returned_date_sk is not null
- TableScan [TS_38] (rows=57591150 width=77)
- default@store_returns,store_returns,Tbl:COMPLETE,Col:NONE,Output:["sr_returned_date_sk","sr_customer_sk","sr_store_sk"]
- <-Map 24 [SIMPLE_EDGE]
- SHUFFLE [RS_45]
- PartitionCols:_col0
- Select Operator [SEL_43] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_117] (rows=36524 width=1119)
- predicate:((d_year = 2000) and d_date_sk is not null)
- TableScan [TS_41] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
+ Select Operator [SEL_22] (rows=28795575 width=77)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_72] (rows=28795575 width=77)
+ predicate:((sr_store_sk = sr_store_sk) and sr_returned_date_sk is not null)
+ TableScan [TS_20] (rows=57591150 width=77)
+ default@store_returns,store_returns,Tbl:COMPLETE,Col:NONE,Output:["sr_returned_date_sk","sr_customer_sk","sr_store_sk","sr_fee"]
+ <-Map 14 [SIMPLE_EDGE]
+ SHUFFLE [RS_27]
+ PartitionCols:_col0
+ Select Operator [SEL_25] (rows=36524 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_73] (rows=36524 width=1119)
+ predicate:((d_year = 2000) and d_date_sk is not null)
+ TableScan [TS_23] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
<-Reducer 5 [SIMPLE_EDGE]
- SHUFFLE [RS_75]
+ SHUFFLE [RS_45]
PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_122] (rows=88000001 width=860)
- Conds:RS_72._col0=RS_73._col0(Inner),Output:["_col1","_col2","_col6"]
+ Merge Join Operator [MERGEJOIN_76] (rows=88000001 width=860)
+ Conds:RS_42._col0=RS_43._col0(Inner),Output:["_col1","_col2","_col6"]
<-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_73]
+ SHUFFLE [RS_43]
PartitionCols:_col0
Select Operator [SEL_19] (rows=80000000 width=860)
Output:["_col0","_col1"]
- Filter Operator [FIL_113] (rows=80000000 width=860)
+ Filter Operator [FIL_71] (rows=80000000 width=860)
predicate:c_customer_sk is not null
TableScan [TS_17] (rows=80000000 width=860)
default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_customer_id"]
<-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_72]
+ SHUFFLE [RS_42]
PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_121] (rows=34842647 width=77)
- Conds:RS_69._col1=RS_70._col0(Inner),Output:["_col0","_col1","_col2"]
+ Merge Join Operator [MERGEJOIN_75] (rows=34842647 width=77)
+ Conds:RS_39._col1=RS_40._col0(Inner),Output:["_col0","_col1","_col2"]
<-Map 9 [SIMPLE_EDGE]
- SHUFFLE [RS_70]
+ SHUFFLE [RS_40]
PartitionCols:_col0
Select Operator [SEL_16] (rows=852 width=1910)
Output:["_col0"]
- Filter Operator [FIL_112] (rows=852 width=1910)
+ Filter Operator [FIL_70] (rows=852 width=1910)
predicate:((s_state = 'NM') and s_store_sk is not null)
TableScan [TS_14] (rows=1704 width=1910)
default@store,store,Tbl:COMPLETE,Col:NONE,Output:["s_store_sk","s_state"]
<-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_69]
+ SHUFFLE [RS_39]
PartitionCols:_col1
Select Operator [SEL_13] (rows=31675133 width=77)
Output:["_col0","_col1","_col2"]
@@ -226,14 +154,14 @@ Stage-0
Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col2, _col1
Select Operator [SEL_9] (rows=63350266 width=77)
Output:["_col2","_col1","_col3"]
- Merge Join Operator [MERGEJOIN_120] (rows=63350266 width=77)
+ Merge Join Operator [MERGEJOIN_74] (rows=63350266 width=77)
Conds:RS_6._col0=RS_7._col0(Inner),Output:["_col1","_col2","_col3"]
<-Map 1 [SIMPLE_EDGE]
SHUFFLE [RS_6]
PartitionCols:_col0
Select Operator [SEL_2] (rows=57591150 width=77)
Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_110] (rows=57591150 width=77)
+ Filter Operator [FIL_68] (rows=57591150 width=77)
predicate:(sr_returned_date_sk is not null and sr_store_sk is not null and sr_customer_sk is not null)
TableScan [TS_0] (rows=57591150 width=77)
default@store_returns,store_returns,Tbl:COMPLETE,Col:NONE,Output:["sr_returned_date_sk","sr_customer_sk","sr_store_sk","sr_fee"]
@@ -242,7 +170,7 @@ Stage-0
PartitionCols:_col0
Select Operator [SEL_5] (rows=36524 width=1119)
Output:["_col0"]
- Filter Operator [FIL_111] (rows=36524 width=1119)
+ Filter Operator [FIL_69] (rows=36524 width=1119)
predicate:((d_year = 2000) and d_date_sk is not null)
TableScan [TS_3] (rows=73049 width=1119)
default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
[10/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b/ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
index ae87ab2..da387d7 100644
--- a/ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
+++ b/ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
@@ -1681,11 +1681,9 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
- Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 5 <- Map 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1710,33 +1708,25 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 3250 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 7
- Map Operator Tree:
- TableScan
- alias: e
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_name = p_name) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_name (type: string), p_partkey (type: int)
+ outputColumnNames: p_name, p_partkey
+ Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: max(p_partkey)
+ keys: p_name (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: int)
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -1782,33 +1772,6 @@ STAGE PLANS:
Reducer 5
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: int)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 13 Data size: 1625 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: max(_col0)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 6 Data size: 750 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0)
keys: KEY._col0 (type: string)
@@ -1825,19 +1788,6 @@ STAGE PLANS:
Map-reduce partition columns: _col2 (type: string)
Statistics: Num rows: 6 Data size: 774 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: int), _col1 (type: boolean)
- Reducer 8
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -1867,10 +1817,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1895,33 +1843,21 @@ STAGE PLANS:
TableScan
alias: part_null
Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 6
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: avg(p_size)
+ keys: p_type (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: struct<count:bigint,sum:double,input:int>)
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -1952,62 +1888,22 @@ STAGE PLANS:
Reducer 4
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: avg(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: struct<count:bigint,sum:double,input:int>)
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: avg(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: double), true (type: boolean), _col0 (type: string)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: double), _col1 (type: boolean)
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -2026,8 +1922,8 @@ POSTHOOK: Input: default@part
POSTHOOK: Input: default@part_null
#### A masked pattern was here ####
192697 almond antique blue firebrick mint Manufacturer#5 Brand#52 MEDIUM BURNISHED TIN 31 LG DRUM 1789.69 ickly ir
-Warning: Shuffle Join MERGEJOIN[58][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
-Warning: Shuffle Join MERGEJOIN[59][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 4' is a cross product
+Warning: Shuffle Join MERGEJOIN[44][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
+Warning: Shuffle Join MERGEJOIN[45][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 4' is a cross product
PREHOOK: query: explain select * from part where p_size BETWEEN (select min(p_size) from part_null where part_null.p_type = part.p_type) AND (select max(p_size) from part_null)
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from part where p_size BETWEEN (select min(p_size) from part_null where part_null.p_type = part.p_type) AND (select max(p_size) from part_null)
@@ -2041,14 +1937,12 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (CUSTOM_SIMPLE_EDGE)
- Reducer 13 <- Map 12 (CUSTOM_SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 11 (CUSTOM_SIMPLE_EDGE), Reducer 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 4 <- Reducer 13 (CUSTOM_SIMPLE_EDGE), Reducer 3 (CUSTOM_SIMPLE_EDGE)
- Reducer 6 <- Map 5 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
- Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE)
+ Reducer 10 <- Map 9 (CUSTOM_SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE), Reducer 8 (CUSTOM_SIMPLE_EDGE)
+ Reducer 4 <- Reducer 10 (CUSTOM_SIMPLE_EDGE), Reducer 3 (CUSTOM_SIMPLE_EDGE)
+ Reducer 6 <- Map 5 (SIMPLE_EDGE)
+ Reducer 8 <- Map 7 (CUSTOM_SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2068,7 +1962,29 @@ STAGE PLANS:
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 10
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: part_null
+ Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: min(p_size)
+ keys: p_type (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: int)
+ Execution mode: llap
+ LLAP IO: no inputs
+ Map 7
Map Operator Tree:
TableScan
alias: part_null
@@ -2088,7 +2004,7 @@ STAGE PLANS:
value expressions: _col0 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
- Map 12
+ Map 9
Map Operator Tree:
TableScan
alias: part_null
@@ -2108,64 +2024,7 @@ STAGE PLANS:
value expressions: _col0 (type: int)
Execution mode: llap
LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: part_null
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 8
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (sq_count_check(_col0) <= 1) (type: boolean)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reducer 13
+ Reducer 10
Execution mode: llap
Reduce Operator Tree:
Group By Operator
@@ -2235,75 +2094,58 @@ STAGE PLANS:
Reducer 6
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), true (type: boolean), _col0 (type: string)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: boolean)
- Reducer 9
+ Reducer 8
Execution mode: llap
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ aggregations: count(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
-
- Stage: Stage-0
- Fetch Operator
- limit: -1
- Processor Tree:
- ListSink
-
-Warning: Shuffle Join MERGEJOIN[58][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
-Warning: Shuffle Join MERGEJOIN[59][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 4' is a cross product
-PREHOOK: query: select * from part where p_size BETWEEN (select min(p_size) from part_null where part_null.p_type = part.p_type) AND (select max(p_size) from part_null)
-PREHOOK: type: QUERY
-PREHOOK: Input: default@part
-PREHOOK: Input: default@part_null
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: complete
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (sq_count_check(_col0) <= 1) (type: boolean)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+
+ Stage: Stage-0
+ Fetch Operator
+ limit: -1
+ Processor Tree:
+ ListSink
+
+Warning: Shuffle Join MERGEJOIN[44][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
+Warning: Shuffle Join MERGEJOIN[45][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 4' is a cross product
+PREHOOK: query: select * from part where p_size BETWEEN (select min(p_size) from part_null where part_null.p_type = part.p_type) AND (select max(p_size) from part_null)
+PREHOOK: type: QUERY
+PREHOOK: Input: default@part
+PREHOOK: Input: default@part_null
#### A masked pattern was here ####
POSTHOOK: query: select * from part where p_size BETWEEN (select min(p_size) from part_null where part_null.p_type = part.p_type) AND (select max(p_size) from part_null)
POSTHOOK: type: QUERY
@@ -2336,7 +2178,7 @@ POSTHOOK: Input: default@part_null
42669 almond antique medium spring khaki Manufacturer#5 Brand#51 STANDARD BURNISHED TIN 6 MED CAN 1611.66 sits haggl
195606 almond aquamarine sandy cyan gainsboro Manufacturer#2 Brand#25 STANDARD PLATED TIN 18 SM PKG 1701.6 ic de
144293 almond antique olive coral navajo Manufacturer#3 Brand#34 STANDARD POLISHED STEEL 45 JUMBO CAN 1337.29 ag furiously about
-Warning: Shuffle Join MERGEJOIN[58][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
+Warning: Shuffle Join MERGEJOIN[44][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
PREHOOK: query: explain select * from part where p_size >= (select min(p_size) from part_null where part_null.p_type = part.p_type) AND p_retailprice <= (select max(p_retailprice) from part_null)
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from part where p_size >= (select min(p_size) from part_null where part_null.p_type = part.p_type) AND p_retailprice <= (select max(p_retailprice) from part_null)
@@ -2350,13 +2192,11 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 10 <- Map 9 (CUSTOM_SIMPLE_EDGE)
- Reducer 12 <- Map 11 (CUSTOM_SIMPLE_EDGE)
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 3 <- Reducer 10 (CUSTOM_SIMPLE_EDGE), Reducer 12 (CUSTOM_SIMPLE_EDGE), Reducer 2 (CUSTOM_SIMPLE_EDGE)
- Reducer 5 <- Map 4 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
- Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+ Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE), Reducer 7 (CUSTOM_SIMPLE_EDGE), Reducer 9 (CUSTOM_SIMPLE_EDGE)
+ Reducer 5 <- Map 4 (SIMPLE_EDGE)
+ Reducer 7 <- Map 6 (CUSTOM_SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (CUSTOM_SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2376,61 +2216,29 @@ STAGE PLANS:
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 11
+ Map 4
Map Operator Tree:
TableScan
alias: part_null
- Statistics: Num rows: 407 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_retailprice (type: double)
- outputColumnNames: p_retailprice
- Statistics: Num rows: 407 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: max(p_retailprice)
+ aggregations: min(p_size)
+ keys: p_type (type: string)
mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: double)
- Execution mode: llap
- LLAP IO: no inputs
- Map 4
- Map Operator Tree:
- TableScan
- alias: part_null
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 31 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Execution mode: llap
- LLAP IO: no inputs
- Map 7
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_type (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 15 Data size: 1575 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: int)
Execution mode: llap
LLAP IO: no inputs
- Map 9
+ Map 6
Map Operator Tree:
TableScan
alias: part_null
@@ -2450,41 +2258,26 @@ STAGE PLANS:
value expressions: _col0 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
- Reducer 10
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (sq_count_check(_col0) <= 1) (type: boolean)
+ Map 8
+ Map Operator Tree:
+ TableScan
+ alias: part_null
+ Statistics: Num rows: 407 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: p_retailprice (type: double)
+ outputColumnNames: p_retailprice
+ Statistics: Num rows: 407 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: max(p_retailprice)
+ mode: hash
+ outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Select Operator
+ Reduce Output Operator
+ sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reducer 12
+ value expressions: _col0 (type: double)
Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: max(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: double)
+ LLAP IO: no inputs
Reducer 2
Execution mode: llap
Reduce Operator Tree:
@@ -2537,62 +2330,57 @@ STAGE PLANS:
Reducer 5
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 34 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), true (type: boolean), _col0 (type: string)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 17 Data size: 1790 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 7 Data size: 735 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col1 (type: boolean)
- Reducer 8
+ Reducer 7
Execution mode: llap
Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: complete
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (sq_count_check(_col0) <= 1) (type: boolean)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reducer 9
+ Execution mode: llap
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: max(VALUE._col0)
mode: mergepartial
outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: double)
Stage: Stage-0
Fetch Operator
@@ -2600,7 +2388,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[58][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
+Warning: Shuffle Join MERGEJOIN[44][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product
PREHOOK: query: select * from part where p_size >= (select min(p_size) from part_null where part_null.p_type = part.p_type) AND p_retailprice <= (select max(p_retailprice) from part_null)
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -2637,9 +2425,7 @@ POSTHOOK: Input: default@part_null
42669 almond antique medium spring khaki Manufacturer#5 Brand#51 STANDARD BURNISHED TIN 6 MED CAN 1611.66 sits haggl
195606 almond aquamarine sandy cyan gainsboro Manufacturer#2 Brand#25 STANDARD PLATED TIN 18 SM PKG 1701.6 ic de
144293 almond antique olive coral navajo Manufacturer#3 Brand#34 STANDARD POLISHED STEEL 45 JUMBO CAN 1337.29 ag furiously about
-Warning: Shuffle Join MERGEJOIN[89][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 2' is a cross product
-Warning: Shuffle Join MERGEJOIN[90][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 11' is a cross product
-Warning: Shuffle Join MERGEJOIN[91][tables = [$hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 12' is a cross product
+Warning: Shuffle Join MERGEJOIN[46][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: explain select * from part where p_brand <> (select min(p_brand) from part ) AND p_size IN (select (p_size) from part p where p.p_type = part.p_type ) AND p_size <> 340
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from part where p_brand <> (select min(p_brand) from part ) AND p_size IN (select (p_size) from part p where p.p_type = part.p_type ) AND p_size <> 340
@@ -2653,16 +2439,11 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 11 <- Map 10 (CUSTOM_SIMPLE_EDGE), Reducer 15 (CUSTOM_SIMPLE_EDGE)
- Reducer 12 <- Reducer 11 (CUSTOM_SIMPLE_EDGE), Reducer 17 (CUSTOM_SIMPLE_EDGE)
- Reducer 13 <- Reducer 12 (SIMPLE_EDGE)
- Reducer 15 <- Map 14 (CUSTOM_SIMPLE_EDGE)
- Reducer 17 <- Map 16 (CUSTOM_SIMPLE_EDGE)
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE), Reducer 5 (CUSTOM_SIMPLE_EDGE), Reducer 7 (CUSTOM_SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
Reducer 5 <- Map 4 (CUSTOM_SIMPLE_EDGE)
Reducer 7 <- Map 6 (CUSTOM_SIMPLE_EDGE)
- Reducer 9 <- Map 8 (SIMPLE_EDGE), Reducer 13 (SIMPLE_EDGE)
+ Reducer 9 <- Map 8 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2683,61 +2464,6 @@ STAGE PLANS:
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 10
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 14
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int)
- outputColumnNames: p_partkey
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(p_partkey)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: bigint)
- Execution mode: llap
- LLAP IO: no inputs
- Map 16
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int)
- outputColumnNames: p_partkey
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(p_partkey)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: bigint)
- Execution mode: llap
- LLAP IO: no inputs
Map 4
Map Operator Tree:
TableScan
@@ -2783,110 +2509,28 @@ STAGE PLANS:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: int)
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: p_type (type: string), p_size (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: llap
LLAP IO: no inputs
- Reducer 11
+ Reducer 2
Execution mode: llap
Reduce Operator Tree:
Merge Join Operator
condition map:
Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
- Reducer 12
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 2704 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 13
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1352 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 15
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count()
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Filter Operator
- predicate: (sq_count_check(_col0) <= 1) (type: boolean)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 17
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 2
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- Inner Join 0 to 2
+ Inner Join 0 to 2
keys:
0
1
@@ -2907,10 +2551,10 @@ STAGE PLANS:
Reduce Operator Tree:
Merge Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col4 (type: string), _col5 (type: int)
- 1 _col0 (type: string), _col1 (type: int)
+ 1 _col1 (type: string), _col0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 1 Data size: 619 Basic stats: COMPLETE Column stats: COMPLETE
File Output Operator
@@ -2958,28 +2602,20 @@ STAGE PLANS:
Reducer 9
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: int)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
+ expressions: _col1 (type: int), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 12 Data size: 1296 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 6 Data size: 648 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -2987,9 +2623,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join MERGEJOIN[89][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 2' is a cross product
-Warning: Shuffle Join MERGEJOIN[90][tables = [$hdt$_4, $hdt$_5]] in Stage 'Reducer 11' is a cross product
-Warning: Shuffle Join MERGEJOIN[91][tables = [$hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 12' is a cross product
+Warning: Shuffle Join MERGEJOIN[46][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: select * from part where p_brand <> (select min(p_brand) from part ) AND p_size IN (select (p_size) from part p where p.p_type = part.p_type ) AND p_size <> 340
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -3034,10 +2668,8 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
- Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
- Reducer 7 <- Map 6 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -3062,33 +2694,25 @@ STAGE PLANS:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 3354 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3354 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col2 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col2 (type: int)
- Statistics: Num rows: 26 Data size: 3354 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 6
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 208 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_partkey (type: int), p_size (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
+ Filter Operator
+ predicate: ((p_size = p_size) and (p_partkey = p_partkey)) (type: boolean)
+ Statistics: Num rows: 6 Data size: 774 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: p_partkey (type: int), p_size (type: int), p_name (type: string)
+ outputColumnNames: p_partkey, p_size, p_name
+ Statistics: Num rows: 6 Data size: 774 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: count(p_name)
+ keys: p_partkey (type: int), p_size (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 3 Data size: 48 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 3 Data size: 48 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col2 (type: bigint)
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -3119,62 +2743,22 @@ STAGE PLANS:
Reducer 4
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int), _col2 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
- outputColumnNames: _col1, _col3, _col4
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: _col3 (type: int), _col4 (type: int), _col1 (type: string)
- outputColumnNames: _col3, _col4, _col1
- Statistics: Num rows: 1 Data size: 129 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(_col1)
- keys: _col3 (type: int), _col4 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: bigint)
- Reducer 5
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: count(VALUE._col0)
keys: KEY._col0 (type: int), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 3 Data size: 48 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col2 (type: bigint), true (type: boolean), _col0 (type: int), _col1 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 3 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col2 (type: int), _col3 (type: int)
sort order: ++
Map-reduce partition columns: _col2 (type: int), _col3 (type: int)
- Statistics: Num rows: 1 Data size: 20 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 3 Data size: 60 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: bigint), _col1 (type: boolean)
- Reducer 7
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int), KEY._col1 (type: int)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 13 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -4188,11 +3772,9 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 6 (SIMPLE_EDGE)
- Reducer 4 <- Map 3 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 5 <- Map 9 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
- Reducer 6 <- Reducer 5 (SIMPLE_EDGE)
- Reducer 8 <- Map 7 (SIMPLE_EDGE)
+ Reducer 2 <- Map 1 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+ Reducer 4 <- Map 3 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
+ Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -4215,58 +3797,41 @@ STAGE PLANS:
Map 3
Map Operator Tree:
TableScan
- alias: pp
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ alias: p
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
predicate: p_type is not null (type: boolean)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: p_type (type: string), p_size (type: int)
+ expressions: p_name (type: string), p_type (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col1 (type: int)
+ key expressions: _col1 (type: string)
sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 7
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_size (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 9
+ Map 6
Map Operator Tree:
TableScan
- alias: p
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
+ alias: pp
+ Statistics: Num rows: 26 Data size: 2808 Basic stats: COMPLETE Column stats: COMPLETE
Filter Operator
- predicate: p_type is not null (type: boolean)
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
+ predicate: ((p_size = p_size) and p_type is not null) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
- expressions: p_name (type: string), p_type (type: string)
+ expressions: p_type (type: string), p_size (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
- key expressions: _col1 (type: string)
+ key expressions: _col0 (type: string)
sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 5850 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: string)
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 13 Data size: 1404 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: int)
Execution mode: llap
LLAP IO: no inputs
Reducer 2
@@ -4301,71 +3866,45 @@ STAGE PLANS:
condition map:
Inner Join 0 to 1
keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 18 Data size: 1944 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 18 Data size: 1944 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col2 (type: int)
+ 0 _col1 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col3
+ Statistics: Num rows: 12 Data size: 1500 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ expressions: _col3 (type: int), _col0 (type: string)
+ outputColumnNames: _col3, _col0
+ Statistics: Num rows: 12 Data size: 1500 Basic stats: COMPLETE Column stats: COMPLETE
+ Group By Operator
+ aggregations: max(_col0)
+ keys: _col3 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 1128 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 6 Data size: 1128 Basic stats: COMPLETE Column stats: COMPLETE
+ value expressions: _col1 (type: string)
Reducer 5
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 16 Data size: 2000 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: max(_col3)
- keys: _col2 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1504 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 8 Data size: 1504 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col1 (type: string)
- Reducer 6
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: max(VALUE._col0)
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1504 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 6 Data size: 1128 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: _col1 (type: string), true (type: boolean), _col0 (type: int)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 8 Data size: 1536 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 6 Data size: 1152 Basic stats: COMPLETE Column stats: COMPLETE
Reduce Output Operator
key expressions: _col2 (type: int)
sort order: +
Map-reduce partition columns: _col2 (type: int)
- Statistics: Num rows: 8 Data size: 1536 Basic stats: COMPLETE Column stats: COMPLETE
+ Statistics: Num rows: 6 Data size: 1152 Basic stats: COMPLETE Column stats: COMPLETE
value expressions: _col0 (type: string), _col1 (type: boolean)
- Reducer 8
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 13 Data size: 52 Basic stats: COMPLETE Column stats: COMPLETE
Stage: Stage-0
Fetch Operator
@@ -4708,8 +4247,7 @@ POSTHOOK: Input: default@part_null
78486 almond azure blanched chiffon midnight Manufacturer#5 Brand#52 LARGE BRUSHED BRASS 23 MED BAG 1464.48 hely blith
155733 almond antique sky peru orange Manufacturer#5 Brand#53 SMALL PLATED BRASS 2 WRAP DRUM 1788.73 furiously. bra
15103 almond aquamarine dodger light gainsboro Manufacturer#5 Brand#53 ECONOMY BURNISHED STEEL 46 LG PACK 1018.1 packages hinder carefu
-Warning: Shuffle Join MERGEJOIN[78][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
-Warning: Shuffle Join MERGEJOIN[79][tables = [$hdt$_2, $hdt$_3]] in Stage 'Reducer 13' is a cross product
+Warning: Shuffle Join MERGEJOIN[39][tables = [$hdt$_0, $hdt$_1]] in Stage 'Reducer 2' is a cross product
PREHOOK: query: explain select * from part_null where p_brand NOT IN (select p_name from part) AND p_name NOT LIKE (select min(p_name) from part_null pp where part_null.p_type = pp.p_type)
PREHOOK: type: QUERY
POSTHOOK: query: explain select * from part_null where p_brand NOT IN (select p_name from part) AND p_name NOT LIKE (select min(p_name) from part_null pp where part_null.p_type = pp.p_type)
@@ -4723,16 +4261,10 @@ STAGE PLANS:
Tez
#### A masked pattern was here ####
Edges:
- Reducer 10 <- Map 9 (SIMPLE_EDGE), Reducer 15 (SIMPLE_EDGE)
- Reducer 11 <- Reducer 10 (SIMPLE_EDGE)
- Reducer 13 <- Map 12 (CUSTOM_SIMPLE_EDGE), Reducer 17 (CUSTOM_SIMPLE_EDGE)
- Reducer 14 <- Reducer 13 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
- Reducer 15 <- Reducer 14 (SIMPLE_EDGE)
- Reducer 17 <- Map 16 (CUSTOM_SIMPLE_EDGE)
- Reducer 19 <- Map 18 (SIMPLE_EDGE)
+ Reducer 10 <- Map 9 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (CUSTOM_SIMPLE_EDGE), Reducer 6 (CUSTOM_SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
- Reducer 4 <- Reducer 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
+ Reducer 4 <- Reducer 10 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
Reducer 6 <- Map 5 (CUSTOM_SIMPLE_EDGE)
Reducer 8 <- Map 7 (SIMPLE_EDGE)
#### A masked pattern was here ####
@@ -4752,63 +4284,11 @@ STAGE PLANS:
value expressions: _col0 (type: int), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: int), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Execution mode: llap
LLAP IO: no inputs
- Map 12
+ Map 5
Map Operator Tree:
TableScan
- alias: part_null
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col1 (type: string)
- Execution mode: llap
- LLAP IO: no inputs
- Map 16
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- expressions: p_partkey (type: int)
- outputColumnNames: p_partkey
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- aggregations: count(p_partkey)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- value expressions: _col0 (type: bigint)
- Execution mode: llap
- LLAP IO: no inputs
- Map 18
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: COMPLETE
- Execution mode: llap
- LLAP IO: no inputs
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
+ alias: part
+ Statistics: Num rows: 26 Data size: 3146 Basic stats: COMPLETE Column stats: COMPLETE
Select Operator
expressions: p_name (type: string)
outputColumnNames: p_name
@@ -4846,141 +4326,46 @@ STAGE PLANS:
TableScan
alias: pp
Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_name (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 16 Data size: 3256 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 8 Data size: 1628 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: p_type (type: string), p_name (type: string)
+ outputColumnNames: p_type, p_name
+ Statistics: Num rows: 8 Data size: 1628 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: min(p_name)
+ keys: p_type (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 8 Data size: 1628 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 8 Data size: 1628 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: string)
Execution mode: llap
LLAP IO: no inputs
Reducer 10
Execution mode: llap
Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: string)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(_col0)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 17 Data size: 3581 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- Reducer 11
- Execution mode: llap
- Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 1685 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 814 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), true (type: boolean), _col0 (type: string)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 8 Data size: 1685 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 814 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: string)
sort order: +
Map-reduce partition columns: _col2 (type: string)
- Statistics: Num rows: 8 Data size: 1685 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 814 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string), _col1 (type: boolean)
- Reducer 13
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 16 Data size: 3400 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 16 Data size: 3400 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- Reducer 14
- Execution mode: llap
- Reduce Operator Tree:
- Merge Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 17 Data size: 3740 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 17 Data size: 3740 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 17 Data size: 3740 Basic stats: COMPLETE Column stats: NONE
- Reducer 15
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 8 Data size: 1760 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 1760 Basic stats: COMPLETE Column stats: NONE
- Reducer 17
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
- Reducer 19
- Execution mode: llap
- Reduce Operator Tree:
- Group By Operator
<TRUNCATED>
[03/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out b/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
index f071824..7687cff 100644
--- a/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
+++ b/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
@@ -227,44 +227,13 @@ PLAN VECTORIZATION:
enabledConditionsMet: [hive.vectorized.execution.enabled IS true]
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-2 depends on stages: Stage-3, Stage-4
+ Stage-2 is a root stage
Stage-1 depends on stages: Stage-2
- Stage-4 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-3
- Spark
-#### A masked pattern was here ####
- Vertices:
- Map 4
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (l_shipmode = 'AIR') (type: boolean)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_orderkey (type: int), l_linenumber (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Spark HashTable Sink Operator
- keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
- Local Work:
- Map Reduce Local Work
-
Stage: Stage-2
Spark
- Edges:
- Reducer 8 <- Map 7 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 3
@@ -289,111 +258,37 @@ STAGE PLANS:
inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
Local Work:
Map Reduce Local Work
- Map 7
+ Map 4
Map Operator Tree:
TableScan
- alias: li
+ alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((l_shipmode = 'AIR') and (l_linenumber = l_linenumber)) (type: boolean)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: l_partkey (type: int), l_linenumber (type: int)
+ expressions: l_orderkey (type: int), l_linenumber (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col2
- input vertices:
- 0 Reducer 6
- Statistics: Num rows: 110 Data size: 13198 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: _col2 (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 110 Data size: 13198 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 110 Data size: 13198 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ Group By Vectorization:
+ vectorOutput: false
+ native: false
+ projectedOutputColumns: null
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ keys:
+ 0 _col1 (type: int), _col4 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
Map Vectorization:
enabled: false
enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
Local Work:
Map Reduce Local Work
- Reducer 8
- Execution mode: vectorized
- Local Work:
- Map Reduce Local Work
- Reduce Vectorization:
- enabled: true
- enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine spark IN [tez, spark] IS true
- groupByVectorOutput: true
- allNative: false
- usesVectorUDFAdaptor: false
- vectorized: true
- Reduce Operator Tree:
- Group By Operator
- Group By Vectorization:
- className: VectorGroupByOperator
- vectorOutput: true
- keyExpressions: col 0
- native: false
- projectedOutputColumns: []
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 55 Data size: 6599 Basic stats: COMPLETE Column stats: NONE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- Map Join Vectorization:
- className: VectorMapJoinInnerLongOperator
- native: true
- nativeConditionsMet: hive.mapjoin.optimized.hashtable IS true, hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine spark IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Small table vectorizes IS true, Optimized Table and Supports Key Types IS true
- outputColumnNames: _col0, _col3
- input vertices:
- 0 Map 4
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Select Vectorization:
- className: VectorSelectOperator
- native: true
- projectedOutputColumns: [1, 0]
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- Group By Vectorization:
- className: VectorGroupByOperator
- vectorOutput: true
- keyExpressions: col 1, col 0
- native: false
- projectedOutputColumns: []
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 60 Data size: 7258 Basic stats: COMPLETE Column stats: NONE
- Spark HashTable Sink Operator
- Spark Hash Table Sink Vectorization:
- className: VectorSparkHashTableSinkOperator
- native: true
- keys:
- 0 _col1 (type: int), _col4 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
Stage: Stage-1
Spark
@@ -476,8 +371,8 @@ STAGE PLANS:
nativeConditionsMet: hive.mapjoin.optimized.hashtable IS true, hive.vectorized.execution.mapjoin.native.enabled IS true, hive.execution.engine spark IN [tez, spark] IS true, One MapJoin Condition IS true, No nullsafe IS true, Small table vectorizes IS true, Optimized Table and Supports Key Types IS true
outputColumnNames: _col0, _col3
input vertices:
- 1 Reducer 8
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
+ 1 Map 4
+ Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col0 (type: int), _col3 (type: int)
outputColumnNames: _col0, _col1
@@ -485,81 +380,18 @@ STAGE PLANS:
className: VectorSelectOperator
native: true
projectedOutputColumns: [0, 2]
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
File Sink Vectorization:
className: VectorFileSinkOperator
native: false
- Statistics: Num rows: 66 Data size: 7983 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-4
- Spark
- Edges:
- Reducer 6 <- Map 5 (GROUP, 2)
-#### A masked pattern was here ####
- Vertices:
- Map 5
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- Group By Vectorization:
- vectorOutput: false
- native: false
- projectedOutputColumns: null
- keys: l_partkey (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Map Vectorization:
- enabled: false
- enabledConditionsNotMet: hive.vectorized.use.vector.serde.deserialize IS false
- inputFileFormats: org.apache.hadoop.mapred.TextInputFormat
- Reducer 6
- Execution mode: vectorized
- Local Work:
- Map Reduce Local Work
- Reduce Vectorization:
- enabled: true
- enableConditionsMet: hive.vectorized.execution.reduce.enabled IS true, hive.execution.engine spark IN [tez, spark] IS true
- groupByVectorOutput: true
- allNative: false
- usesVectorUDFAdaptor: false
- vectorized: true
- Reduce Operator Tree:
- Group By Operator
- Group By Vectorization:
- className: VectorGroupByOperator
- vectorOutput: true
- keyExpressions: col 0
- native: false
- projectedOutputColumns: []
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Spark HashTable Sink Operator
- Spark Hash Table Sink Vectorization:
- className: VectorSparkHashTableSinkOperator
- native: true
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
-
Stage: Stage-0
Fetch Operator
limit: -1
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_exists.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_exists.q.out b/ql/src/test/results/clientpositive/subquery_exists.q.out
index 412c579..b271529 100644
--- a/ql/src/test/results/clientpositive/subquery_exists.q.out
+++ b/ql/src/test/results/clientpositive/subquery_exists.q.out
@@ -17,94 +17,10 @@ where exists
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-2 depends on stages: Stage-3
- Stage-1 depends on stages: Stage-2
+ Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: key, value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: key (type: string), value (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (value > 'val_9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string), _col1 (type: string)
- 1 _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col3
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col3 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-1
Map Reduce
Map Operator Tree:
@@ -121,11 +37,25 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ alias: a
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((value = value) and (key = key) and (value > 'val_9')) (type: boolean)
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: string), value (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -277,91 +207,10 @@ where exists
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-2 depends on stages: Stage-3
- Stage-1 depends on stages: Stage-2
+ Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-1
Map Reduce
Map Operator Tree:
@@ -379,11 +228,25 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string)
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
+ alias: a
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (value = value) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: value (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -392,10 +255,10 @@ STAGE PLANS:
0 _col1 (type: string)
1 _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 605 Data size: 6427 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_exists_having.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_exists_having.q.out b/ql/src/test/results/clientpositive/subquery_exists_having.q.out
index 51c98ed..2c41ff6 100644
--- a/ql/src/test/results/clientpositive/subquery_exists_having.q.out
+++ b/ql/src/test/results/clientpositive/subquery_exists_having.q.out
@@ -20,9 +20,7 @@ having exists
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-1 is a root stage
- Stage-2 depends on stages: Stage-1, Stage-3
- Stage-4 is a root stage
- Stage-3 depends on stages: Stage-4
+ Stage-2 depends on stages: Stage-1
Stage-0 depends on stages: Stage-2
STAGE PLANS:
@@ -73,11 +71,25 @@ STAGE PLANS:
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ alias: a
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((key = key) and (value > 'val_9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: key (type: string)
+ outputColumnNames: _col0
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -95,93 +107,6 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string)
- outputColumnNames: key
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: key (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: a
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (value > 'val_9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col2
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-0
Fetch Operator
limit: -1
@@ -267,45 +192,33 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (value > 'val_9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((key = key) and (value > 'val_9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string)
- outputColumnNames: key
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: key (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Demux Operator
- Statistics: Num rows: 1166 Data size: 12387 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 583 Data size: 6193 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 583 Data size: 6193 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 291 Data size: 3091 Basic stats: COMPLETE Column stats: NONE
Mux Operator
- Statistics: Num rows: 583 Data size: 6193 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 874 Data size: 9284 Basic stats: COMPLETE Column stats: NONE
Join Operator
condition map:
Left Semi Join 0 to 1
@@ -321,78 +234,23 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 583 Data size: 6193 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string)
- mode: complete
- outputColumnNames: _col0
- Statistics: Num rows: 291 Data size: 3091 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 1457 Data size: 15478 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col2
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 583 Data size: 6193 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Mux Operator
- Statistics: Num rows: 1457 Data size: 15478 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 874 Data size: 9284 Basic stats: COMPLETE Column stats: NONE
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Semi Join 0 to 1
keys:
0 _col0 (type: string)
1 _col0 (type: string)
- outputColumnNames: _col2
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string)
- outputColumnNames: _col0
+ File Output Operator
+ compressed: false
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 583 Data size: 6193 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/subquery_in_having.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/subquery_in_having.q.out b/ql/src/test/results/clientpositive/subquery_in_having.q.out
index 2d44a15..7473928 100644
--- a/ql/src/test/results/clientpositive/subquery_in_having.q.out
+++ b/ql/src/test/results/clientpositive/subquery_in_having.q.out
@@ -1091,100 +1091,15 @@ group by key, value
having count(*) in (select count(*) from src s1 where s1.key > '9' group by s1.key )
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-1 depends on stages: Stage-3
- Stage-7 depends on stages: Stage-1, Stage-5 , consists of Stage-8, Stage-2
- Stage-8 has a backup stage: Stage-2
- Stage-6 depends on stages: Stage-8
+ Stage-1 is a root stage
+ Stage-6 depends on stages: Stage-1, Stage-4 , consists of Stage-7, Stage-2
+ Stage-7 has a backup stage: Stage-2
+ Stage-5 depends on stages: Stage-7
Stage-2
- Stage-5 is a root stage
- Stage-0 depends on stages: Stage-6, Stage-2
+ Stage-4 is a root stage
+ Stage-0 depends on stages: Stage-5, Stage-2
STAGE PLANS:
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: src
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Demux Operator
- Statistics: Num rows: 1000 Data size: 10624 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 1500 Data size: 15936 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 1500 Data size: 15936 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-1
Map Reduce
Map Operator Tree:
@@ -1201,16 +1116,26 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
+ alias: src
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (value = value) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Demux Operator
- Statistics: Num rows: 501 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 750 Data size: 7968 Basic stats: COMPLETE Column stats: NONE
Mux Operator
- Statistics: Num rows: 751 Data size: 7962 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1125 Data size: 11952 Basic stats: COMPLETE Column stats: NONE
Join Operator
condition map:
Inner Join 0 to 1
@@ -1240,9 +1165,9 @@ STAGE PLANS:
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2650 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 375 Data size: 3984 Basic stats: COMPLETE Column stats: NONE
Mux Operator
- Statistics: Num rows: 751 Data size: 7962 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1125 Data size: 11952 Basic stats: COMPLETE Column stats: NONE
Join Operator
condition map:
Inner Join 0 to 1
@@ -1269,10 +1194,10 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-7
+ Stage: Stage-6
Conditional Operator
- Stage: Stage-8
+ Stage: Stage-7
Map Reduce Local Work
Alias -> Map Local Tables:
$INTNAME1
@@ -1286,7 +1211,7 @@ STAGE PLANS:
0 _col2 (type: bigint)
1 _col0 (type: bigint)
- Stage: Stage-6
+ Stage: Stage-5
Map Reduce
Map Operator Tree:
TableScan
@@ -1341,7 +1266,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Stage: Stage-5
+ Stage: Stage-4
Map Reduce
Map Operator Tree:
TableScan
@@ -1934,12 +1859,11 @@ POSTHOOK: type: QUERY
POSTHOOK: Output: default@src_null
POSTHOOK: Lineage: src_null.key SIMPLE [(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col1, type:string, comment:), ]
POSTHOOK: Lineage: src_null.value SIMPLE [(values__tmp__table__1)values__tmp__table__1.FieldSchema(name:tmp_values_col2, type:string, comment:), ]
-Warning: Map Join MAPJOIN[235][bigTable=?] in task 'Stage-1:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[236][bigTable=?] in task 'Stage-8:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[237][bigTable=?] in task 'Stage-9:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[238][bigTable=?] in task 'Stage-14:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[239][bigTable=?] in task 'Stage-17:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[240][bigTable=?] in task 'Stage-21:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[185][bigTable=?] in task 'Stage-1:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[186][bigTable=?] in task 'Stage-8:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[187][bigTable=?] in task 'Stage-9:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[188][bigTable=?] in task 'Stage-14:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[189][bigTable=?] in task 'Stage-15:MAPRED' is a cross product
PREHOOK: query: explain
select key, value, count(*)
from src_null b
@@ -1956,36 +1880,27 @@ having count(*) not in (select count(*) from src_null s1 where s1.key > '9' and
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
Stage-6 is a root stage
- Stage-37 depends on stages: Stage-6
- Stage-1 depends on stages: Stage-37
+ Stage-27 depends on stages: Stage-6
+ Stage-1 depends on stages: Stage-27
Stage-2 depends on stages: Stage-1
- Stage-28 depends on stages: Stage-2, Stage-8 , consists of Stage-36, Stage-3
- Stage-36 has a backup stage: Stage-3
- Stage-27 depends on stages: Stage-36
- Stage-26 depends on stages: Stage-3, Stage-16, Stage-27 , consists of Stage-35, Stage-4
- Stage-35 has a backup stage: Stage-4
- Stage-25 depends on stages: Stage-35
- Stage-4
+ Stage-20 depends on stages: Stage-2, Stage-8 , consists of Stage-26, Stage-3
+ Stage-26 has a backup stage: Stage-3
+ Stage-19 depends on stages: Stage-26
+ Stage-4 depends on stages: Stage-3, Stage-14, Stage-19
Stage-3
Stage-12 is a root stage
- Stage-39 depends on stages: Stage-12
- Stage-9 depends on stages: Stage-39
+ Stage-29 depends on stages: Stage-12
+ Stage-9 depends on stages: Stage-29
Stage-10 depends on stages: Stage-9
- Stage-38 depends on stages: Stage-10
- Stage-8 depends on stages: Stage-38
- Stage-20 is a root stage
- Stage-41 depends on stages: Stage-20
- Stage-17 depends on stages: Stage-41
- Stage-18 depends on stages: Stage-17
- Stage-40 depends on stages: Stage-18
- Stage-14 depends on stages: Stage-40
- Stage-15 depends on stages: Stage-14
- Stage-16 depends on stages: Stage-15, Stage-22
- Stage-24 is a root stage
- Stage-42 depends on stages: Stage-24
- Stage-21 depends on stages: Stage-42
- Stage-22 depends on stages: Stage-21
- Stage-0 depends on stages: Stage-25, Stage-4
+ Stage-28 depends on stages: Stage-10
+ Stage-8 depends on stages: Stage-28
+ Stage-18 is a root stage
+ Stage-31 depends on stages: Stage-18
+ Stage-15 depends on stages: Stage-31
+ Stage-16 depends on stages: Stage-15
+ Stage-30 depends on stages: Stage-16
+ Stage-14 depends on stages: Stage-30
+ Stage-0 depends on stages: Stage-4
STAGE PLANS:
Stage: Stage-6
@@ -2021,7 +1936,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-37
+ Stage: Stage-27
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_0:$hdt$_1:$hdt$_1:src_null
@@ -2184,10 +2099,10 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-28
+ Stage: Stage-20
Conditional Operator
- Stage: Stage-36
+ Stage: Stage-26
Map Reduce Local Work
Alias -> Map Local Tables:
$INTNAME1
@@ -2201,7 +2116,7 @@ STAGE PLANS:
0 _col1 (type: string)
1 _col0 (type: string)
- Stage: Stage-27
+ Stage: Stage-19
Map Reduce
Map Operator Tree:
TableScan
@@ -2222,92 +2137,89 @@ STAGE PLANS:
Local Work:
Map Reduce Local Work
- Stage: Stage-26
- Conditional Operator
-
- Stage: Stage-35
- Map Reduce Local Work
- Alias -> Map Local Tables:
- $INTNAME1
- Fetch Operator
- limit: -1
- Alias -> Map Local Operator Tree:
- $INTNAME1
- TableScan
- HashTable Sink Operator
- keys:
- 0 _col1 (type: string), _col2 (type: bigint)
- 1 _col1 (type: string), _col3 (type: bigint)
-
- Stage: Stage-25
+ Stage: Stage-4
Map Reduce
Map Operator Tree:
TableScan
- Map Join Operator
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col2 (type: bigint)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col2 (type: bigint)
+ Statistics: Num rows: 2 Data size: 443 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: string), _col4 (type: bigint), _col5 (type: bigint)
+ TableScan
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: bigint)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: bigint)
+ Statistics: Num rows: 4 Data size: 806 Basic stats: COMPLETE Column stats: NONE
+ Reduce Operator Tree:
+ Demux Operator
+ Statistics: Num rows: 6 Data size: 1249 Basic stats: COMPLETE Column stats: NONE
+ Mux Operator
+ Statistics: Num rows: 7 Data size: 1457 Basic stats: COMPLETE Column stats: NONE
+ Join Operator
condition map:
Left Outer Join0 to 1
keys:
0 _col1 (type: string), _col2 (type: bigint)
- 1 _col1 (type: string), _col3 (type: bigint)
+ 1 _col1 (type: string), _col0 (type: bigint)
outputColumnNames: _col0, _col1, _col2, _col4, _col5, _col8
- Statistics: Num rows: 2 Data size: 487 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
Filter Operator
- predicate: CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col8 is not null) THEN (false) WHEN (_col2 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END (type: boolean)
- Statistics: Num rows: 1 Data size: 243 Basic stats: COMPLETE Column stats: NONE
+ predicate: (not CASE WHEN ((_col4 = 0)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col2 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (true) ELSE (false) END) (type: boolean)
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
Select Operator
expressions: _col0 (type: string), _col1 (type: string), _col2 (type: bigint)
outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 243 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 1 Data size: 243 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Local Work:
- Map Reduce Local Work
-
- Stage: Stage-4
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col1 (type: string), _col2 (type: bigint)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col2 (type: bigint)
- Statistics: Num rows: 2 Data size: 443 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col4 (type: bigint), _col5 (type: bigint)
- TableScan
- Reduce Output Operator
- key expressions: _col1 (type: string), _col3 (type: bigint)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col3 (type: bigint)
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- value expressions: _col2 (type: boolean)
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col1 (type: string), _col2 (type: bigint)
- 1 _col1 (type: string), _col3 (type: bigint)
- outputColumnNames: _col0, _col1, _col2, _col4, _col5, _col8
- Statistics: Num rows: 2 Data size: 487 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: CASE WHEN ((_col4 = 0)) THEN (true) WHEN (_col4 is null) THEN (true) WHEN (_col8 is not null) THEN (false) WHEN (_col2 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (false) ELSE (true) END (type: boolean)
- Statistics: Num rows: 1 Data size: 243 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: KEY._col0 (type: string), KEY._col1 (type: bigint)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 3 Data size: 624 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: string), _col1 (type: string), _col2 (type: bigint)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 243 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 1 Data size: 243 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ expressions: _col1 (type: bigint), _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 3 Data size: 624 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (_col0 = _col0) (type: boolean)
+ Statistics: Num rows: 1 Data size: 208 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: bigint), _col1 (type: string), true (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 1 Data size: 208 Basic stats: COMPLETE Column stats: NONE
+ Mux Operator
+ Statistics: Num rows: 7 Data size: 1457 Basic stats: COMPLETE Column stats: NONE
+ Join Operator
+ condition map:
+ Left Outer Join0 to 1
+ keys:
+ 0 _col1 (type: string), _col2 (type: bigint)
+ 1 _col1 (type: string), _col0 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2, _col4, _col5, _col8
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
+ Filter Operator
+ predicate: (not CASE WHEN ((_col4 = 0)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col2 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (true) ELSE (false) END) (type: boolean)
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string), _col2 (type: bigint)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-3
Map Reduce
@@ -2375,7 +2287,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-39
+ Stage: Stage-29
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_1:$hdt$_1:$hdt$_2:$hdt$_2:$hdt$_3:$hdt$_3:src_null
@@ -2539,7 +2451,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-38
+ Stage: Stage-28
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_1:$hdt$_1:$hdt$_1:s1
@@ -2619,7 +2531,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-20
+ Stage: Stage-18
Map Reduce
Map Operator Tree:
TableScan
@@ -2652,7 +2564,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-41
+ Stage: Stage-31
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_2:$hdt$_2:$hdt$_2:$hdt$_2:$hdt$_3:$hdt$_3:$hdt$_4:$hdt$_4:src_null
@@ -2672,7 +2584,7 @@ STAGE PLANS:
0
1
- Stage: Stage-17
+ Stage: Stage-15
Map Reduce
Map Operator Tree:
TableScan
@@ -2785,7 +2697,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-18
+ Stage: Stage-16
Map Reduce
Map Operator Tree:
TableScan
@@ -2816,7 +2728,7 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-40
+ Stage: Stage-30
Map Reduce Local Work
Alias -> Map Local Tables:
$hdt$_2:$hdt$_2:$hdt$_2:$hdt$_2:$hdt$_2:s1
@@ -2880,11 +2792,11 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 4 Data size: 806 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col2 (type: bigint), _col0 (type: string)
- outputColumnNames: _col2, _col1
+ expressions: _col0 (type: string), _col2 (type: bigint)
+ outputColumnNames: _col1, _col2
Statistics: Num rows: 4 Data size: 806 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col2 (type: bigint), _col1 (type: string)
+ keys: _col1 (type: string), _col2 (type: bigint)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 4 Data size: 806 Basic stats: COMPLETE Column stats: NONE
@@ -2895,302 +2807,17 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Stage: Stage-15
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: bigint), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: bigint), _col1 (type: string)
- Statistics: Num rows: 4 Data size: 806 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: bigint), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 2 Data size: 403 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: bigint), _col1 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 2 Data size: 403 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-16
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: bigint)
- sort order: +
- Map-reduce partition columns: _col0 (type: bigint)
- Statistics: Num rows: 2 Data size: 403 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string), _col2 (type: boolean)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: bigint)
- sort order: +
- Map-reduce partition columns: _col0 (type: bigint)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Reduce Operator Tree:
- Demux Operator
- Statistics: Num rows: 3 Data size: 403 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 4 Data size: 537 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: bigint)
- 1 _col0 (type: bigint)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Group By Operator
- keys: KEY._col0 (type: bigint)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 134 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 4 Data size: 537 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: bigint)
- 1 _col0 (type: bigint)
- outputColumnNames: _col1, _col2, _col3
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-24
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 58 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 58 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 58 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 58 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 29 Data size: 2910 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-42
- Map Reduce Local Work
- Alias -> Map Local Tables:
- $hdt$_2:$hdt$_3:$hdt$_3:$hdt$_4:$hdt$_4:src_null
- Fetch Operator
- limit: -1
- Alias -> Map Local Operator Tree:
- $hdt$_2:$hdt$_3:$hdt$_3:$hdt$_4:$hdt$_4:src_null
- TableScan
- alias: src_null
- Statistics: Num rows: 58 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: _col0
- Statistics: Num rows: 58 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- HashTable Sink Operator
- keys:
- 0
- 1
-
- Stage: Stage-21
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 29 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 29 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 29 Data size: 5820 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- TableScan
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1682 Data size: 339242 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (_col0 <> _col1) (type: boolean)
- Statistics: Num rows: 1682 Data size: 339242 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 1682 Data size: 339242 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1682 Data size: 339242 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1682 Data size: 339242 Basic stats: COMPLETE Column stats: NONE
- Local Work:
- Map Reduce Local Work
- Reduce Operator Tree:
- Demux Operator
- Statistics: Num rows: 1711 Data size: 345062 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 2566 Data size: 517492 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col1, _col3
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Filter Operator
- predicate: _col3 is null (type: boolean)
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Group By Operator
- aggregations: count()
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 855 Data size: 172430 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), true (type: boolean)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 855 Data size: 172430 Basic stats: COMPLETE Column stats: NONE
- Mux Operator
- Statistics: Num rows: 2566 Data size: 517492 Basic stats: COMPLETE Column stats: NONE
- Join Operator
- condition map:
- Left Outer Join0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col1, _col3
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Filter Operator
- predicate: _col3 is null (type: boolean)
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE
- Group By Operator
- aggregations: count()
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-22
- Map Reduce
- Map Operator Tree:
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- value expressions: _col2 (type: bigint)
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Select Operator
- expressions: _col2 (type: bigint)
- outputColumnNames: _col2
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- Group By Operator
- keys: _col2 (type: bigint)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
-Warning: Map Join MAPJOIN[235][bigTable=?] in task 'Stage-1:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[236][bigTable=?] in task 'Stage-8:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[237][bigTable=?] in task 'Stage-9:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[238][bigTable=?] in task 'Stage-14:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[239][bigTable=?] in task 'Stage-17:MAPRED' is a cross product
-Warning: Map Join MAPJOIN[240][bigTable=?] in task 'Stage-21:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[185][bigTable=?] in task 'Stage-1:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[186][bigTable=?] in task 'Stage-8:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[187][bigTable=?] in task 'Stage-9:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[188][bigTable=?] in task 'Stage-14:MAPRED' is a cross product
+Warning: Map Join MAPJOIN[189][bigTable=?] in task 'Stage-15:MAPRED' is a cross product
PREHOOK: query: select key, value, count(*)
from src_null b
where NOT EXISTS (select key from src_null where src_null.value <> b.value)
[15/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
HIVE-15905 : Inefficient plan for correlated subqueries (Vineet Garg via Ashutosh Chauhan)
Signed-off-by: Ashutosh Chauhan <ha...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/hive/repo
Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/bddf5a7a
Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/bddf5a7a
Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/bddf5a7a
Branch: refs/heads/master
Commit: bddf5a7a974fcfc4f350123f561da5f4ddcf43e0
Parents: b14ef6d
Author: Vineet Garg <vg...@hortonworks.com>
Authored: Mon Feb 13 20:23:00 2017 -0800
Committer: Ashutosh Chauhan <ha...@apache.org>
Committed: Tue Feb 14 17:25:12 2017 -0800
----------------------------------------------------------------------
.../calcite/rules/HiveRelDecorrelator.java | 718 +--
.../queries/clientpositive/subquery_multi.q | 8 +-
.../clientpositive/constprog_partitioner.q.out | 117 +-
.../clientpositive/llap/explainuser_1.q.out | 684 +--
.../clientpositive/llap/subquery_exists.q.out | 169 +-
.../clientpositive/llap/subquery_in.q.out | 2591 +++--------
.../clientpositive/llap/subquery_multi.q.out | 2788 +++---------
.../clientpositive/llap/subquery_notin.q.out | 4222 +++++-------------
.../clientpositive/llap/subquery_scalar.q.out | 2679 +++--------
.../clientpositive/llap/subquery_views.q.out | 556 +--
.../llap/vector_mapjoin_reduce.q.out | 196 +-
.../results/clientpositive/perf/query1.q.out | 190 +-
.../results/clientpositive/perf/query16.q.out | 222 +-
.../results/clientpositive/perf/query30.q.out | 375 +-
.../results/clientpositive/perf/query6.q.out | 403 +-
.../results/clientpositive/perf/query69.q.out | 615 ++-
.../results/clientpositive/perf/query81.q.out | 375 +-
.../test/results/clientpositive/semijoin5.q.out | 138 +-
.../spark/constprog_partitioner.q.out | 87 +-
.../clientpositive/spark/subquery_exists.q.out | 167 +-
.../clientpositive/spark/subquery_in.q.out | 2406 +++-------
.../spark/vector_mapjoin_reduce.q.out | 216 +-
.../clientpositive/subquery_exists.q.out | 221 +-
.../clientpositive/subquery_exists_having.q.out | 232 +-
.../clientpositive/subquery_in_having.q.out | 641 +--
.../clientpositive/subquery_notexists.q.out | 209 +-
.../subquery_notexists_having.q.out | 223 +-
.../clientpositive/subquery_notin_having.q.out | 644 +--
.../subquery_unqualcolumnrefs.q.out | 687 +--
.../clientpositive/vector_mapjoin_reduce.q.out | 322 +-
30 files changed, 6219 insertions(+), 16882 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
----------------------------------------------------------------------
diff --git a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
index 9c26801..5f37fc1 100644
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
@@ -35,6 +35,7 @@ import org.apache.calcite.rel.core.Aggregate;
import org.apache.calcite.rel.core.AggregateCall;
import org.apache.calcite.rel.core.Correlate;
import org.apache.calcite.rel.core.CorrelationId;
+import org.apache.calcite.rel.core.Filter;
import org.apache.calcite.rel.core.JoinRelType;
import org.apache.calcite.rel.core.Project;
import org.apache.calcite.rel.core.RelFactories;
@@ -115,10 +116,12 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.NavigableMap;
+import java.util.Objects;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.TreeSet;
+import javax.annotation.Nonnull;
/**
* NOTE: this whole logic is replicated from Calcite's RelDecorrelator
@@ -211,7 +214,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
RelNode newRootRel = decorrelator.removeCorrelationViaRule(rootRel);
- if (!decorrelator.cm.mapCorVarToCorRel.isEmpty()) {
+ if (!decorrelator.cm.mapCorToCorRel.isEmpty()) {
newRootRel = decorrelator.decorrelate(newRootRel);
}
@@ -266,16 +269,16 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
private Function2<RelNode, RelNode, Void> createCopyHook() {
return new Function2<RelNode, RelNode, Void>() {
public Void apply(RelNode oldNode, RelNode newNode) {
- if (cm.mapRefRelToCorVar.containsKey(oldNode)) {
- cm.mapRefRelToCorVar.putAll(newNode,
- cm.mapRefRelToCorVar.get(oldNode));
+ if (cm.mapRefRelToCorRef.containsKey(oldNode)) {
+ cm.mapRefRelToCorRef.putAll(newNode,
+ cm.mapRefRelToCorRef.get(oldNode));
}
if (oldNode instanceof LogicalCorrelate
&& newNode instanceof LogicalCorrelate) {
LogicalCorrelate oldCor = (LogicalCorrelate) oldNode;
CorrelationId c = oldCor.getCorrelationId();
- if (cm.mapCorVarToCorRel.get(c) == oldNode) {
- cm.mapCorVarToCorRel.put(c, newNode);
+ if (cm.mapCorToCorRel.get(c) == oldNode) {
+ cm.mapCorToCorRel.put(c, newNode);
}
if (generatedCorRels.contains(oldNode)) {
@@ -355,7 +358,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
List<RelNode> newInputs = Lists.newArrayList();
for (int i = 0; i < oldInputs.size(); ++i) {
final Frame frame = getInvoke(oldInputs.get(i), rel);
- if (frame == null || !frame.corVarOutputPos.isEmpty()) {
+ if (frame == null || !frame.corDefOutputs.isEmpty()) {
// if input is not rewritten, or if it produces correlated
// variables, terminate rewrite
return null;
@@ -372,7 +375,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// the output position should not change since there are no corVars
// coming from below.
return register(rel, newRel, identityMap(rel.getRowType().getFieldCount()),
- ImmutableSortedMap.<Correlation, Integer>of());
+ ImmutableSortedMap.<CorDef, Integer>of());
}
/**
@@ -388,7 +391,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
//
// Sort itself should not reference cor vars.
- assert !cm.mapRefRelToCorVar.containsKey(rel);
+ assert !cm.mapRefRelToCorRef.containsKey(rel);
// Sort only references field positions in collations field.
// The collations field in the newRel now need to refer to the
@@ -406,7 +409,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
Mappings.TargetMapping mapping =
Mappings.target(
- frame.oldToNewOutputPos,
+ frame.oldToNewOutputs,
oldInput.getRowType().getFieldCount(),
newInput.getRowType().getFieldCount());
@@ -416,8 +419,8 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final RelNode newSort = HiveSortLimit.create(newInput, newCollation, rel.offset, rel.fetch);
// Sort does not change input ordering
- return register(rel, newSort, frame.oldToNewOutputPos,
- frame.corVarOutputPos);
+ return register(rel, newSort, frame.oldToNewOutputs,
+ frame.corDefOutputs);
}
/**
* Rewrite Sort.
@@ -432,7 +435,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
//
// Sort itself should not reference cor vars.
- assert !cm.mapRefRelToCorVar.containsKey(rel);
+ assert !cm.mapRefRelToCorRef.containsKey(rel);
// Sort only references field positions in collations field.
// The collations field in the newRel now need to refer to the
@@ -450,7 +453,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
Mappings.TargetMapping mapping =
Mappings.target(
- frame.oldToNewOutputPos,
+ frame.oldToNewOutputs,
oldInput.getRowType().getFieldCount(),
newInput.getRowType().getFieldCount());
@@ -460,8 +463,8 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final RelNode newSort = HiveSortLimit.create(newInput, newCollation, rel.offset, rel.fetch);
// Sort does not change input ordering
- return register(rel, newSort, frame.oldToNewOutputPos,
- frame.corVarOutputPos);
+ return register(rel, newSort, frame.oldToNewOutputs,
+ frame.corDefOutputs);
}
/**
@@ -493,7 +496,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
//
// Aggregate itself should not reference cor vars.
- assert !cm.mapRefRelToCorVar.containsKey(rel);
+ assert !cm.mapRefRelToCorRef.containsKey(rel);
final RelNode oldInput = rel.getInput();
final Frame frame = getInvoke(oldInput, rel);
@@ -502,16 +505,10 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
return null;
}
- //I think this is a bug in Calcite where Aggregate seems to always expect
- // correlated variable in nodes underneath it which is not true for queries such as
- // select p.empno, li.mgr from (select distinct empno as empno from emp) p join emp li on p.empno= li.empno where li.sal = 1
- // and li.deptno in (select deptno from emp where JOB = 'AIR' AND li.mgr=mgr)
-
- //assert !frame.corVarOutputPos.isEmpty();
final RelNode newInput = frame.r;
// map from newInput
- Map<Integer, Integer> mapNewInputToProjOutputPos = Maps.newHashMap();
+ Map<Integer, Integer> mapNewInputToProjOutputs = new HashMap<>();
final int oldGroupKeyCount = rel.getGroupSet().cardinality();
// Project projects the original expressions,
@@ -533,25 +530,25 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
omittedConstants.put(i, constant);
continue;
}
- int newInputPos = frame.oldToNewOutputPos.get(i);
+ int newInputPos = frame.oldToNewOutputs.get(i);
projects.add(RexInputRef.of2(newInputPos, newInputOutput));
- mapNewInputToProjOutputPos.put(newInputPos, newPos);
+ mapNewInputToProjOutputs.put(newInputPos, newPos);
newPos++;
}
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos = new TreeMap<>();
- if (!frame.corVarOutputPos.isEmpty()) {
+ final SortedMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!frame.corDefOutputs.isEmpty()) {
// If input produces correlated variables, move them to the front,
// right after any existing GROUP BY fields.
// Now add the corVars from the input, starting from
// position oldGroupKeyCount.
- for (Map.Entry<Correlation, Integer> entry
- : frame.corVarOutputPos.entrySet()) {
+ for (Map.Entry<CorDef, Integer> entry
+ : frame.corDefOutputs.entrySet()) {
projects.add(RexInputRef.of2(entry.getValue(), newInputOutput));
- mapCorVarToOutputPos.put(entry.getKey(), newPos);
- mapNewInputToProjOutputPos.put(entry.getValue(), newPos);
+ corDefOutputs.put(entry.getKey(), newPos);
+ mapNewInputToProjOutputs.put(entry.getValue(), newPos);
newPos++;
}
}
@@ -559,9 +556,9 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// add the remaining fields
final int newGroupKeyCount = newPos;
for (int i = 0; i < newInputOutput.size(); i++) {
- if (!mapNewInputToProjOutputPos.containsKey(i)) {
+ if (!mapNewInputToProjOutputs.containsKey(i)) {
projects.add(RexInputRef.of2(i, newInputOutput));
- mapNewInputToProjOutputPos.put(i, newPos);
+ mapNewInputToProjOutputs.put(i, newPos);
newPos++;
}
}
@@ -587,13 +584,13 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// newInput
Map<Integer, Integer> combinedMap = Maps.newHashMap();
- for (Integer oldInputPos : frame.oldToNewOutputPos.keySet()) {
+ for (Integer oldInputPos : frame.oldToNewOutputs.keySet()) {
combinedMap.put(oldInputPos,
- mapNewInputToProjOutputPos.get(
- frame.oldToNewOutputPos.get(oldInputPos)));
+ mapNewInputToProjOutputs.get(
+ frame.oldToNewOutputs.get(oldInputPos)));
}
- register(oldInput, newProject, combinedMap, mapCorVarToOutputPos);
+ register(oldInput, newProject, combinedMap, corDefOutputs);
// now it's time to rewrite the Aggregate
final ImmutableBitSet newGroupSet = ImmutableBitSet.range(newGroupKeyCount);
@@ -642,7 +639,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final List<RexNode> postProjects = new ArrayList<>(relBuilder.fields());
for (Map.Entry<Integer, RexLiteral> entry
: omittedConstants.descendingMap().entrySet()) {
- postProjects.add(entry.getKey() + frame.corVarOutputPos.size(),
+ postProjects.add(entry.getKey() + frame.corDefOutputs.size(),
entry.getValue());
}
relBuilder.project(postProjects);
@@ -650,7 +647,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// Aggregate does not change input ordering so corVars will be
// located at the same position as the input newProject.
- return register(rel, relBuilder.build(), combinedMap, mapCorVarToOutputPos);
+ return register(rel, relBuilder.build(), combinedMap, corDefOutputs);
}
public Frame getInvoke(RelNode r, RelNode parent) {
@@ -689,7 +686,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
//
// Aggregate itself should not reference cor vars.
- assert !cm.mapRefRelToCorVar.containsKey(rel);
+ assert !cm.mapRefRelToCorRef.containsKey(rel);
final RelNode oldInput = rel.getInput();
final Frame frame = getInvoke(oldInput, rel);
@@ -701,7 +698,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final RelNode newInput = frame.r;
// map from newInput
- Map<Integer, Integer> mapNewInputToProjOutputPos = Maps.newHashMap();
+ Map<Integer, Integer> mapNewInputToProjOutputs = new HashMap<>();
final int oldGroupKeyCount = rel.getGroupSet().cardinality();
// Project projects the original expressions,
@@ -723,25 +720,25 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
omittedConstants.put(i, constant);
continue;
}
- int newInputPos = frame.oldToNewOutputPos.get(i);
+ int newInputPos = frame.oldToNewOutputs.get(i);
projects.add(RexInputRef.of2(newInputPos, newInputOutput));
- mapNewInputToProjOutputPos.put(newInputPos, newPos);
+ mapNewInputToProjOutputs.put(newInputPos, newPos);
newPos++;
}
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos = new TreeMap<>();
- if (!frame.corVarOutputPos.isEmpty()) {
+ final SortedMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ if (!frame.corDefOutputs.isEmpty()) {
// If input produces correlated variables, move them to the front,
// right after any existing GROUP BY fields.
// Now add the corVars from the input, starting from
// position oldGroupKeyCount.
- for (Map.Entry<Correlation, Integer> entry
- : frame.corVarOutputPos.entrySet()) {
+ for (Map.Entry<CorDef, Integer> entry
+ : frame.corDefOutputs.entrySet()) {
projects.add(RexInputRef.of2(entry.getValue(), newInputOutput));
- mapCorVarToOutputPos.put(entry.getKey(), newPos);
- mapNewInputToProjOutputPos.put(entry.getValue(), newPos);
+ corDefOutputs.put(entry.getKey(), newPos);
+ mapNewInputToProjOutputs.put(entry.getValue(), newPos);
newPos++;
}
}
@@ -749,9 +746,9 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// add the remaining fields
final int newGroupKeyCount = newPos;
for (int i = 0; i < newInputOutput.size(); i++) {
- if (!mapNewInputToProjOutputPos.containsKey(i)) {
+ if (!mapNewInputToProjOutputs.containsKey(i)) {
projects.add(RexInputRef.of2(i, newInputOutput));
- mapNewInputToProjOutputPos.put(i, newPos);
+ mapNewInputToProjOutputs.put(i, newPos);
newPos++;
}
}
@@ -776,13 +773,13 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// newInput
Map<Integer, Integer> combinedMap = Maps.newHashMap();
- for (Integer oldInputPos : frame.oldToNewOutputPos.keySet()) {
+ for (Integer oldInputPos : frame.oldToNewOutputs.keySet()) {
combinedMap.put(oldInputPos,
- mapNewInputToProjOutputPos.get(
- frame.oldToNewOutputPos.get(oldInputPos)));
+ mapNewInputToProjOutputs.get(
+ frame.oldToNewOutputs.get(oldInputPos)));
}
- register(oldInput, newProject, combinedMap, mapCorVarToOutputPos);
+ register(oldInput, newProject, combinedMap, corDefOutputs);
// now it's time to rewrite the Aggregate
final ImmutableBitSet newGroupSet = ImmutableBitSet.range(newGroupKeyCount);
@@ -827,7 +824,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final List<RexNode> postProjects = new ArrayList<>(relBuilder.fields());
for (Map.Entry<Integer, RexLiteral> entry
: omittedConstants.descendingMap().entrySet()) {
- postProjects.add(entry.getKey() + frame.corVarOutputPos.size(),
+ postProjects.add(entry.getKey() + frame.corDefOutputs.size(),
entry.getValue());
}
relBuilder.project(postProjects);
@@ -835,7 +832,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// Aggregate does not change input ordering so corVars will be
// located at the same position as the input newProject.
- return register(rel, relBuilder.build(), combinedMap, mapCorVarToOutputPos);
+ return register(rel, relBuilder.build(), combinedMap, corDefOutputs);
}
}
@@ -862,16 +859,12 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// If this LogicalProject has correlated reference, create value generator
// and produce the correlated variables in the new output.
- if (cm.mapRefRelToCorVar.containsKey(rel)) {
- decorrelateInputWithValueGenerator(rel);
-
- // The old input should be mapped to the LogicalJoin created by
- // rewriteInputWithValueGenerator().
- frame = map.get(oldInput);
+ if (cm.mapRefRelToCorRef.containsKey(rel)) {
+ frame = decorrelateInputWithValueGenerator(rel);
}
// LogicalProject projects the original expressions
- final Map<Integer, Integer> mapOldToNewOutputPos = Maps.newHashMap();
+ final Map<Integer, Integer> mapOldToNewOutputs = new HashMap<>();
int newPos;
for (newPos = 0; newPos < oldProjects.size(); newPos++) {
projects.add(
@@ -879,39 +872,24 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
Pair.of(
decorrelateExpr(oldProjects.get(newPos)),
relOutput.get(newPos).getName()));
- mapOldToNewOutputPos.put(newPos, newPos);
+ mapOldToNewOutputs.put(newPos, newPos);
}
// Project any correlated variables the input wants to pass along.
- // There could be situation e.g. multiple correlated variables refering to
- // same outer variable, in which case Project will be created with multiple
- // fields with same name. Hive doesn't allow HiveProject with multiple fields
- // having same name. So to avoid that we keep a set of all fieldnames and
- // on encountering an existing one a new field/column name is generated
- final Set<String> corrFieldName = Sets.newHashSet();
- int pos = 0;
-
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos = new TreeMap<>();
- for (Map.Entry<Correlation, Integer> entry : frame.corVarOutputPos.entrySet()) {
- final RelDataTypeField field = frame.r.getRowType().getFieldList().get(entry.getValue());
- RexNode projectChild = (RexNode) new RexInputRef(entry.getValue(), field.getType());
- String fieldName = field.getName();
- if(corrFieldName.contains(fieldName))
- {
- fieldName = SemanticAnalyzer.getColumnInternalName(pos++);
- }
-
- projects.add(Pair.of(projectChild ,fieldName));
- corrFieldName.add(fieldName);
- mapCorVarToOutputPos.put(entry.getKey(), newPos);
+ final SortedMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) {
+ projects.add(
+ RexInputRef.of2(entry.getValue(),
+ frame.r.getRowType().getFieldList()));
+ corDefOutputs.put(entry.getKey(), newPos);
newPos++;
}
RelNode newProject = HiveProject.create(frame.r, Pair.left(projects), SqlValidatorUtil.uniquify(Pair.right(projects)));
- return register(rel, newProject, mapOldToNewOutputPos,
- mapCorVarToOutputPos);
+ return register(rel, newProject, mapOldToNewOutputs,
+ corDefOutputs);
}
}
/**
@@ -941,16 +919,12 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// If this LogicalProject has correlated reference, create value generator
// and produce the correlated variables in the new output.
- if (cm.mapRefRelToCorVar.containsKey(rel)) {
- decorrelateInputWithValueGenerator(rel);
-
- // The old input should be mapped to the LogicalJoin created by
- // rewriteInputWithValueGenerator().
- frame = map.get(oldInput);
+ if (cm.mapRefRelToCorRef.containsKey(rel)) {
+ frame = decorrelateInputWithValueGenerator(rel);
}
// LogicalProject projects the original expressions
- final Map<Integer, Integer> mapOldToNewOutputPos = Maps.newHashMap();
+ final Map<Integer, Integer> mapOldToNewOutputs = new HashMap<>();
int newPos;
for (newPos = 0; newPos < oldProjects.size(); newPos++) {
projects.add(
@@ -958,23 +932,23 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
Pair.of(
decorrelateExpr(oldProjects.get(newPos)),
relOutput.get(newPos).getName()));
- mapOldToNewOutputPos.put(newPos, newPos);
+ mapOldToNewOutputs.put(newPos, newPos);
}
// Project any correlated variables the input wants to pass along.
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos = new TreeMap<>();
- for (Map.Entry<Correlation, Integer> entry : frame.corVarOutputPos.entrySet()) {
+ final SortedMap<CorDef, Integer> corDefOutputs = new TreeMap<>();
+ for (Map.Entry<CorDef, Integer> entry : frame.corDefOutputs.entrySet()) {
projects.add(
RexInputRef.of2(entry.getValue(),
frame.r.getRowType().getFieldList()));
- mapCorVarToOutputPos.put(entry.getKey(), newPos);
+ corDefOutputs.put(entry.getKey(), newPos);
newPos++;
}
RelNode newProject = HiveProject.create(frame.r, Pair.left(projects), Pair.right(projects));
- return register(rel, newProject, mapOldToNewOutputPos,
- mapCorVarToOutputPos);
+ return register(rel, newProject, mapOldToNewOutputs,
+ corDefOutputs);
}
/**
@@ -988,17 +962,17 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
* @return RelNode the root of the resultant RelNode tree
*/
private RelNode createValueGenerator(
- Iterable<Correlation> correlations,
+ Iterable<CorRef> correlations,
int valueGenFieldOffset,
- SortedMap<Correlation, Integer> mapCorVarToOutputPos) {
- final Map<RelNode, List<Integer>> mapNewInputToOutputPos =
+ SortedMap<CorDef, Integer> corDefOutputs) {
+ final Map<RelNode, List<Integer>> mapNewInputToOutputs =
new HashMap<>();
final Map<RelNode, Integer> mapNewInputToNewOffset = new HashMap<>();
// Input provides the definition of a correlated variable.
// Add to map all the referenced positions (relative to each input rel).
- for (Correlation corVar : correlations) {
+ for (CorRef corVar : correlations) {
final int oldCorVarOffset = corVar.field;
final RelNode oldInput = getCorRel(corVar);
@@ -1007,21 +981,21 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
assert frame != null;
final RelNode newInput = frame.r;
- final List<Integer> newLocalOutputPosList;
- if (!mapNewInputToOutputPos.containsKey(newInput)) {
- newLocalOutputPosList = Lists.newArrayList();
+ final List<Integer> newLocalOutputs;
+ if (!mapNewInputToOutputs.containsKey(newInput)) {
+ newLocalOutputs = new ArrayList<>();
} else {
- newLocalOutputPosList =
- mapNewInputToOutputPos.get(newInput);
+ newLocalOutputs =
+ mapNewInputToOutputs.get(newInput);
}
- final int newCorVarOffset = frame.oldToNewOutputPos.get(oldCorVarOffset);
+ final int newCorVarOffset = frame.oldToNewOutputs.get(oldCorVarOffset);
// Add all unique positions referenced.
- if (!newLocalOutputPosList.contains(newCorVarOffset)) {
- newLocalOutputPosList.add(newCorVarOffset);
+ if (!newLocalOutputs.contains(newCorVarOffset)) {
+ newLocalOutputs.add(newCorVarOffset);
}
- mapNewInputToOutputPos.put(newInput, newLocalOutputPosList);
+ mapNewInputToOutputs.put(newInput, newLocalOutputs);
}
int offset = 0;
@@ -1031,24 +1005,24 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// To make sure the plan does not change in terms of join order,
// join these rels based on their occurrence in cor var list which
// is sorted.
- final Set<RelNode> joinedInputRelSet = Sets.newHashSet();
+ final Set<RelNode> joinedInputs = new HashSet<>();
RelNode r = null;
- for (Correlation corVar : correlations) {
+ for (CorRef corVar : correlations) {
final RelNode oldInput = getCorRel(corVar);
assert oldInput != null;
final RelNode newInput = map.get(oldInput).r;
assert newInput != null;
- if (!joinedInputRelSet.contains(newInput)) {
+ if (!joinedInputs.contains(newInput)) {
RelNode project =
RelOptUtil.createProject(
newInput,
- mapNewInputToOutputPos.get(newInput));
+ mapNewInputToOutputs.get(newInput));
RelNode distinct = RelOptUtil.createDistinctRel(project);
RelOptCluster cluster = distinct.getCluster();
- joinedInputRelSet.add(newInput);
+ joinedInputs.add(newInput);
mapNewInputToNewOffset.put(newInput, offset);
offset += distinct.getRowType().getFieldCount();
@@ -1067,32 +1041,29 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// the join output, leaving room for valueGenFieldOffset because
// valueGenerators are joined with the original left input of the rel
// referencing correlated variables.
- for (Correlation corVar : correlations) {
+ for (CorRef corRef : correlations) {
// The first input of a Correlator is always the rel defining
// the correlated variables.
- final RelNode oldInput = getCorRel(corVar);
+ final RelNode oldInput = getCorRel(corRef);
assert oldInput != null;
final Frame frame = map.get(oldInput);
final RelNode newInput = frame.r;
assert newInput != null;
- final List<Integer> newLocalOutputPosList =
- mapNewInputToOutputPos.get(newInput);
+ final List<Integer> newLocalOutputs =
+ mapNewInputToOutputs.get(newInput);
- final int newLocalOutputPos = frame.oldToNewOutputPos.get(corVar.field);
+ final int newLocalOutput = frame.oldToNewOutputs.get(corRef.field);
- // newOutputPos is the index of the cor var in the referenced
+ // newOutput is the index of the cor var in the referenced
// position list plus the offset of referenced position list of
// each newInput.
- final int newOutputPos =
- newLocalOutputPosList.indexOf(newLocalOutputPos)
+ final int newOutput =
+ newLocalOutputs.indexOf(newLocalOutput)
+ mapNewInputToNewOffset.get(newInput)
+ valueGenFieldOffset;
- if (mapCorVarToOutputPos.containsKey(corVar)) {
- assert mapCorVarToOutputPos.get(corVar) == newOutputPos;
- }
- mapCorVarToOutputPos.put(corVar, newOutputPos);
+ corDefOutputs.put(corRef.def(), newOutput);
}
return r;
@@ -1101,33 +1072,57 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
//this returns the source of corVar i.e. Rel which produces cor var
// value. Therefore it is always LogicalCorrelate's left input which is outer query
- private RelNode getCorRel(Correlation corVar) {
- final RelNode r = cm.mapCorVarToCorRel.get(corVar.corr);
-
+ private RelNode getCorRel(CorRef corVar) {
+ final RelNode r = cm.mapCorToCorRel.get(corVar.corr);
RelNode ret = r.getInput(0);
return ret;
}
- private void decorrelateInputWithValueGenerator(RelNode rel) {
+ private Frame decorrelateInputWithValueGenerator(RelNode rel) {
// currently only handles one input input
assert rel.getInputs().size() == 1;
RelNode oldInput = rel.getInput(0);
final Frame frame = map.get(oldInput);
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos =
- new TreeMap<>(frame.corVarOutputPos);
+ final SortedMap<CorDef, Integer> corDefOutputs =
+ new TreeMap<>(frame.corDefOutputs);
+
+ final Collection<CorRef> corVarList = cm.mapRefRelToCorRef.get(rel);
+
+ // Try to populate correlation variables using local fields.
+ // This means that we do not need a value generator.
+ if (rel instanceof Filter) {
+ SortedMap<CorDef, Integer> map = new TreeMap<>();
+ for (CorRef correlation : corVarList) {
+ final CorDef def = correlation.def();
+ if (corDefOutputs.containsKey(def) || map.containsKey(def)) {
+ continue;
+ }
+ try {
+ findCorrelationEquivalent(correlation, ((Filter) rel).getCondition());
+ } catch (Util.FoundOne e) {
+ map.put(def, (Integer) e.getNode());
+ }
+ }
+ // If all correlation variables are now satisfied, skip creating a value
+ // generator.
+ if (map.size() == corVarList.size()) {
+ map.putAll(frame.corDefOutputs);
+ return register(oldInput, frame.r,
+ frame.oldToNewOutputs, map);
+ }
+ }
- final Collection<Correlation> corVarList = cm.mapRefRelToCorVar.get(rel);
int leftInputOutputCount = frame.r.getRowType().getFieldCount();
- // can directly add positions into mapCorVarToOutputPos since join
+ // can directly add positions into corDefOutputs since join
// does not change the output ordering from the inputs.
RelNode valueGen =
createValueGenerator(
corVarList,
leftInputOutputCount,
- mapCorVarToOutputPos);
+ corDefOutputs);
RelNode join =
LogicalJoin.create(frame.r, valueGen, rexBuilder.makeLiteral(true),
@@ -1136,7 +1131,66 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// LogicalJoin or LogicalFilter does not change the old input ordering. All
// input fields from newLeftInput(i.e. the original input to the old
// LogicalFilter) are in the output and in the same position.
- register(oldInput, join, frame.oldToNewOutputPos, mapCorVarToOutputPos);
+ return register(oldInput, join, frame.oldToNewOutputs, corDefOutputs);
+ }
+
+
+ /** Finds a {@link RexInputRef} that is equivalent to a {@link CorRef},
+ * and if found, throws a {@link Util.FoundOne}. */
+ private void findCorrelationEquivalent(CorRef correlation, RexNode e)
+ throws Util.FoundOne {
+ switch (e.getKind()) {
+ case EQUALS:
+ final RexCall call = (RexCall) e;
+ final List<RexNode> operands = call.getOperands();
+ if (references(operands.get(0), correlation)
+ && operands.get(1) instanceof RexInputRef) {
+ throw new Util.FoundOne(((RexInputRef) operands.get(1)).getIndex());
+ }
+ if (references(operands.get(1), correlation)
+ && operands.get(0) instanceof RexInputRef) {
+ throw new Util.FoundOne(((RexInputRef) operands.get(0)).getIndex());
+ }
+ break;
+ case AND:
+ for (RexNode operand : ((RexCall) e).getOperands()) {
+ findCorrelationEquivalent(correlation, operand);
+ }
+ }
+ }
+
+ private boolean references(RexNode e, CorRef correlation) {
+ switch (e.getKind()) {
+ case CAST:
+ final RexNode operand = ((RexCall) e).getOperands().get(0);
+ if (isWidening(e.getType(), operand.getType())) {
+ return references(operand, correlation);
+ }
+ return false;
+ case FIELD_ACCESS:
+ final RexFieldAccess f = (RexFieldAccess) e;
+ if (f.getField().getIndex() == correlation.field
+ && f.getReferenceExpr() instanceof RexCorrelVariable) {
+ if (((RexCorrelVariable) f.getReferenceExpr()).id == correlation.corr) {
+ return true;
+ }
+ }
+ // fall through
+ default:
+ return false;
+ }
+ }
+
+ /** Returns whether one type is just a widening of another.
+ *
+ * <p>For example:<ul>
+ * <li>{@code VARCHAR(10)} is a widening of {@code VARCHAR(5)}.
+ * <li>{@code VARCHAR(10)} is a widening of {@code VARCHAR(10) NOT NULL}.
+ * </ul>
+ */
+ private boolean isWidening(RelDataType type, RelDataType type1) {
+ return type.getSqlTypeName() == type1.getSqlTypeName()
+ && type.getPrecision() >= type1.getPrecision();
}
public Frame decorrelateRel(HiveFilter rel) throws SemanticException {
@@ -1166,25 +1220,20 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// If this LogicalFilter has correlated reference, create value generator
// and produce the correlated variables in the new output.
- if (cm.mapRefRelToCorVar.containsKey(rel)) {
- decorrelateInputWithValueGenerator(rel);
-
- // The old input should be mapped to the newly created LogicalJoin by
- // rewriteInputWithValueGenerator().
- frame = map.get(oldInput);
+ if (cm.mapRefRelToCorRef.containsKey(rel)) {
+ frame = decorrelateInputWithValueGenerator(rel);
}
// Replace the filter expression to reference output of the join
// Map filter to the new filter over join
- RelNode newFilter = new HiveFilter(rel.getCluster(), rel.getTraitSet(), frame.r,
- decorrelateExpr(rel.getCondition()));
+ relBuilder.push(frame.r).filter(decorrelateExpr(rel.getCondition()));
// Filter does not change the input ordering.
// Filter rel does not permute the input.
// All corvars produced by filter will have the same output positions in the
// input rel.
- return register(rel, newFilter, frame.oldToNewOutputPos,
- frame.corVarOutputPos);
+ return register(rel, relBuilder.build(), frame.oldToNewOutputs,
+ frame.corDefOutputs);
}
}
@@ -1219,26 +1268,22 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// If this LogicalFilter has correlated reference, create value generator
// and produce the correlated variables in the new output.
- if (cm.mapRefRelToCorVar.containsKey(rel)) {
- decorrelateInputWithValueGenerator(rel);
+ if (cm.mapRefRelToCorRef.containsKey(rel)) {
+ frame = decorrelateInputWithValueGenerator(rel);
- // The old input should be mapped to the newly created LogicalJoin by
- // rewriteInputWithValueGenerator().
- frame = map.get(oldInput);
}
// Replace the filter expression to reference output of the join
// Map filter to the new filter over join
- RelNode newFilter = new HiveFilter(rel.getCluster(), rel.getTraitSet(), frame.r,
- decorrelateExpr(rel.getCondition()));
+ relBuilder.push(frame.r).filter(decorrelateExpr(rel.getCondition()));
// Filter does not change the input ordering.
// Filter rel does not permute the input.
// All corvars produced by filter will have the same output positions in the
// input rel.
- return register(rel, newFilter, frame.oldToNewOutputPos,
- frame.corVarOutputPos);
+ return register(rel, relBuilder.build(), frame.oldToNewOutputs,
+ frame.corDefOutputs);
}
/**
@@ -1268,18 +1313,18 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
return null;
}
- if (rightFrame.corVarOutputPos.isEmpty()) {
+ if (rightFrame.corDefOutputs.isEmpty()) {
return null;
}
assert rel.getRequiredColumns().cardinality()
- <= rightFrame.corVarOutputPos.keySet().size();
+ <= rightFrame.corDefOutputs.keySet().size();
// Change correlator rel into a join.
// Join all the correlated variables produced by this correlator rel
// with the values generated and propagated from the right input
- final SortedMap<Correlation, Integer> corVarOutputPos =
- new TreeMap<>(rightFrame.corVarOutputPos);
+ final SortedMap<CorDef, Integer> corDefOutputs =
+ new TreeMap<>(rightFrame.corDefOutputs);
final List<RexNode> conditions = new ArrayList<>();
final List<RelDataTypeField> newLeftOutput =
leftFrame.r.getRowType().getFieldList();
@@ -1288,14 +1333,14 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final List<RelDataTypeField> newRightOutput =
rightFrame.r.getRowType().getFieldList();
- for (Map.Entry<Correlation, Integer> rightOutputPos
- : Lists.newArrayList(corVarOutputPos.entrySet())) {
- final Correlation corVar = rightOutputPos.getKey();
- if (!corVar.corr.equals(rel.getCorrelationId())) {
+ for (Map.Entry<CorDef, Integer> rightOutput
+ : new ArrayList<>(corDefOutputs.entrySet())) {
+ final CorDef corDef = rightOutput.getKey();
+ if (!corDef.corr.equals(rel.getCorrelationId())) {
continue;
}
- final int newLeftPos = leftFrame.oldToNewOutputPos.get(corVar.field);
- final int newRightPos = rightOutputPos.getValue();
+ final int newLeftPos = leftFrame.oldToNewOutputs.get(corDef.field);
+ final int newRightPos = rightOutput.getValue();
conditions.add(
rexBuilder.makeCall(SqlStdOperatorTable.EQUALS,
RexInputRef.of(newLeftPos, newLeftOutput),
@@ -1303,23 +1348,23 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
newRightOutput.get(newRightPos).getType())));
// remove this cor var from output position mapping
- corVarOutputPos.remove(corVar);
+ corDefOutputs.remove(corDef);
}
// Update the output position for the cor vars: only pass on the cor
// vars that are not used in the join key.
- for (Correlation corVar : corVarOutputPos.keySet()) {
- int newPos = corVarOutputPos.get(corVar) + newLeftFieldCount;
- corVarOutputPos.put(corVar, newPos);
+ for (CorDef corDef : corDefOutputs.keySet()) {
+ int newPos = corDefOutputs.get(corDef) + newLeftFieldCount;
+ corDefOutputs.put(corDef, newPos);
}
// then add any cor var from the left input. Do not need to change
// output positions.
- corVarOutputPos.putAll(leftFrame.corVarOutputPos);
+ corDefOutputs.putAll(leftFrame.corDefOutputs);
// Create the mapping between the output of the old correlation rel
// and the new join rel
- final Map<Integer, Integer> mapOldToNewOutputPos = Maps.newHashMap();
+ final Map<Integer, Integer> mapOldToNewOutputs = new HashMap<>();
int oldLeftFieldCount = oldLeft.getRowType().getFieldCount();
@@ -1328,13 +1373,13 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
== oldLeftFieldCount + oldRightFieldCount;
// Left input positions are not changed.
- mapOldToNewOutputPos.putAll(leftFrame.oldToNewOutputPos);
+ mapOldToNewOutputs.putAll(leftFrame.oldToNewOutputs);
// Right input positions are shifted by newLeftFieldCount.
for (int i = 0; i < oldRightFieldCount; i++) {
- mapOldToNewOutputPos.put(
+ mapOldToNewOutputs.put(
i + oldLeftFieldCount,
- rightFrame.oldToNewOutputPos.get(i) + newLeftFieldCount);
+ rightFrame.oldToNewOutputs.get(i) + newLeftFieldCount);
}
final RexNode condition =
@@ -1343,7 +1388,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
LogicalJoin.create(leftFrame.r, rightFrame.r, condition,
ImmutableSet.<CorrelationId>of(), rel.getJoinType().toJoinType());
- return register(rel, newJoin, mapOldToNewOutputPos, corVarOutputPos);
+ return register(rel, newJoin, mapOldToNewOutputs, corDefOutputs);
}
public Frame decorrelateRel(HiveJoin rel) throws SemanticException{
@@ -1369,7 +1414,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// Create the mapping between the output of the old correlation rel
// and the new join rel
- Map<Integer, Integer> mapOldToNewOutputPos = Maps.newHashMap();
+ Map<Integer, Integer> mapOldToNewOutputs = Maps.newHashMap();
int oldLeftFieldCount = oldLeft.getRowType().getFieldCount();
int newLeftFieldCount = leftFrame.r.getRowType().getFieldCount();
@@ -1379,24 +1424,24 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
== oldLeftFieldCount + oldRightFieldCount;
// Left input positions are not changed.
- mapOldToNewOutputPos.putAll(leftFrame.oldToNewOutputPos);
+ mapOldToNewOutputs.putAll(leftFrame.oldToNewOutputs);
// Right input positions are shifted by newLeftFieldCount.
for (int i = 0; i < oldRightFieldCount; i++) {
- mapOldToNewOutputPos.put(i + oldLeftFieldCount,
- rightFrame.oldToNewOutputPos.get(i) + newLeftFieldCount);
+ mapOldToNewOutputs.put(i + oldLeftFieldCount,
+ rightFrame.oldToNewOutputs.get(i) + newLeftFieldCount);
}
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos =
- new TreeMap<>(leftFrame.corVarOutputPos);
+ final SortedMap<CorDef, Integer> corDefOutputs =
+ new TreeMap<>(leftFrame.corDefOutputs);
// Right input positions are shifted by newLeftFieldCount.
- for (Map.Entry<Correlation, Integer> entry
- : rightFrame.corVarOutputPos.entrySet()) {
- mapCorVarToOutputPos.put(entry.getKey(),
+ for (Map.Entry<CorDef, Integer> entry
+ : rightFrame.corDefOutputs.entrySet()) {
+ corDefOutputs.put(entry.getKey(),
entry.getValue() + newLeftFieldCount);
}
- return register(rel, newJoin, mapOldToNewOutputPos, mapCorVarToOutputPos);
+ return register(rel, newJoin, mapOldToNewOutputs, corDefOutputs);
}
/**
* Rewrite LogicalJoin.
@@ -1427,7 +1472,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// Create the mapping between the output of the old correlation rel
// and the new join rel
- Map<Integer, Integer> mapOldToNewOutputPos = Maps.newHashMap();
+ Map<Integer, Integer> mapOldToNewOutputs = Maps.newHashMap();
int oldLeftFieldCount = oldLeft.getRowType().getFieldCount();
int newLeftFieldCount = leftFrame.r.getRowType().getFieldCount();
@@ -1437,24 +1482,24 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
== oldLeftFieldCount + oldRightFieldCount;
// Left input positions are not changed.
- mapOldToNewOutputPos.putAll(leftFrame.oldToNewOutputPos);
+ mapOldToNewOutputs.putAll(leftFrame.oldToNewOutputs);
// Right input positions are shifted by newLeftFieldCount.
for (int i = 0; i < oldRightFieldCount; i++) {
- mapOldToNewOutputPos.put(i + oldLeftFieldCount,
- rightFrame.oldToNewOutputPos.get(i) + newLeftFieldCount);
+ mapOldToNewOutputs.put(i + oldLeftFieldCount,
+ rightFrame.oldToNewOutputs.get(i) + newLeftFieldCount);
}
- final SortedMap<Correlation, Integer> mapCorVarToOutputPos =
- new TreeMap<>(leftFrame.corVarOutputPos);
+ final SortedMap<CorDef, Integer> corDefOutputs =
+ new TreeMap<>(leftFrame.corDefOutputs);
// Right input positions are shifted by newLeftFieldCount.
- for (Map.Entry<Correlation, Integer> entry
- : rightFrame.corVarOutputPos.entrySet()) {
- mapCorVarToOutputPos.put(entry.getKey(),
+ for (Map.Entry<CorDef, Integer> entry
+ : rightFrame.corDefOutputs.entrySet()) {
+ corDefOutputs.put(entry.getKey(),
entry.getValue() + newLeftFieldCount);
}
- return register(rel, newJoin, mapOldToNewOutputPos, mapCorVarToOutputPos);
+ return register(rel, newJoin, mapOldToNewOutputs, corDefOutputs);
}
private RexInputRef getNewForOldInputRef(RexInputRef oldInputRef) {
@@ -1490,8 +1535,8 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// figure out the newLocalOrdinal, relative to the newInput.
int newLocalOrdinal = oldLocalOrdinal;
- if (!frame.oldToNewOutputPos.isEmpty()) {
- newLocalOrdinal = frame.oldToNewOutputPos.get(oldLocalOrdinal);
+ if (!frame.oldToNewOutputs.isEmpty()) {
+ newLocalOrdinal = frame.oldToNewOutputs.get(oldLocalOrdinal);
}
newOrdinal += newLocalOrdinal;
@@ -1623,11 +1668,11 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// check that all correlated refs in the filter condition are
// used in the join(as field access).
- Set<Correlation> corVarInFilter =
- Sets.newHashSet(cm.mapRefRelToCorVar.get(filter));
+ Set<CorRef> corVarInFilter =
+ Sets.newHashSet(cm.mapRefRelToCorRef.get(filter));
for (RexFieldAccess correlatedJoinKey : correlatedJoinKeys) {
- corVarInFilter.remove(cm.mapFieldAccessToCorVar.get(correlatedJoinKey));
+ corVarInFilter.remove(cm.mapFieldAccessToCorRef.get(correlatedJoinKey));
}
if (!corVarInFilter.isEmpty()) {
@@ -1636,10 +1681,10 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// Check that the correlated variables referenced in these
// comparisons do come from the correlatorRel.
- corVarInFilter.addAll(cm.mapRefRelToCorVar.get(filter));
+ corVarInFilter.addAll(cm.mapRefRelToCorRef.get(filter));
- for (Correlation corVar : corVarInFilter) {
- if (cm.mapCorVarToCorRel.get(corVar.corr) != correlate) {
+ for (CorRef corVar : corVarInFilter) {
+ if (cm.mapCorToCorRel.get(corVar.corr) != correlate) {
return false;
}
}
@@ -1648,9 +1693,9 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// if project has any correlated reference, make sure they are also
// provided by the current correlate. They will be projected out of the LHS
// of the correlate.
- if ((project != null) && cm.mapRefRelToCorVar.containsKey(project)) {
- for (Correlation corVar : cm.mapRefRelToCorVar.get(project)) {
- if (cm.mapCorVarToCorRel.get(corVar.corr) != correlate) {
+ if ((project != null) && cm.mapRefRelToCorRef.containsKey(project)) {
+ for (CorRef corVar : cm.mapRefRelToCorRef.get(project)) {
+ if (cm.mapCorToCorRel.get(corVar.corr) != correlate) {
return false;
}
}
@@ -1665,8 +1710,8 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
* @param correlate Correlator
*/
private void removeCorVarFromTree(LogicalCorrelate correlate) {
- if (cm.mapCorVarToCorRel.get(correlate.getCorrelationId()) == correlate) {
- cm.mapCorVarToCorRel.remove(correlate.getCorrelationId());
+ if (cm.mapCorToCorRel.get(correlate.getCorrelationId()) == correlate) {
+ cm.mapCorToCorRel.remove(correlate.getCorrelationId());
}
}
@@ -1706,11 +1751,9 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
/** Registers a relational expression and the relational expression it became
* after decorrelation. */
Frame register(RelNode rel, RelNode newRel,
- Map<Integer, Integer> oldToNewOutputPos,
- SortedMap<Correlation, Integer> corVarToOutputPos) {
- assert allLessThan(oldToNewOutputPos.keySet(),
- newRel.getRowType().getFieldCount(), Litmus.THROW);
- final Frame frame = new Frame(newRel, corVarToOutputPos, oldToNewOutputPos);
+ Map<Integer, Integer> oldToNewOutputs,
+ SortedMap<CorDef, Integer> corDefOutputs) {
+ final Frame frame = new Frame(rel, newRel, corDefOutputs, oldToNewOutputs);
map.put(rel, frame);
return frame;
}
@@ -1744,15 +1787,16 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
if (frame != null) {
// try to find in this input rel the position of cor var
- final Correlation corVar = cm.mapFieldAccessToCorVar.get(fieldAccess);
+ final CorRef corRef = cm.mapFieldAccessToCorRef.get(fieldAccess);
- if (corVar != null) {
- Integer newInputPos = frame.corVarOutputPos.get(corVar);
+ if (corRef != null) {
+ Integer newInputPos = frame.corDefOutputs.get(corRef.def());
if (newInputPos != null) {
// This input rel does produce the cor var referenced.
// Assume fieldAccess has the correct type info.
return new RexInputRef(newInputPos + newInputOutputOffset,
- fieldAccess.getType());
+ frame.r.getRowType().getFieldList().get(newInputPos)
+ .getType());
}
}
@@ -1767,7 +1811,12 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
@Override public RexNode visitInputRef(RexInputRef inputRef) {
- return getNewForOldInputRef(inputRef);
+ final RexInputRef ref = getNewForOldInputRef(inputRef);
+ if (ref.getIndex() == inputRef.getIndex()
+ && ref.getType() == inputRef.getType()) {
+ return inputRef; // re-use old object, to prevent needless expr cloning
+ }
+ return ref;
}
}
@@ -1839,9 +1888,9 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
@Override public RexNode visitFieldAccess(RexFieldAccess fieldAccess) {
- if (cm.mapFieldAccessToCorVar.containsKey(fieldAccess)) {
+ if (cm.mapFieldAccessToCorRef.containsKey(fieldAccess)) {
// if it is a corVar, change it to be input ref.
- Correlation corVar = cm.mapFieldAccessToCorVar.get(fieldAccess);
+ CorRef corVar = cm.mapFieldAccessToCorRef.get(fieldAccess);
// corVar offset should point to the leftInput of currentRel,
// which is the Correlator.
@@ -2083,7 +2132,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
int nullIndicatorPos;
if ((right instanceof LogicalFilter)
- && cm.mapRefRelToCorVar.containsKey(right)) {
+ && cm.mapRefRelToCorRef.containsKey(right)) {
// rightInputRel has this shape:
//
// LogicalFilter (references corvar)
@@ -2169,7 +2218,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
nullIndicatorPos =
left.getRowType().getFieldCount()
+ rightJoinKeys.get(0).getIndex();
- } else if (cm.mapRefRelToCorVar.containsKey(project)) {
+ } else if (cm.mapRefRelToCorRef.containsKey(project)) {
// check filter input contains no correlation
if (RelOptUtil.getVariablesUsed(right).size() > 0) {
return;
@@ -2295,7 +2344,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
if ((right instanceof LogicalFilter)
- && cm.mapRefRelToCorVar.containsKey(right)) {
+ && cm.mapRefRelToCorRef.containsKey(right)) {
// rightInputRel has this shape:
//
// LogicalFilter (references corvar)
@@ -2412,7 +2461,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// first change the filter condition into a join condition
joinCond =
removeCorrelationExpr(filter.getCondition(), false);
- } else if (cm.mapRefRelToCorVar.containsKey(aggInputProject)) {
+ } else if (cm.mapRefRelToCorRef.containsKey(aggInputProject)) {
// check rightInputRel contains no correlation
if (RelOptUtil.getVariablesUsed(right).size() > 0) {
return;
@@ -2727,8 +2776,8 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
// need to update the mapCorVarToCorRel Update the output position
// for the cor vars: only pass on the cor vars that are not used in
// the join key.
- if (cm.mapCorVarToCorRel.get(correlate.getCorrelationId()) == correlate) {
- cm.mapCorVarToCorRel.put(correlate.getCorrelationId(), newCorrelate);
+ if (cm.mapCorToCorRel.get(correlate.getCorrelationId()) == correlate) {
+ cm.mapCorToCorRel.put(correlate.getCorrelationId(), newCorrelate);
}
RelNode newOutput =
@@ -2739,25 +2788,40 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
/**
- * {@code Correlation} here represents a unique reference to a correlation
- * field.
- * For instance, if a RelNode references emp.name multiple times, it would
- * result in multiple {@code Correlation} objects that differ just in
- * {@link Correlation#uniqueKey}.
+ * A unique reference to a correlation field.
+ *
+ * <p>For instance, if a RelNode references emp.name multiple times, it would
+ * result in multiple {@code CorRef} objects that differ just in
+ * {@link CorRef#uniqueKey}.
*/
- static class Correlation
- implements Comparable<Correlation> {
+ static class CorRef implements Comparable<CorRef> {
public final int uniqueKey;
public final CorrelationId corr;
public final int field;
- Correlation(CorrelationId corr, int field, int uniqueKey) {
+ CorRef(CorrelationId corr, int field, int uniqueKey) {
this.corr = corr;
this.field = field;
this.uniqueKey = uniqueKey;
}
- public int compareTo(Correlation o) {
+ @Override public String toString() {
+ return corr.getName() + '.' + field;
+ }
+
+ @Override public int hashCode() {
+ return Objects.hash(uniqueKey, corr, field);
+ }
+
+ @Override public boolean equals(Object o) {
+ return this == o
+ || o instanceof CorRef
+ && uniqueKey == ((CorRef) o).uniqueKey
+ && corr == ((CorRef) o).corr
+ && field == ((CorRef) o).field;
+ }
+
+ public int compareTo(@Nonnull CorRef o) {
int c = corr.compareTo(o.corr);
if (c != 0) {
return c;
@@ -2768,6 +2832,44 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
return Integer.compare(uniqueKey, o.uniqueKey);
}
+
+ public CorDef def() {
+ return new CorDef(corr, field);
+ }
+ }
+
+ /** A correlation and a field. */
+ static class CorDef implements Comparable<CorDef> {
+ public final CorrelationId corr;
+ public final int field;
+
+ CorDef(CorrelationId corr, int field) {
+ this.corr = corr;
+ this.field = field;
+ }
+
+ @Override public String toString() {
+ return corr.getName() + '.' + field;
+ }
+
+ @Override public int hashCode() {
+ return Objects.hash(corr, field);
+ }
+
+ @Override public boolean equals(Object o) {
+ return this == o
+ || o instanceof CorDef
+ && corr == ((CorDef) o).corr
+ && field == ((CorDef) o).field;
+ }
+
+ public int compareTo(@Nonnull CorDef o) {
+ int c = corr.compareTo(o.corr);
+ if (c != 0) {
+ return c;
+ }
+ return Integer.compare(field, o.field);
+ }
}
/** A map of the locations of
@@ -2779,61 +2881,60 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
*
* <p>There are three maps:<ol>
*
- * <li>mapRefRelToCorVars map a rel node to the correlated variables it
- * references;
+ * <li>{@link #mapRefRelToCorRef} maps a {@link RelNode} to the correlated
+ * variables it references;
*
- * <li>mapCorVarToCorRel maps a correlated variable to the correlatorRel
- * providing it;
+ * <li>{@link #mapCorToCorRel} maps a correlated variable to the
+ * {@link Correlate} providing it;
*
- * <li>mapFieldAccessToCorVar maps a rex field access to
- * the cor var it represents. Because typeFlattener does not clone or
+ * <li>{@link #mapFieldAccessToCorRef} maps a rex field access to
+ * the corVar it represents. Because typeFlattener does not clone or
* modify a correlated field access this map does not need to be
* updated.
*
* </ol> */
private static class CorelMap {
- private final Multimap<RelNode, Correlation> mapRefRelToCorVar;
- private final SortedMap<CorrelationId, RelNode> mapCorVarToCorRel;
- private final Map<RexFieldAccess, Correlation> mapFieldAccessToCorVar;
+ private final Multimap<RelNode, CorRef> mapRefRelToCorRef;
+ private final SortedMap<CorrelationId, RelNode> mapCorToCorRel;
+ private final Map<RexFieldAccess, CorRef> mapFieldAccessToCorRef;
// TODO: create immutable copies of all maps
- private CorelMap(Multimap<RelNode, Correlation> mapRefRelToCorVar,
- SortedMap<CorrelationId, RelNode> mapCorVarToCorRel,
- Map<RexFieldAccess, Correlation> mapFieldAccessToCorVar) {
- this.mapRefRelToCorVar = mapRefRelToCorVar;
- this.mapCorVarToCorRel = mapCorVarToCorRel;
- this.mapFieldAccessToCorVar = ImmutableMap.copyOf(mapFieldAccessToCorVar);
+ private CorelMap(Multimap<RelNode, CorRef> mapRefRelToCorRef,
+ SortedMap<CorrelationId, RelNode> mapCorToCorRel,
+ Map<RexFieldAccess, CorRef> mapFieldAccessToCorRef) {
+ this.mapRefRelToCorRef = mapRefRelToCorRef;
+ this.mapCorToCorRel = mapCorToCorRel;
+ this.mapFieldAccessToCorRef = ImmutableMap.copyOf(mapFieldAccessToCorRef);
}
@Override public String toString() {
- return "mapRefRelToCorVar=" + mapRefRelToCorVar
- + "\nmapCorVarToCorRel=" + mapCorVarToCorRel
- + "\nmapFieldAccessToCorVar=" + mapFieldAccessToCorVar
- + "\n";
+ return "mapRefRelToCorRef=" + mapRefRelToCorRef
+ + "\nmapCorToCorRel=" + mapCorToCorRel
+ + "\nmapFieldAccessToCorRef=" + mapFieldAccessToCorRef
+ + "\n";
}
@Override public boolean equals(Object obj) {
return obj == this
- || obj instanceof CorelMap
- && mapRefRelToCorVar.equals(((CorelMap) obj).mapRefRelToCorVar)
- && mapCorVarToCorRel.equals(((CorelMap) obj).mapCorVarToCorRel)
- && mapFieldAccessToCorVar.equals(
- ((CorelMap) obj).mapFieldAccessToCorVar);
+ || obj instanceof CorelMap
+ && mapRefRelToCorRef.equals(((CorelMap) obj).mapRefRelToCorRef)
+ && mapCorToCorRel.equals(((CorelMap) obj).mapCorToCorRel)
+ && mapFieldAccessToCorRef.equals(
+ ((CorelMap) obj).mapFieldAccessToCorRef);
}
@Override public int hashCode() {
- return com.google.common.base.Objects.hashCode(mapRefRelToCorVar,
- mapCorVarToCorRel,
- mapFieldAccessToCorVar);
+ return Objects.hash(mapRefRelToCorRef, mapCorToCorRel,
+ mapFieldAccessToCorRef);
}
/** Creates a CorelMap with given contents. */
public static CorelMap of(
- SortedSetMultimap<RelNode, Correlation> mapRefRelToCorVar,
- SortedMap<CorrelationId, RelNode> mapCorVarToCorRel,
- Map<RexFieldAccess, Correlation> mapFieldAccessToCorVar) {
- return new CorelMap(mapRefRelToCorVar, mapCorVarToCorRel,
- mapFieldAccessToCorVar);
+ SortedSetMultimap<RelNode, CorRef> mapRefRelToCorVar,
+ SortedMap<CorrelationId, RelNode> mapCorToCorRel,
+ Map<RexFieldAccess, CorRef> mapFieldAccessToCorVar) {
+ return new CorelMap(mapRefRelToCorVar, mapCorToCorRel,
+ mapFieldAccessToCorVar);
}
/**
@@ -2842,27 +2943,26 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
* @return whether there are any correlating variables
*/
public boolean hasCorrelation() {
- return !mapCorVarToCorRel.isEmpty();
+ return !mapCorToCorRel.isEmpty();
}
}
/** Builds a {@link org.apache.calcite.sql2rel.RelDecorrelator.CorelMap}. */
private static class CorelMapBuilder extends HiveRelShuttleImpl {
- final SortedMap<CorrelationId, RelNode> mapCorVarToCorRel =
- new TreeMap<>();
-
- final SortedSetMultimap<RelNode, Correlation> mapRefRelToCorVar =
- Multimaps.newSortedSetMultimap(
- Maps.<RelNode, Collection<Correlation>>newHashMap(),
- new Supplier<TreeSet<Correlation>>() {
- public TreeSet<Correlation> get() {
- Bug.upgrade("use MultimapBuilder when we're on Guava-16");
- return Sets.newTreeSet();
- }
- });
-
- final Map<RexFieldAccess, Correlation> mapFieldAccessToCorVar =
- new HashMap<>();
+ final SortedMap<CorrelationId, RelNode> mapCorToCorRel =
+ new TreeMap<>();
+
+ final SortedSetMultimap<RelNode, CorRef> mapRefRelToCorRef =
+ Multimaps.newSortedSetMultimap(
+ new HashMap<RelNode, Collection<CorRef>>(),
+ new Supplier<TreeSet<CorRef>>() {
+ public TreeSet<CorRef> get() {
+ Bug.upgrade("use MultimapBuilder when we're on Guava-16");
+ return Sets.newTreeSet();
+ }
+ });
+
+ final Map<RexFieldAccess, CorRef> mapFieldAccessToCorVar = new HashMap<>();
final Holder<Integer> offset = Holder.of(0);
int corrIdGenerator = 0;
@@ -2872,7 +2972,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
/** Creates a CorelMap by iterating over a {@link RelNode} tree. */
CorelMap build(RelNode rel) {
stripHep(rel).accept(this);
- return new CorelMap(mapRefRelToCorVar, mapCorVarToCorRel,
+ return new CorelMap(mapRefRelToCorRef, mapCorToCorRel,
mapFieldAccessToCorVar);
}
@@ -2902,7 +3002,7 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
}
@Override public RelNode visit(LogicalCorrelate correlate) {
- mapCorVarToCorRel.put(correlate.getCorrelationId(), correlate);
+ mapCorToCorRel.put(correlate.getCorrelationId(), correlate);
return visitJoin(correlate);
}
@@ -2963,21 +3063,19 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
final RexNode ref = fieldAccess.getReferenceExpr();
if (ref instanceof RexCorrelVariable) {
final RexCorrelVariable var = (RexCorrelVariable) ref;
- if(mapFieldAccessToCorVar.containsKey(fieldAccess))
- {
+ if (mapFieldAccessToCorVar.containsKey(fieldAccess)) {
//for cases where different Rel nodes are referring to
// same correlation var (e.g. in case of NOT IN)
// avoid generating another correlation var
// and record the 'rel' is using the same correlation
- mapRefRelToCorVar.put(rel, mapFieldAccessToCorVar.get(fieldAccess));
- }
- else {
- final Correlation correlation =
- new Correlation(var.id,
- fieldAccess.getField().getIndex(),
- corrIdGenerator++);
+ mapRefRelToCorRef.put(rel,
+ mapFieldAccessToCorVar.get(fieldAccess));
+ } else {
+ final CorRef correlation =
+ new CorRef(var.id, fieldAccess.getField().getIndex(),
+ corrIdGenerator++);
mapFieldAccessToCorVar.put(fieldAccess, correlation);
- mapRefRelToCorVar.put(rel, correlation);
+ mapRefRelToCorRef.put(rel, correlation);
}
}
return super.visitFieldAccess(fieldAccess);
@@ -2996,14 +3094,20 @@ public class HiveRelDecorrelator implements ReflectiveVisitor {
* among its output fields. */
static class Frame {
final RelNode r;
- final ImmutableSortedMap<Correlation, Integer> corVarOutputPos;
- final ImmutableMap<Integer, Integer> oldToNewOutputPos;
+ final ImmutableSortedMap<CorDef, Integer> corDefOutputs;
+ final ImmutableSortedMap<Integer, Integer> oldToNewOutputs;
- Frame(RelNode r, SortedMap<Correlation, Integer> corVarOutputPos,
- Map<Integer, Integer> oldToNewOutputPos) {
+ Frame(RelNode oldRel, RelNode r, SortedMap<CorDef, Integer> corDefOutputs,
+ Map<Integer, Integer> oldToNewOutputs) {
this.r = Preconditions.checkNotNull(r);
- this.corVarOutputPos = ImmutableSortedMap.copyOf(corVarOutputPos);
- this.oldToNewOutputPos = ImmutableSortedMap.copyOf(oldToNewOutputPos);
+ this.corDefOutputs = ImmutableSortedMap.copyOf(corDefOutputs);
+ this.oldToNewOutputs = ImmutableSortedMap.copyOf(oldToNewOutputs);
+ assert allLessThan(corDefOutputs.values(),
+ r.getRowType().getFieldCount(), Litmus.THROW);
+ assert allLessThan(oldToNewOutputs.keySet(),
+ oldRel.getRowType().getFieldCount(), Litmus.THROW);
+ assert allLessThan(oldToNewOutputs.values(),
+ r.getRowType().getFieldCount(), Litmus.THROW);
}
}
}
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/queries/clientpositive/subquery_multi.q
----------------------------------------------------------------------
diff --git a/ql/src/test/queries/clientpositive/subquery_multi.q b/ql/src/test/queries/clientpositive/subquery_multi.q
index aff7f20..0362cc3 100644
--- a/ql/src/test/queries/clientpositive/subquery_multi.q
+++ b/ql/src/test/queries/clientpositive/subquery_multi.q
@@ -58,8 +58,12 @@ explain select * from part_null where p_name IN (select p_name from part_null) A
select * from part_null where p_name IN (select p_name from part_null) AND NOT EXISTS (select c from tempty);
-- corr, mix of IN/NOT IN
-explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type));
-select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type) AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type));
+explain select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type)
+ AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type
+ AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type));
+select * from part_null where p_name IN ( select p_name from part where part.p_type = part_null.p_type)
+ AND p_brand NOT IN (select p_container from part where part.p_type = part_null.p_type
+ AND p_brand IN (select p_brand from part pp where part.p_type = pp.p_type));
-- mix of corr and uncorr
explain select * from part_null where p_name IN ( select p_name from part) AND p_brand IN (select p_brand from part where part.p_type = part_null.p_type);
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/constprog_partitioner.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/constprog_partitioner.q.out b/ql/src/test/results/clientpositive/constprog_partitioner.q.out
index 2a44269..8c7f9d3 100644
--- a/ql/src/test/results/clientpositive/constprog_partitioner.q.out
+++ b/ql/src/test/results/clientpositive/constprog_partitioner.q.out
@@ -80,95 +80,10 @@ WHERE li.l_linenumber = 1 AND
li.l_orderkey IN (SELECT l_orderkey FROM lineitem WHERE l_shipmode = 'AIR' AND l_linenumber = li.l_linenumber)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-3 is a root stage
- Stage-2 depends on stages: Stage-3
- Stage-1 depends on stages: Stage-2
+ Stage-1 is a root stage
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-3
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: li
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_linenumber (type: int)
- outputColumnNames: l_linenumber
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: l_linenumber (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-2
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (l_shipmode = 'AIR') (type: boolean)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_orderkey (type: int), l_linenumber (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: int)
- sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
- TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
Stage: Stage-1
Map Reduce
Map Operator Tree:
@@ -189,11 +104,25 @@ STAGE PLANS:
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: int), _col2 (type: int)
TableScan
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
+ alias: lineitem
+ Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((l_shipmode = 'AIR') and (l_linenumber = l_linenumber)) (type: boolean)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: l_orderkey (type: int), l_linenumber (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
@@ -202,14 +131,14 @@ STAGE PLANS:
0 _col0 (type: int), 1 (type: int)
1 _col0 (type: int), _col1 (type: int)
outputColumnNames: _col1, _col2
- Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), _col2 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 60 Data size: 7257 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 55 Data size: 6598 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
[08/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/perf/query16.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/query16.q.out b/ql/src/test/results/clientpositive/perf/query16.q.out
index 1be5fba..dceb3a5 100644
--- a/ql/src/test/results/clientpositive/perf/query16.q.out
+++ b/ql/src/test/results/clientpositive/perf/query16.q.out
@@ -1,5 +1,4 @@
-Warning: Shuffle Join MERGEJOIN[141][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_4]] in Stage 'Reducer 13' is a cross product
-Warning: Shuffle Join MERGEJOIN[142][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_4]] in Stage 'Reducer 25' is a cross product
+Warning: Shuffle Join MERGEJOIN[106][tables = [$hdt$_2, $hdt$_3, $hdt$_1, $hdt$_4]] in Stage 'Reducer 18' is a cross product
PREHOOK: query: explain select
count(distinct cs_order_number) as `order count`
,sum(cs_ext_ship_cost) as `total shipping cost`
@@ -61,19 +60,16 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 13 <- Map 12 (CUSTOM_SIMPLE_EDGE), Map 17 (CUSTOM_SIMPLE_EDGE), Map 18 (CUSTOM_SIMPLE_EDGE), Map 19 (CUSTOM_SIMPLE_EDGE)
-Reducer 14 <- Reducer 13 (SIMPLE_EDGE)
-Reducer 15 <- Map 20 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
+Reducer 13 <- Map 12 (SIMPLE_EDGE)
+Reducer 15 <- Map 14 (SIMPLE_EDGE), Reducer 19 (SIMPLE_EDGE)
Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
+Reducer 18 <- Map 17 (CUSTOM_SIMPLE_EDGE), Map 20 (CUSTOM_SIMPLE_EDGE), Map 21 (CUSTOM_SIMPLE_EDGE), Map 22 (CUSTOM_SIMPLE_EDGE)
+Reducer 19 <- Reducer 18 (SIMPLE_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 9 (SIMPLE_EDGE)
-Reducer 22 <- Map 21 (SIMPLE_EDGE), Reducer 26 (SIMPLE_EDGE)
-Reducer 23 <- Reducer 22 (SIMPLE_EDGE)
-Reducer 25 <- Map 24 (CUSTOM_SIMPLE_EDGE), Map 27 (CUSTOM_SIMPLE_EDGE), Map 28 (CUSTOM_SIMPLE_EDGE), Map 29 (CUSTOM_SIMPLE_EDGE)
-Reducer 26 <- Reducer 25 (SIMPLE_EDGE)
Reducer 3 <- Map 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
Reducer 4 <- Map 11 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 16 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
-Reducer 6 <- Reducer 23 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
+Reducer 5 <- Reducer 13 (SIMPLE_EDGE), Reducer 4 (SIMPLE_EDGE)
+Reducer 6 <- Reducer 16 (SIMPLE_EDGE), Reducer 5 (SIMPLE_EDGE)
Reducer 7 <- Reducer 6 (SIMPLE_EDGE)
Reducer 8 <- Reducer 7 (SIMPLE_EDGE)
@@ -82,194 +78,156 @@ Stage-0
limit:100
Stage-1
Reducer 8
- File Output Operator [FS_94]
- Limit [LIM_93] (rows=1 width=344)
+ File Output Operator [FS_73]
+ Limit [LIM_72] (rows=1 width=344)
Number of rows:100
- Select Operator [SEL_92] (rows=1 width=344)
+ Select Operator [SEL_71] (rows=1 width=344)
Output:["_col0","_col1","_col2"]
<-Reducer 7 [SIMPLE_EDGE]
- SHUFFLE [RS_91]
- Group By Operator [GBY_89] (rows=1 width=344)
+ SHUFFLE [RS_70]
+ Group By Operator [GBY_68] (rows=1 width=344)
Output:["_col0","_col1","_col2"],aggregations:["count(DISTINCT KEY._col0:0._col0)","sum(VALUE._col1)","sum(VALUE._col2)"]
<-Reducer 6 [SIMPLE_EDGE]
- SHUFFLE [RS_88]
- Group By Operator [GBY_87] (rows=1534538622412395776 width=1)
+ SHUFFLE [RS_67]
+ Group By Operator [GBY_66] (rows=1395035081047425024 width=1)
Output:["_col0","_col1","_col2","_col3"],aggregations:["count(DISTINCT _col4)","sum(_col5)","sum(_col6)"],keys:_col4
- Select Operator [SEL_86] (rows=1534538622412395776 width=1)
+ Select Operator [SEL_65] (rows=1395035081047425024 width=1)
Output:["_col4","_col5","_col6"]
- Filter Operator [FIL_85] (rows=1534538622412395776 width=1)
+ Filter Operator [FIL_64] (rows=1395035081047425024 width=1)
predicate:_col16 is null
- Select Operator [SEL_84] (rows=3069077244824791552 width=1)
+ Select Operator [SEL_63] (rows=2790070162094850048 width=1)
Output:["_col4","_col5","_col6","_col16"]
- Merge Join Operator [MERGEJOIN_149] (rows=3069077244824791552 width=1)
- Conds:RS_81._col3, _col4=RS_82._col0, _col1(Inner),Output:["_col4","_col5","_col6","_col14"]
- <-Reducer 23 [SIMPLE_EDGE]
- SHUFFLE [RS_82]
+ Merge Join Operator [MERGEJOIN_112] (rows=2790070162094850048 width=1)
+ Conds:RS_60._col3, _col4=RS_61._col0, _col1(Inner),Output:["_col4","_col5","_col6","_col14"]
+ <-Reducer 16 [SIMPLE_EDGE]
+ SHUFFLE [RS_61]
PartitionCols:_col0, _col1
- Group By Operator [GBY_67] (rows=2536427365110644736 width=1)
+ Group By Operator [GBY_46] (rows=2536427365110644736 width=1)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 22 [SIMPLE_EDGE]
- SHUFFLE [RS_66]
+ <-Reducer 15 [SIMPLE_EDGE]
+ SHUFFLE [RS_45]
PartitionCols:_col0, _col1
- Group By Operator [GBY_65] (rows=5072854730221289472 width=1)
+ Group By Operator [GBY_44] (rows=5072854730221289472 width=1)
Output:["_col0","_col1"],keys:_col2, _col3
- Select Operator [SEL_64] (rows=5072854730221289472 width=1)
+ Select Operator [SEL_43] (rows=5072854730221289472 width=1)
Output:["_col2","_col3"]
- Filter Operator [FIL_63] (rows=5072854730221289472 width=1)
+ Filter Operator [FIL_42] (rows=5072854730221289472 width=1)
predicate:(_col2 <> _col0)
- Merge Join Operator [MERGEJOIN_147] (rows=5072854730221289472 width=1)
- Conds:RS_60._col1=RS_61._col1(Inner),Output:["_col0","_col2","_col3"]
- <-Map 21 [SIMPLE_EDGE]
- SHUFFLE [RS_60]
+ Merge Join Operator [MERGEJOIN_110] (rows=5072854730221289472 width=1)
+ Conds:RS_39._col1=RS_40._col1(Inner),Output:["_col0","_col2","_col3"]
+ <-Map 14 [SIMPLE_EDGE]
+ SHUFFLE [RS_39]
PartitionCols:_col1
- Select Operator [SEL_41] (rows=287989836 width=135)
+ Select Operator [SEL_20] (rows=287989836 width=135)
Output:["_col0","_col1"]
- TableScan [TS_40] (rows=287989836 width=135)
+ TableScan [TS_19] (rows=287989836 width=135)
default@catalog_sales,cs2,Tbl:COMPLETE,Col:NONE,Output:["cs_warehouse_sk","cs_order_number"]
- <-Reducer 26 [SIMPLE_EDGE]
- SHUFFLE [RS_61]
+ <-Reducer 19 [SIMPLE_EDGE]
+ SHUFFLE [RS_40]
PartitionCols:_col1
- Select Operator [SEL_59] (rows=4611686018427387903 width=1)
+ Select Operator [SEL_38] (rows=4611686018427387903 width=1)
Output:["_col0","_col1"]
- Group By Operator [GBY_58] (rows=4611686018427387903 width=1)
+ Group By Operator [GBY_37] (rows=4611686018427387903 width=1)
Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 25 [SIMPLE_EDGE]
- SHUFFLE [RS_57]
+ <-Reducer 18 [SIMPLE_EDGE]
+ SHUFFLE [RS_36]
PartitionCols:_col0, _col1
- Group By Operator [GBY_56] (rows=9223372036854775807 width=1)
+ Group By Operator [GBY_35] (rows=9223372036854775807 width=1)
Output:["_col0","_col1"],keys:_col4, _col3
- Select Operator [SEL_55] (rows=9223372036854775807 width=1)
+ Select Operator [SEL_34] (rows=9223372036854775807 width=1)
Output:["_col4","_col3"]
- Merge Join Operator [MERGEJOIN_142] (rows=9223372036854775807 width=1)
+ Merge Join Operator [MERGEJOIN_106] (rows=9223372036854775807 width=1)
Conds:(Inner),(Inner),(Inner),Output:["_col3","_col4"]
- <-Map 24 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_50]
- Select Operator [SEL_43] (rows=73049 width=4)
- TableScan [TS_42] (rows=73049 width=1119)
+ <-Map 17 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_29]
+ Select Operator [SEL_22] (rows=73049 width=4)
+ TableScan [TS_21] (rows=73049 width=1119)
default@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE
- <-Map 27 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_51]
- Select Operator [SEL_45] (rows=60 width=4)
- TableScan [TS_44] (rows=60 width=2045)
+ <-Map 20 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_30]
+ Select Operator [SEL_24] (rows=60 width=4)
+ TableScan [TS_23] (rows=60 width=2045)
default@call_center,call_center,Tbl:COMPLETE,Col:COMPLETE
- <-Map 28 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_52]
- Select Operator [SEL_47] (rows=40000000 width=4)
- TableScan [TS_46] (rows=40000000 width=1014)
+ <-Map 21 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_31]
+ Select Operator [SEL_26] (rows=40000000 width=4)
+ TableScan [TS_25] (rows=40000000 width=1014)
default@customer_address,customer_address,Tbl:COMPLETE,Col:COMPLETE
- <-Map 29 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_53]
- Select Operator [SEL_49] (rows=287989836 width=135)
+ <-Map 22 [CUSTOM_SIMPLE_EDGE]
+ PARTITION_ONLY_SHUFFLE [RS_32]
+ Select Operator [SEL_28] (rows=287989836 width=135)
Output:["_col0","_col1"]
- TableScan [TS_48] (rows=287989836 width=135)
+ TableScan [TS_27] (rows=287989836 width=135)
default@catalog_sales,cs1,Tbl:COMPLETE,Col:NONE,Output:["cs_warehouse_sk","cs_order_number"]
<-Reducer 5 [SIMPLE_EDGE]
- SHUFFLE [RS_81]
+ SHUFFLE [RS_60]
PartitionCols:_col3, _col4
- Merge Join Operator [MERGEJOIN_148] (rows=2790070162094850048 width=1)
- Conds:RS_78._col4=RS_79._col0(Left Outer),Output:["_col3","_col4","_col5","_col6","_col14"]
- <-Reducer 16 [SIMPLE_EDGE]
- SHUFFLE [RS_79]
+ Merge Join Operator [MERGEJOIN_111] (rows=421645953 width=135)
+ Conds:RS_57._col4=RS_58._col0(Left Outer),Output:["_col3","_col4","_col5","_col6","_col14"]
+ <-Reducer 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_58]
PartitionCols:_col0
- Select Operator [SEL_39] (rows=2536427365110644736 width=1)
+ Select Operator [SEL_18] (rows=7199720 width=106)
Output:["_col0","_col1"]
- Group By Operator [GBY_38] (rows=2536427365110644736 width=1)
+ Group By Operator [GBY_17] (rows=7199720 width=106)
Output:["_col0"],keys:KEY._col0
- <-Reducer 15 [SIMPLE_EDGE]
- SHUFFLE [RS_37]
+ <-Map 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_16]
PartitionCols:_col0
- Group By Operator [GBY_36] (rows=5072854730221289472 width=1)
- Output:["_col0"],keys:_col0
- Merge Join Operator [MERGEJOIN_146] (rows=5072854730221289472 width=1)
- Conds:RS_32._col0=RS_33._col0(Inner),Output:["_col0"]
- <-Map 20 [SIMPLE_EDGE]
- SHUFFLE [RS_33]
- PartitionCols:_col0
- Select Operator [SEL_31] (rows=28798881 width=106)
- Output:["_col0"]
- TableScan [TS_30] (rows=28798881 width=106)
- default@catalog_returns,cr1,Tbl:COMPLETE,Col:NONE,Output:["cr_order_number"]
- <-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_32]
- PartitionCols:_col0
- Group By Operator [GBY_28] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 13 [SIMPLE_EDGE]
- SHUFFLE [RS_27]
- PartitionCols:_col0
- Group By Operator [GBY_26] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col3
- Merge Join Operator [MERGEJOIN_141] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),(Inner),Output:["_col3"]
- <-Map 12 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_20]
- Select Operator [SEL_13] (rows=73049 width=4)
- TableScan [TS_12] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:COMPLETE
- <-Map 17 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_21]
- Select Operator [SEL_15] (rows=60 width=4)
- TableScan [TS_14] (rows=60 width=2045)
- default@call_center,call_center,Tbl:COMPLETE,Col:COMPLETE
- <-Map 18 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_22]
- Select Operator [SEL_17] (rows=40000000 width=4)
- TableScan [TS_16] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:COMPLETE
- <-Map 19 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_23]
- Select Operator [SEL_19] (rows=287989836 width=135)
- Output:["_col0"]
- TableScan [TS_18] (rows=287989836 width=135)
- default@catalog_sales,cs1,Tbl:COMPLETE,Col:NONE,Output:["cs_order_number"]
+ Group By Operator [GBY_15] (rows=14399440 width=106)
+ Output:["_col0"],keys:cr_order_number
+ Filter Operator [FIL_103] (rows=14399440 width=106)
+ predicate:(cr_order_number = cr_order_number)
+ TableScan [TS_12] (rows=28798881 width=106)
+ default@catalog_returns,cr1,Tbl:COMPLETE,Col:NONE,Output:["cr_order_number"]
<-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_78]
+ SHUFFLE [RS_57]
PartitionCols:_col4
- Merge Join Operator [MERGEJOIN_145] (rows=383314495 width=135)
- Conds:RS_75._col2=RS_76._col0(Inner),Output:["_col3","_col4","_col5","_col6"]
+ Merge Join Operator [MERGEJOIN_109] (rows=383314495 width=135)
+ Conds:RS_54._col2=RS_55._col0(Inner),Output:["_col3","_col4","_col5","_col6"]
<-Map 11 [SIMPLE_EDGE]
- SHUFFLE [RS_76]
+ SHUFFLE [RS_55]
PartitionCols:_col0
Select Operator [SEL_11] (rows=30 width=2045)
Output:["_col0"]
- Filter Operator [FIL_136] (rows=30 width=2045)
+ Filter Operator [FIL_102] (rows=30 width=2045)
predicate:((cc_county) IN ('Ziebach County', 'Levy County', 'Huron County', 'Franklin Parish', 'Daviess County') and cc_call_center_sk is not null)
TableScan [TS_9] (rows=60 width=2045)
default@call_center,call_center,Tbl:COMPLETE,Col:NONE,Output:["cc_call_center_sk","cc_county"]
<-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_75]
+ SHUFFLE [RS_54]
PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_144] (rows=348467716 width=135)
- Conds:RS_72._col1=RS_73._col0(Inner),Output:["_col2","_col3","_col4","_col5","_col6"]
+ Merge Join Operator [MERGEJOIN_108] (rows=348467716 width=135)
+ Conds:RS_51._col1=RS_52._col0(Inner),Output:["_col2","_col3","_col4","_col5","_col6"]
<-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_73]
+ SHUFFLE [RS_52]
PartitionCols:_col0
Select Operator [SEL_8] (rows=20000000 width=1014)
Output:["_col0"]
- Filter Operator [FIL_135] (rows=20000000 width=1014)
+ Filter Operator [FIL_101] (rows=20000000 width=1014)
predicate:((ca_state = 'NY') and ca_address_sk is not null)
TableScan [TS_6] (rows=40000000 width=1014)
default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
<-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_72]
+ SHUFFLE [RS_51]
PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_143] (rows=316788826 width=135)
- Conds:RS_69._col0=RS_70._col0(Inner),Output:["_col1","_col2","_col3","_col4","_col5","_col6"]
+ Merge Join Operator [MERGEJOIN_107] (rows=316788826 width=135)
+ Conds:RS_48._col0=RS_49._col0(Inner),Output:["_col1","_col2","_col3","_col4","_col5","_col6"]
<-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_69]
+ SHUFFLE [RS_48]
PartitionCols:_col0
Select Operator [SEL_2] (rows=287989836 width=135)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
- Filter Operator [FIL_133] (rows=287989836 width=135)
+ Filter Operator [FIL_99] (rows=287989836 width=135)
predicate:(cs_ship_date_sk is not null and cs_ship_addr_sk is not null and cs_call_center_sk is not null)
TableScan [TS_0] (rows=287989836 width=135)
default@catalog_sales,cs1,Tbl:COMPLETE,Col:NONE,Output:["cs_ship_date_sk","cs_ship_addr_sk","cs_call_center_sk","cs_warehouse_sk","cs_order_number","cs_ext_ship_cost","cs_net_profit"]
<-Map 9 [SIMPLE_EDGE]
- SHUFFLE [RS_70]
+ SHUFFLE [RS_49]
PartitionCols:_col0
Select Operator [SEL_5] (rows=8116 width=1119)
Output:["_col0"]
- Filter Operator [FIL_134] (rows=8116 width=1119)
+ Filter Operator [FIL_100] (rows=8116 width=1119)
predicate:(d_date BETWEEN '2001-4-01' AND 2001-05-31 01:00:00.0 and d_date_sk is not null)
TableScan [TS_3] (rows=73049 width=1119)
default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_date"]
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/perf/query30.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/query30.q.out b/ql/src/test/results/clientpositive/perf/query30.q.out
index 8d38fe2..3bb4f81 100644
--- a/ql/src/test/results/clientpositive/perf/query30.q.out
+++ b/ql/src/test/results/clientpositive/perf/query30.q.out
@@ -1,4 +1,3 @@
-Warning: Shuffle Join MERGEJOIN[154][tables = [$hdt$_4, $hdt$_5, $hdt$_6]] in Stage 'Reducer 22' is a cross product
PREHOOK: query: explain with customer_total_return as
(select wr_returning_customer_sk as ctr_customer_sk
,ca_state as ctr_state,
@@ -60,249 +59,163 @@ POSTHOOK: type: QUERY
Plan optimized by CBO.
Vertex dependency in root stage
-Reducer 10 <- Reducer 9 (SIMPLE_EDGE)
-Reducer 14 <- Map 13 (SIMPLE_EDGE), Map 19 (SIMPLE_EDGE)
-Reducer 15 <- Map 20 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
+Reducer 10 <- Reducer 16 (SIMPLE_EDGE), Reducer 9 (SIMPLE_EDGE)
+Reducer 14 <- Map 13 (SIMPLE_EDGE), Map 17 (SIMPLE_EDGE)
+Reducer 15 <- Map 18 (SIMPLE_EDGE), Reducer 14 (SIMPLE_EDGE)
Reducer 16 <- Reducer 15 (SIMPLE_EDGE)
-Reducer 17 <- Reducer 16 (SIMPLE_EDGE), Reducer 23 (SIMPLE_EDGE)
-Reducer 18 <- Reducer 17 (SIMPLE_EDGE)
-Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
-Reducer 22 <- Map 21 (CUSTOM_SIMPLE_EDGE), Map 30 (CUSTOM_SIMPLE_EDGE), Reducer 27 (CUSTOM_SIMPLE_EDGE)
-Reducer 23 <- Reducer 22 (SIMPLE_EDGE)
-Reducer 25 <- Map 24 (SIMPLE_EDGE), Map 28 (SIMPLE_EDGE)
-Reducer 26 <- Map 29 (SIMPLE_EDGE), Reducer 25 (SIMPLE_EDGE)
-Reducer 27 <- Reducer 26 (SIMPLE_EDGE)
+Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 5 (SIMPLE_EDGE)
Reducer 3 <- Reducer 10 (SIMPLE_EDGE), Reducer 2 (SIMPLE_EDGE)
-Reducer 4 <- Reducer 18 (SIMPLE_EDGE), Reducer 3 (SIMPLE_EDGE)
-Reducer 5 <- Reducer 4 (SIMPLE_EDGE)
-Reducer 8 <- Map 11 (SIMPLE_EDGE), Map 7 (SIMPLE_EDGE)
-Reducer 9 <- Map 12 (SIMPLE_EDGE), Reducer 8 (SIMPLE_EDGE)
+Reducer 4 <- Reducer 3 (SIMPLE_EDGE)
+Reducer 7 <- Map 11 (SIMPLE_EDGE), Map 6 (SIMPLE_EDGE)
+Reducer 8 <- Map 12 (SIMPLE_EDGE), Reducer 7 (SIMPLE_EDGE)
+Reducer 9 <- Reducer 8 (SIMPLE_EDGE)
Stage-0
Fetch Operator
limit:100
Stage-1
- Reducer 5
- File Output Operator [FS_102]
- Limit [LIM_101] (rows=100 width=1)
+ Reducer 4
+ File Output Operator [FS_67]
+ Limit [LIM_66] (rows=100 width=860)
Number of rows:100
- Select Operator [SEL_100] (rows=930023387364950016 width=1)
+ Select Operator [SEL_65] (rows=32266667 width=860)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12"]
- <-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_99]
- Select Operator [SEL_98] (rows=930023387364950016 width=1)
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_64]
+ Select Operator [SEL_63] (rows=32266667 width=860)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12"]
- Filter Operator [FIL_97] (rows=930023387364950016 width=1)
+ Filter Operator [FIL_62] (rows=32266667 width=860)
predicate:(_col2 > CASE WHEN (_col20 is null) THEN (null) ELSE (_col19) END)
- Select Operator [SEL_96] (rows=2790070162094850048 width=1)
+ Select Operator [SEL_61] (rows=96800003 width=860)
Output:["_col2","_col6","_col8","_col9","_col10","_col11","_col12","_col13","_col14","_col15","_col16","_col17","_col18","_col19","_col20"]
- Merge Join Operator [MERGEJOIN_162] (rows=2790070162094850048 width=1)
- Conds:RS_93._col17=RS_94._col2(Left Outer),Output:["_col1","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col18","_col19","_col20"]
- <-Reducer 18 [SIMPLE_EDGE]
- SHUFFLE [RS_94]
- PartitionCols:_col2
- Select Operator [SEL_86] (rows=2536427365110644736 width=1)
- Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_85] (rows=2536427365110644736 width=1)
- Output:["_col0","_col1"],aggregations:["avg(VALUE._col0)"],keys:KEY._col0
- <-Reducer 17 [SIMPLE_EDGE]
- SHUFFLE [RS_84]
- PartitionCols:_col0
- Group By Operator [GBY_83] (rows=5072854730221289472 width=1)
- Output:["_col0","_col1"],aggregations:["avg(_col2)"],keys:_col3
- Select Operator [SEL_82] (rows=5072854730221289472 width=1)
- Output:["_col3","_col2"]
- Merge Join Operator [MERGEJOIN_161] (rows=5072854730221289472 width=1)
- Conds:RS_79._col1=RS_80._col0(Inner),Output:["_col2","_col3"]
- <-Reducer 16 [SIMPLE_EDGE]
- SHUFFLE [RS_79]
- PartitionCols:_col1
- Select Operator [SEL_45] (rows=22000000 width=1014)
- Output:["_col1","_col2"]
- Group By Operator [GBY_44] (rows=22000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
- <-Reducer 15 [SIMPLE_EDGE]
- SHUFFLE [RS_43]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_42] (rows=44000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col7, _col1
- Select Operator [SEL_41] (rows=44000000 width=1014)
- Output:["_col7","_col1","_col3"]
- Merge Join Operator [MERGEJOIN_159] (rows=44000000 width=1014)
- Conds:RS_38._col2=RS_39._col0(Inner),Output:["_col1","_col3","_col7"]
- <-Map 20 [SIMPLE_EDGE]
- SHUFFLE [RS_39]
- PartitionCols:_col0
- Select Operator [SEL_34] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_148] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_32] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_38]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_158] (rows=15838314 width=92)
- Conds:RS_35._col0=RS_36._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Map 13 [SIMPLE_EDGE]
- SHUFFLE [RS_35]
- PartitionCols:_col0
- Select Operator [SEL_28] (rows=14398467 width=92)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_146] (rows=14398467 width=92)
- predicate:(wr_returned_date_sk is not null and wr_returning_addr_sk is not null)
- TableScan [TS_26] (rows=14398467 width=92)
- default@web_returns,web_returns,Tbl:COMPLETE,Col:NONE,Output:["wr_returned_date_sk","wr_returning_customer_sk","wr_returning_addr_sk","wr_return_amt"]
- <-Map 19 [SIMPLE_EDGE]
- SHUFFLE [RS_36]
- PartitionCols:_col0
- Select Operator [SEL_31] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_147] (rows=36524 width=1119)
- predicate:((d_year = 2002) and d_date_sk is not null)
- TableScan [TS_29] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Reducer 23 [SIMPLE_EDGE]
- SHUFFLE [RS_80]
+ Merge Join Operator [MERGEJOIN_105] (rows=96800003 width=860)
+ Conds:RS_58._col0=RS_59._col0(Inner),Output:["_col1","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col18","_col19","_col20"]
+ <-Reducer 10 [SIMPLE_EDGE]
+ SHUFFLE [RS_59]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_104] (rows=24200000 width=1014)
+ Conds:RS_51._col1=RS_52._col2(Left Outer),Output:["_col0","_col2","_col3","_col4"]
+ <-Reducer 16 [SIMPLE_EDGE]
+ SHUFFLE [RS_52]
+ PartitionCols:_col2
+ Select Operator [SEL_50] (rows=5500000 width=1014)
+ Output:["_col0","_col1","_col2"]
+ Group By Operator [GBY_49] (rows=5500000 width=1014)
+ Output:["_col0","_col1"],aggregations:["avg(_col2)"],keys:_col0
+ Select Operator [SEL_45] (rows=11000000 width=1014)
+ Output:["_col0","_col2"]
+ Group By Operator [GBY_44] (rows=11000000 width=1014)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
+ <-Reducer 15 [SIMPLE_EDGE]
+ SHUFFLE [RS_43]
PartitionCols:_col0
- Group By Operator [GBY_77] (rows=4611686018427387903 width=1)
- Output:["_col0"],keys:KEY._col0
- <-Reducer 22 [SIMPLE_EDGE]
- SHUFFLE [RS_76]
+ Group By Operator [GBY_42] (rows=22000000 width=1014)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col7, _col1
+ Select Operator [SEL_41] (rows=22000000 width=1014)
+ Output:["_col7","_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_103] (rows=22000000 width=1014)
+ Conds:RS_38._col2=RS_39._col0(Inner),Output:["_col1","_col3","_col7"]
+ <-Map 18 [SIMPLE_EDGE]
+ SHUFFLE [RS_39]
+ PartitionCols:_col0
+ Select Operator [SEL_34] (rows=20000000 width=1014)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_98] (rows=20000000 width=1014)
+ predicate:((ca_state = ca_state) and ca_address_sk is not null)
+ TableScan [TS_32] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ <-Reducer 14 [SIMPLE_EDGE]
+ SHUFFLE [RS_38]
+ PartitionCols:_col2
+ Merge Join Operator [MERGEJOIN_102] (rows=15838314 width=92)
+ Conds:RS_35._col0=RS_36._col0(Inner),Output:["_col1","_col2","_col3"]
+ <-Map 13 [SIMPLE_EDGE]
+ SHUFFLE [RS_35]
+ PartitionCols:_col0
+ Select Operator [SEL_28] (rows=14398467 width=92)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_96] (rows=14398467 width=92)
+ predicate:(wr_returned_date_sk is not null and wr_returning_addr_sk is not null)
+ TableScan [TS_26] (rows=14398467 width=92)
+ default@web_returns,web_returns,Tbl:COMPLETE,Col:NONE,Output:["wr_returned_date_sk","wr_returning_customer_sk","wr_returning_addr_sk","wr_return_amt"]
+ <-Map 17 [SIMPLE_EDGE]
+ SHUFFLE [RS_36]
+ PartitionCols:_col0
+ Select Operator [SEL_31] (rows=36524 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_97] (rows=36524 width=1119)
+ predicate:((d_year = 2002) and d_date_sk is not null)
+ TableScan [TS_29] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
+ <-Reducer 9 [SIMPLE_EDGE]
+ SHUFFLE [RS_51]
+ PartitionCols:_col1
+ Select Operator [SEL_25] (rows=22000000 width=1014)
+ Output:["_col0","_col1","_col2"]
+ Group By Operator [GBY_24] (rows=22000000 width=1014)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
+ <-Reducer 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_23]
+ PartitionCols:_col0, _col1
+ Group By Operator [GBY_22] (rows=44000000 width=1014)
+ Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col7, _col1
+ Select Operator [SEL_21] (rows=44000000 width=1014)
+ Output:["_col7","_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_101] (rows=44000000 width=1014)
+ Conds:RS_18._col2=RS_19._col0(Inner),Output:["_col1","_col3","_col7"]
+ <-Map 12 [SIMPLE_EDGE]
+ SHUFFLE [RS_19]
PartitionCols:_col0
- Group By Operator [GBY_75] (rows=9223372036854775807 width=1)
- Output:["_col0"],keys:_col2
- Merge Join Operator [MERGEJOIN_154] (rows=9223372036854775807 width=1)
- Conds:(Inner),(Inner),Output:["_col2"]
- <-Map 21 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_70]
- Select Operator [SEL_47] (rows=80000000 width=4)
- TableScan [TS_46] (rows=80000000 width=860)
- default@customer,customer,Tbl:COMPLETE,Col:COMPLETE
- <-Map 30 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_72]
- Select Operator [SEL_69] (rows=40000000 width=4)
- TableScan [TS_68] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:COMPLETE
- <-Reducer 27 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_71]
- Select Operator [SEL_67] (rows=22000000 width=1014)
- Output:["_col1"]
- Group By Operator [GBY_66] (rows=22000000 width=1014)
- Output:["_col0","_col1"],keys:KEY._col0, KEY._col1
- <-Reducer 26 [SIMPLE_EDGE]
- SHUFFLE [RS_65]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_64] (rows=44000000 width=1014)
- Output:["_col0","_col1"],keys:_col7, _col1
- Select Operator [SEL_63] (rows=44000000 width=1014)
- Output:["_col7","_col1"]
- Merge Join Operator [MERGEJOIN_153] (rows=44000000 width=1014)
- Conds:RS_60._col2=RS_61._col0(Inner),Output:["_col1","_col7"]
- <-Map 29 [SIMPLE_EDGE]
- SHUFFLE [RS_61]
- PartitionCols:_col0
- Select Operator [SEL_56] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_151] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_54] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 25 [SIMPLE_EDGE]
- SHUFFLE [RS_60]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_152] (rows=15838314 width=92)
- Conds:RS_57._col0=RS_58._col0(Inner),Output:["_col1","_col2"]
- <-Map 24 [SIMPLE_EDGE]
- SHUFFLE [RS_57]
- PartitionCols:_col0
- Select Operator [SEL_50] (rows=14398467 width=92)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_149] (rows=14398467 width=92)
- predicate:(wr_returned_date_sk is not null and wr_returning_addr_sk is not null)
- TableScan [TS_48] (rows=14398467 width=92)
- default@web_returns,web_returns,Tbl:COMPLETE,Col:NONE,Output:["wr_returned_date_sk","wr_returning_customer_sk","wr_returning_addr_sk"]
- <-Map 28 [SIMPLE_EDGE]
- SHUFFLE [RS_58]
- PartitionCols:_col0
- Select Operator [SEL_53] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_150] (rows=36524 width=1119)
- predicate:((d_year = 2002) and d_date_sk is not null)
- TableScan [TS_51] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_93]
- PartitionCols:_col17
- Merge Join Operator [MERGEJOIN_160] (rows=96800003 width=860)
- Conds:RS_90._col0=RS_91._col0(Inner),Output:["_col1","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13","_col17","_col18"]
- <-Reducer 10 [SIMPLE_EDGE]
- SHUFFLE [RS_91]
+ Select Operator [SEL_14] (rows=40000000 width=1014)
+ Output:["_col0","_col1"]
+ Filter Operator [FIL_95] (rows=40000000 width=1014)
+ predicate:ca_address_sk is not null
+ TableScan [TS_12] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ <-Reducer 7 [SIMPLE_EDGE]
+ SHUFFLE [RS_18]
+ PartitionCols:_col2
+ Merge Join Operator [MERGEJOIN_100] (rows=15838314 width=92)
+ Conds:RS_15._col0=RS_16._col0(Inner),Output:["_col1","_col2","_col3"]
+ <-Map 11 [SIMPLE_EDGE]
+ SHUFFLE [RS_16]
+ PartitionCols:_col0
+ Select Operator [SEL_11] (rows=36524 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_94] (rows=36524 width=1119)
+ predicate:((d_year = 2002) and d_date_sk is not null)
+ TableScan [TS_9] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
+ <-Map 6 [SIMPLE_EDGE]
+ SHUFFLE [RS_15]
+ PartitionCols:_col0
+ Select Operator [SEL_8] (rows=14398467 width=92)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_93] (rows=14398467 width=92)
+ predicate:(wr_returned_date_sk is not null and wr_returning_addr_sk is not null and wr_returning_customer_sk is not null)
+ TableScan [TS_6] (rows=14398467 width=92)
+ default@web_returns,web_returns,Tbl:COMPLETE,Col:NONE,Output:["wr_returned_date_sk","wr_returning_customer_sk","wr_returning_addr_sk","wr_return_amt"]
+ <-Reducer 2 [SIMPLE_EDGE]
+ SHUFFLE [RS_58]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_99] (rows=88000001 width=860)
+ Conds:RS_55._col2=RS_56._col0(Inner),Output:["_col0","_col1","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_55]
+ PartitionCols:_col2
+ Select Operator [SEL_2] (rows=80000000 width=860)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13"]
+ Filter Operator [FIL_91] (rows=80000000 width=860)
+ predicate:(c_customer_sk is not null and c_current_addr_sk is not null)
+ TableScan [TS_0] (rows=80000000 width=860)
+ default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_customer_id","c_current_addr_sk","c_salutation","c_first_name","c_last_name","c_preferred_cust_flag","c_birth_day","c_birth_month","c_birth_year","c_birth_country","c_login","c_email_address","c_last_review_date"]
+ <-Map 5 [SIMPLE_EDGE]
+ SHUFFLE [RS_56]
PartitionCols:_col0
- Group By Operator [GBY_24] (rows=22000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1
- <-Reducer 9 [SIMPLE_EDGE]
- SHUFFLE [RS_23]
- PartitionCols:_col0, _col1
- Group By Operator [GBY_22] (rows=44000000 width=1014)
- Output:["_col0","_col1","_col2"],aggregations:["sum(_col3)"],keys:_col1, _col7
- Select Operator [SEL_21] (rows=44000000 width=1014)
- Output:["_col1","_col7","_col3"]
- Merge Join Operator [MERGEJOIN_157] (rows=44000000 width=1014)
- Conds:RS_18._col2=RS_19._col0(Inner),Output:["_col1","_col3","_col7"]
- <-Map 12 [SIMPLE_EDGE]
- SHUFFLE [RS_19]
- PartitionCols:_col0
- Select Operator [SEL_14] (rows=40000000 width=1014)
- Output:["_col0","_col1"]
- Filter Operator [FIL_145] (rows=40000000 width=1014)
- predicate:ca_address_sk is not null
- TableScan [TS_12] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
- <-Reducer 8 [SIMPLE_EDGE]
- SHUFFLE [RS_18]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_156] (rows=15838314 width=92)
- Conds:RS_15._col0=RS_16._col0(Inner),Output:["_col1","_col2","_col3"]
- <-Map 11 [SIMPLE_EDGE]
- SHUFFLE [RS_16]
- PartitionCols:_col0
- Select Operator [SEL_11] (rows=36524 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_144] (rows=36524 width=1119)
- predicate:((d_year = 2002) and d_date_sk is not null)
- TableScan [TS_9] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year"]
- <-Map 7 [SIMPLE_EDGE]
- SHUFFLE [RS_15]
- PartitionCols:_col0
- Select Operator [SEL_8] (rows=14398467 width=92)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_143] (rows=14398467 width=92)
- predicate:(wr_returned_date_sk is not null and wr_returning_addr_sk is not null and wr_returning_customer_sk is not null)
- TableScan [TS_6] (rows=14398467 width=92)
- default@web_returns,web_returns,Tbl:COMPLETE,Col:NONE,Output:["wr_returned_date_sk","wr_returning_customer_sk","wr_returning_addr_sk","wr_return_amt"]
- <-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_90]
- PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_155] (rows=88000001 width=860)
- Conds:RS_87._col2=RS_88._col0(Inner),Output:["_col0","_col1","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13"]
- <-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_87]
- PartitionCols:_col2
- Select Operator [SEL_2] (rows=80000000 width=860)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6","_col7","_col8","_col9","_col10","_col11","_col12","_col13"]
- Filter Operator [FIL_141] (rows=80000000 width=860)
- predicate:(c_customer_sk is not null and c_current_addr_sk is not null)
- TableScan [TS_0] (rows=80000000 width=860)
- default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_customer_id","c_current_addr_sk","c_salutation","c_first_name","c_last_name","c_preferred_cust_flag","c_birth_day","c_birth_month","c_birth_year","c_birth_country","c_login","c_email_address","c_last_review_date"]
- <-Map 6 [SIMPLE_EDGE]
- SHUFFLE [RS_88]
- PartitionCols:_col0
- Select Operator [SEL_5] (rows=20000000 width=1014)
- Output:["_col0"]
- Filter Operator [FIL_142] (rows=20000000 width=1014)
- predicate:((ca_state = 'IL') and ca_address_sk is not null)
- TableScan [TS_3] (rows=40000000 width=1014)
- default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
+ Select Operator [SEL_5] (rows=20000000 width=1014)
+ Output:["_col0"]
+ Filter Operator [FIL_92] (rows=20000000 width=1014)
+ predicate:((ca_state = 'IL') and ca_address_sk is not null)
+ TableScan [TS_3] (rows=40000000 width=1014)
+ default@customer_address,customer_address,Tbl:COMPLETE,Col:NONE,Output:["ca_address_sk","ca_state"]
[04/15] hive git commit: HIVE-15905 : Inefficient plan for correlated
subqueries (Vineet Garg via Ashutosh Chauhan)
Posted by ha...@apache.org.
http://git-wip-us.apache.org/repos/asf/hive/blob/bddf5a7a/ql/src/test/results/clientpositive/spark/subquery_in.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_in.q.out b/ql/src/test/results/clientpositive/spark/subquery_in.q.out
index 0fde046..1ef17d9 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_in.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_in.q.out
@@ -132,9 +132,7 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Map 3 (PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -157,37 +155,22 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- Map 5
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -205,42 +188,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -472,12 +419,10 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Reducer 4 (PARTITION-LEVEL SORT, 2), Reducer 9 (PARTITION-LEVEL SORT, 2)
+ Reducer 5 <- Reducer 4 (GROUP, 2)
Reducer 6 <- Reducer 5 (GROUP, 2)
- Reducer 7 <- Reducer 6 (GROUP, 2)
- Reducer 9 <- Map 8 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -500,31 +445,15 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: p_mfgr (type: string), p_size (type: int)
- sort order: ++
- Map-reduce partition columns: p_mfgr (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- TopN Hash Memory Usage: 0.1
- Map 8
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_mfgr (type: string)
- outputColumnNames: p_mfgr
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_mfgr = p_mfgr) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: p_mfgr (type: string), p_size (type: int)
+ sort order: ++
+ Map-reduce partition columns: p_mfgr (type: string)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ TopN Hash Memory Usage: 0.1
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -551,7 +480,7 @@ STAGE PLANS:
Select Operator
expressions: KEY.reducesinkkey0 (type: string), KEY.reducesinkkey1 (type: int)
outputColumnNames: _col2, _col5
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
PTF Operator
Function definitions:
Input definition
@@ -572,92 +501,60 @@ STAGE PLANS:
window function: GenericUDAFRankEvaluator
window frame: PRECEDING(MAX)~FOLLOWING(MAX)
isPivotResult: true
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: (rank_window_0 <= 2) (type: boolean)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col2 (type: string), _col5 (type: int)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: min(_col1)
+ keys: _col0 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: int)
Reducer 5
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: min(_col1)
- keys: _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Reducer 6
- Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: int)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: int)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 7 Data size: 865 Basic stats: COMPLETE Column stats: NONE
- Reducer 7
+ Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ Reducer 6
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 370 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 3 Data size: 370 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: int)
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
- Statistics: Num rows: 3 Data size: 370 Basic stats: COMPLETE Column stats: NONE
- Reducer 9
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -715,10 +612,8 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Reducer 4 (GROUP, 2)
- Reducer 7 <- Map 6 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
+ Reducer 4 <- Map 3 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -741,37 +636,18 @@ STAGE PLANS:
alias: a
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (key > '9') (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- Map 6
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: value
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((value = value) and (key > '9')) (type: boolean)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: value (type: string)
+ keys: key (type: string), value (type: string)
mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -791,53 +667,21 @@ STAGE PLANS:
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 4
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col2 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reducer 5
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
- Reducer 7
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -1075,7 +919,6 @@ POSTHOOK: Input: default@lineitem
#### A masked pattern was here ####
108570 8571
4297 1798
-Warning: Shuffle Join JOIN[14][tables = [$hdt$_3, $hdt$_4]] in Work 'Reducer 11' is a cross product
PREHOOK: query: explain select sum(l_extendedprice) from lineitem, part where p_partkey = l_partkey and l_quantity IN (select avg(l_quantity) from lineitem where l_partkey = p_partkey)
PREHOOK: type: QUERY
POSTHOOK: query: explain select sum(l_extendedprice) from lineitem, part where p_partkey = l_partkey and l_quantity IN (select avg(l_quantity) from lineitem where l_partkey = p_partkey)
@@ -1088,14 +931,11 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 11 <- Map 10 (PARTITION-LEVEL SORT, 1), Map 13 (PARTITION-LEVEL SORT, 1)
- Reducer 12 <- Reducer 11 (GROUP, 2)
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Map 5 (PARTITION-LEVEL SORT, 2)
- Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2), Reducer 9 (PARTITION-LEVEL SORT, 2)
+ Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2), Reducer 8 (PARTITION-LEVEL SORT, 2)
Reducer 4 <- Reducer 3 (GROUP, 1)
- Reducer 7 <- Map 6 (PARTITION-LEVEL SORT, 2), Reducer 12 (PARTITION-LEVEL SORT, 2)
+ Reducer 7 <- Map 6 (GROUP, 2)
Reducer 8 <- Reducer 7 (GROUP, 2)
- Reducer 9 <- Reducer 8 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1116,29 +956,6 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: double), _col2 (type: double)
- Map 10
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 100 Data size: 400 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 100 Data size: 400 Basic stats: COMPLETE Column stats: COMPLETE
- Map 13
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_partkey (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
Map 5
Map Operator Tree:
TableScan
@@ -1161,49 +978,22 @@ STAGE PLANS:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_partkey (type: int), l_quantity (type: double)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: double)
- Reducer 11
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0
- 1
- outputColumnNames: _col1
- Statistics: Num rows: 2600 Data size: 327700 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: int)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 2600 Data size: 327700 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 2600 Data size: 327700 Basic stats: COMPLETE Column stats: NONE
- Reducer 12
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: int)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1300 Data size: 163850 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1300 Data size: 163850 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
+ Filter Operator
+ predicate: (l_partkey = l_partkey) (type: boolean)
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: avg(l_quantity)
+ keys: l_partkey (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: struct<count:bigint,sum:double,input:double>)
+ Reducer 2
Reduce Operator Tree:
Join Operator
condition map:
@@ -1228,7 +1018,7 @@ STAGE PLANS:
0 _col1 (type: double), _col3 (type: int)
1 _col0 (type: double), _col1 (type: int)
outputColumnNames: _col2
- Statistics: Num rows: 392 Data size: 49494 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 121 Data size: 14517 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: sum(_col2)
mode: hash
@@ -1254,60 +1044,34 @@ STAGE PLANS:
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 7
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 1430 Data size: 180235 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: int), _col1 (type: double)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 1430 Data size: 180235 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: avg(_col1)
- keys: _col2 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1430 Data size: 180235 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1430 Data size: 180235 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: struct<count:bigint,sum:double,input:double>)
- Reducer 8
- Reduce Operator Tree:
Group By Operator
aggregations: avg(VALUE._col0)
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 715 Data size: 90117 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col1 (type: double), _col0 (type: int)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 715 Data size: 90117 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: double), _col1 (type: int)
sort order: ++
Map-reduce partition columns: _col0 (type: double), _col1 (type: int)
- Statistics: Num rows: 715 Data size: 90117 Basic stats: COMPLETE Column stats: NONE
- Reducer 9
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Reducer 8
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: double), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 357 Data size: 44995 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 12 Data size: 1439 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: double), _col1 (type: int)
sort order: ++
Map-reduce partition columns: _col0 (type: double), _col1 (type: int)
- Statistics: Num rows: 357 Data size: 44995 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 12 Data size: 1439 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -1315,7 +1079,6 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[14][tables = [$hdt$_3, $hdt$_4]] in Work 'Reducer 11' is a cross product
PREHOOK: query: select sum(l_extendedprice) from lineitem, part where p_partkey = l_partkey and l_quantity IN (select avg(l_quantity) from lineitem where l_partkey = p_partkey)
PREHOOK: type: QUERY
PREHOOK: Input: default@lineitem
@@ -1339,10 +1102,8 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Reducer 4 (GROUP, 2)
- Reducer 7 <- Map 6 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
+ Reducer 4 <- Map 3 (GROUP PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1368,43 +1129,27 @@ STAGE PLANS:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int)
- Map 6
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_type (type: string)
- outputColumnNames: p_type
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_type = p_type) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: p_type (type: string)
+ keys: p_type (type: string), p_size (type: int)
mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col4 (type: string), _col5 (type: int)
- 1 _col0 (type: string), _col1 (type: int)
+ 1 _col1 (type: string), _col0 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
File Output Operator
@@ -1416,57 +1161,25 @@ STAGE PLANS:
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 4
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: int)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reducer 5
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col0 (type: string), _col1 (type: int)
- mode: hash
+ mode: complete
outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reducer 7
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 3 Data size: 363 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: int), _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 3 Data size: 363 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col0 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
+ Statistics: Num rows: 3 Data size: 363 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -1779,11 +1492,9 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
Reducer 3 <- Reducer 2 (GROUP, 1)
- Reducer 5 <- Map 4 (PARTITION-LEVEL SORT, 2), Reducer 8 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Reducer 5 (GROUP, 2)
- Reducer 8 <- Map 7 (GROUP, 2)
+ Reducer 5 <- Map 4 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1805,35 +1516,23 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
- Map 7
- Map Operator Tree:
- TableScan
- alias: e
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_name (type: string)
- outputColumnNames: p_name
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_name (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_name = p_name) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: p_name (type: string), p_partkey (type: int)
+ outputColumnNames: p_name, p_partkey
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: p_name (type: string), p_partkey (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -1868,56 +1567,20 @@ STAGE PLANS:
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 5
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col0 (type: int)
- outputColumnNames: _col2, _col0
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col0 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: int)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: int), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: int)
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reducer 8
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -2100,10 +1763,9 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Reducer 4 (GROUP, 2)
- Reducer 7 <- Map 6 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
+ Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
+ Reducer 6 <- Map 5 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2116,9 +1778,9 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col1 (type: string), _col5 (type: int), _col5 (type: int)
- sort order: +++
- Map-reduce partition columns: _col1 (type: string), _col5 (type: int), _col5 (type: int)
+ key expressions: _col1 (type: string), _col5 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col5 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col6 (type: string), _col7 (type: double), _col8 (type: string)
Map 3
@@ -2136,7 +1798,7 @@ STAGE PLANS:
Map-reduce partition columns: _col2 (type: int), _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
- Map 6
+ Map 5
Map Operator Tree:
TableScan
alias: part
@@ -2159,15 +1821,15 @@ STAGE PLANS:
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Semi Join 0 to 1
keys:
- 0 _col1 (type: string), _col5 (type: int), _col5 (type: int)
- 1 _col0 (type: string), _col2 (type: int), _col1 (type: int)
+ 0 _col1 (type: string), _col5 (type: int)
+ 1 _col0 (type: string), _col1 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -2182,33 +1844,21 @@ STAGE PLANS:
1 _col0 (type: int), (_col0 + 121150) (type: int)
outputColumnNames: _col1, _col3
Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col1 (type: string), _col3 (type: int)
- mode: hash
+ Select Operator
+ expressions: _col1 (type: string), _col3 (type: int)
outputColumnNames: _col0, _col1
Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reducer 5
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: int)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: int), _col1 (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col2 (type: int), _col1 (type: int)
- sort order: +++
- Map-reduce partition columns: _col0 (type: string), _col2 (type: int), _col1 (type: int)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reducer 7
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ Reducer 6
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -2250,8 +1900,7 @@ STAGE PLANS:
Spark
Edges:
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 4 <- Map 3 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2274,88 +1923,52 @@ STAGE PLANS:
TableScan
alias: p
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_partkey (type: int), p_name (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col2 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col2 (type: int)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_partkey (type: int), p_size (type: int)
- outputColumnNames: p_partkey, p_size
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((p_size = p_size) and (p_partkey = p_partkey)) (type: boolean)
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: p_partkey (type: int), p_size (type: int)
+ keys: p_partkey (type: int), p_name (type: string), p_size (type: int)
mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ key expressions: _col0 (type: int), _col1 (type: string), _col2 (type: int)
+ sort order: +++
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: string), _col2 (type: int)
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col0 (type: int), _col1 (type: string), _col5 (type: int)
- 1 _col0 (type: int), _col1 (type: string), _col2 (type: int)
+ 1 _col1 (type: int), _col0 (type: string), _col2 (type: int)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8
- Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 4
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int), _col2 (type: int)
- 1 _col0 (type: int), _col1 (type: int)
- outputColumnNames: _col1, _col3, _col4
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col3 (type: int), _col1 (type: string), _col4 (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: string), _col2 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: string), _col2 (type: int)
- sort order: +++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: string), _col2 (type: int)
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: int), KEY._col1 (type: int)
+ keys: KEY._col0 (type: int), KEY._col1 (type: string), KEY._col2 (type: int)
mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 3 Data size: 363 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: string), _col0 (type: int), _col2 (type: int)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 3 Data size: 363 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col1 (type: int), _col0 (type: string), _col2 (type: int)
+ sort order: +++
+ Map-reduce partition columns: _col1 (type: int), _col0 (type: string), _col2 (type: int)
+ Statistics: Num rows: 3 Data size: 363 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -2410,8 +2023,7 @@ STAGE PLANS:
Spark
Edges:
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 4 <- Map 3 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2434,88 +2046,52 @@ STAGE PLANS:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_brand (type: string), p_type (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_brand (type: string)
- outputColumnNames: p_brand
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_brand (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_brand = p_brand) (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: p_type (type: string), p_brand (type: string)
+ outputColumnNames: p_type, p_brand
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: p_type (type: string), p_brand (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
condition map:
- Left Semi Join 0 to 1
+ Inner Join 0 to 1
keys:
0 _col1 (type: string), _col2 (type: string)
1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0
- Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 4
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: string), _col2 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
- Reduce Operator Tree:
Group By Operator
- keys: KEY._col0 (type: string)
+ keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 6 Data size: 726 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -2949,13 +2525,11 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 11 <- Map 10 (GROUP, 2)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
Reducer 3 <- Reducer 2 (GROUP, 2)
- Reducer 4 <- Reducer 11 (PARTITION-LEVEL SORT, 2), Reducer 3 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (PARTITION-LEVEL SORT, 2), Reducer 9 (PARTITION-LEVEL SORT, 2)
- Reducer 7 <- Reducer 6 (GROUP, 2)
- Reducer 9 <- Map 8 (GROUP, 2)
+ Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 2), Reducer 8 (PARTITION-LEVEL SORT, 2)
+ Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 8 <- Map 7 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2972,7 +2546,25 @@ STAGE PLANS:
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Map 10
+ Map 5
+ Map Operator Tree:
+ TableScan
+ alias: sc
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key = key) (type: boolean)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: key (type: string), value (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Map 7
Map Operator Tree:
TableScan
alias: s1
@@ -2994,69 +2586,6 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: bigint)
- Map 5
- Map Operator Tree:
- TableScan
- alias: sc
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- Map 8
- Map Operator Tree:
- TableScan
- alias: src
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: key (type: string)
- outputColumnNames: key
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: key (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reducer 11
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: bigint)
- outputColumnNames: _col1
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: _col1 is not null (type: boolean)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: bigint)
- outputColumnNames: _col0
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: bigint)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: bigint)
- sort order: +
- Map-reduce partition columns: _col0 (type: bigint)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -3115,56 +2644,49 @@ STAGE PLANS:
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 6
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col2, _col1
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col2 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Reducer 7
- Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: string)
sort order: ++
Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reducer 9
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Reducer 8
Reduce Operator Tree:
Group By Operator
+ aggregations: count(VALUE._col0)
keys: KEY._col0 (type: string)
mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: bigint)
+ outputColumnNames: _col1
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: _col1 is not null (type: boolean)
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: bigint)
+ outputColumnNames: _col0
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: bigint)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: bigint)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: bigint)
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -3628,10 +3150,8 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Map 4 (PARTITION-LEVEL SORT, 2)
Reducer 3 <- Reducer 2 (SORT, 1)
- Reducer 5 <- Map 4 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
- Reducer 7 <- Map 6 (GROUP, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -3655,37 +3175,22 @@ STAGE PLANS:
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (p_size < 10) (type: boolean)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((p_size < 10) and (p_mfgr = p_mfgr)) (type: boolean)
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: p_mfgr (type: string), p_name (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 8 Data size: 968 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
- Map 6
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_mfgr (type: string)
- outputColumnNames: p_mfgr
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: p_mfgr (type: string)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -3718,42 +3223,6 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Reducer 7
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -3791,10 +3260,8 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Map 8 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 7 <- Map 6 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
+ Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Map 5 (PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -3815,58 +3282,39 @@ STAGE PLANS:
Map 3
Map Operator Tree:
TableScan
- alias: pp
+ alias: p
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: p_type is not null (type: boolean)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: p_type (type: string), p_size (type: int)
+ expressions: p_name (type: string), p_type (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col1 (type: int)
+ key expressions: _col1 (type: string)
sort order: +
- Map-reduce partition columns: _col1 (type: int)
+ Map-reduce partition columns: _col1 (type: string)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
<TRUNCATED>