You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by jc...@apache.org on 2018/11/03 16:35:18 UTC
[03/16] hive git commit: HIVE-20778: Join reordering may not be
triggered if all joins in plan are created by decorrelation logic (Vineet
Garg via Jesus Camacho Rodriguez)
http://git-wip-us.apache.org/repos/asf/hive/blob/ae1eb15d/ql/src/test/results/clientpositive/spark/subquery_select.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/spark/subquery_select.q.out b/ql/src/test/results/clientpositive/spark/subquery_select.q.out
index ad7c8a3..c3c554c 100644
--- a/ql/src/test/results/clientpositive/spark/subquery_select.q.out
+++ b/ql/src/test/results/clientpositive/spark/subquery_select.q.out
@@ -1,4 +1,4 @@
-Warning: Shuffle Join JOIN[21][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+Warning: Shuffle Join JOIN[24][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: explain SELECT p_size, p_size IN (
SELECT MAX(p_size) FROM part)
FROM part
@@ -19,8 +19,8 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
+ Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 1), Reducer 7 (PARTITION-LEVEL SORT, 1)
Reducer 5 <- Map 4 (GROUP, 1)
Reducer 7 <- Map 4 (GROUP, 1)
#### A masked pattern was here ####
@@ -35,9 +35,10 @@ STAGE PLANS:
outputColumnNames: _col0
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
Execution mode: vectorized
Map 4
Map Operator Tree:
@@ -62,35 +63,33 @@ STAGE PLANS:
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col2
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ sort order:
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int), _col2 (type: boolean)
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col1, _col2, _col4
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ 0
+ 1
+ outputColumnNames: _col0, _col2, _col3, _col4
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: int), CASE WHEN ((_col1 = 0L)) THEN (false) WHEN (_col4 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col2 < _col1)) THEN (null) ELSE (false) END (type: boolean)
+ expressions: _col0 (type: int), CASE WHEN ((_col3 = 0L)) THEN (false) WHEN (_col2 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (null) ELSE (false) END (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -103,15 +102,16 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col0)
- mode: complete
+ Select Operator
+ expressions: _col0 (type: int), true (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: boolean)
Reducer 7
Execution mode: vectorized
Reduce Operator Tree:
@@ -120,16 +120,15 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), true (type: boolean)
+ Group By Operator
+ aggregations: count(), count(_col0)
+ mode: complete
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: boolean)
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Stage: Stage-0
Fetch Operator
@@ -137,7 +136,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[21][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+Warning: Shuffle Join JOIN[24][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: SELECT p_size, p_size IN (
SELECT MAX(p_size) FROM part)
FROM part
@@ -153,29 +152,29 @@ POSTHOOK: Input: default@part
1 false
10 false
12 false
+14 false
+14 false
+17 false
+18 false
19 false
-28 false
-34 false
-42 false
2 false
2 false
2 false
2 false
-6 false
-6 false
-7 false
-14 false
-14 false
-17 false
-18 false
23 false
25 false
27 false
+28 false
31 false
+34 false
39 false
40 false
+42 false
45 false
46 true
+6 false
+6 false
+7 false
PREHOOK: query: EXPLAIN SELECT p_size, p_size IN (
SELECT MAX(p_size) FROM part p where p.p_type = part.p_type)
FROM part
@@ -340,32 +339,32 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
1 true
-40 true
-34 true
-42 true
+10 true
+12 true
14 true
+14 true
+17 false
+18 true
19 true
2 true
-6 true
2 true
2 true
-39 true
-10 true
-27 true
-6 true
-18 true
-45 true
-7 true
-46 true
-23 true
-17 false
-31 true
-14 true
2 true
+23 true
25 true
+27 true
28 true
-12 true
-Warning: Shuffle Join JOIN[21][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+31 true
+34 true
+39 true
+40 true
+42 true
+45 true
+46 true
+6 true
+6 true
+7 true
+Warning: Shuffle Join JOIN[24][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: explain SELECT p_size, p_size NOT IN (
SELECT MAX(p_size) FROM part)
FROM part
@@ -386,8 +385,8 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
+ Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 1), Reducer 7 (PARTITION-LEVEL SORT, 1)
Reducer 5 <- Map 4 (GROUP, 1)
Reducer 7 <- Map 4 (GROUP, 1)
#### A masked pattern was here ####
@@ -402,9 +401,10 @@ STAGE PLANS:
outputColumnNames: _col0
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
Execution mode: vectorized
Map 4
Map Operator Tree:
@@ -429,35 +429,33 @@ STAGE PLANS:
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col2
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ sort order:
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int), _col2 (type: boolean)
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col1, _col2, _col4
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ 0
+ 1
+ outputColumnNames: _col0, _col2, _col3, _col4
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: int), CASE WHEN ((_col1 = 0L)) THEN (true) WHEN (_col4 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col2 < _col1)) THEN (null) ELSE (true) END (type: boolean)
+ expressions: _col0 (type: int), CASE WHEN ((_col3 = 0L)) THEN (true) WHEN (_col2 is not null) THEN (false) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (null) ELSE (true) END (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -470,15 +468,16 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col0)
- mode: complete
+ Select Operator
+ expressions: _col0 (type: int), true (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: boolean)
Reducer 7
Execution mode: vectorized
Reduce Operator Tree:
@@ -487,16 +486,15 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), true (type: boolean)
+ Group By Operator
+ aggregations: count(), count(_col0)
+ mode: complete
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: boolean)
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Stage: Stage-0
Fetch Operator
@@ -504,7 +502,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[21][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+Warning: Shuffle Join JOIN[24][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: SELECT p_size, p_size NOT IN (
SELECT MAX(p_size) FROM part)
FROM part
@@ -520,29 +518,29 @@ POSTHOOK: Input: default@part
1 true
10 true
12 true
+14 true
+14 true
+17 true
+18 true
19 true
-28 true
-34 true
-42 true
2 true
2 true
2 true
2 true
-6 true
-6 true
-7 true
-14 true
-14 true
-17 true
-18 true
23 true
25 true
27 true
+28 true
31 true
+34 true
39 true
40 true
+42 true
45 true
46 false
+6 true
+6 true
+7 true
PREHOOK: query: EXPLAIN SELECT p_size, p_size NOT IN (
SELECT MAX(p_size) FROM part p where p.p_type = part.p_type)
FROM part
@@ -646,9 +644,9 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: int)
+ key expressions: _col0 (type: int), _col1 (type: string)
sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: int)
+ Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
value expressions: _col2 (type: boolean)
Reducer 2
@@ -685,9 +683,9 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col3, _col4
Statistics: Num rows: 9 Data size: 1223 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
+ key expressions: _col1 (type: int), _col0 (type: string)
sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Map-reduce partition columns: _col1 (type: int), _col0 (type: string)
Statistics: Num rows: 9 Data size: 1223 Basic stats: COMPLETE Column stats: NONE
value expressions: _col3 (type: bigint), _col4 (type: bigint)
Reducer 4
@@ -696,8 +694,8 @@ STAGE PLANS:
condition map:
Left Outer Join 0 to 1
keys:
- 0 _col0 (type: string), _col1 (type: int)
- 1 _col1 (type: string), _col0 (type: int)
+ 0 _col1 (type: int), _col0 (type: string)
+ 1 _col0 (type: int), _col1 (type: string)
outputColumnNames: _col1, _col3, _col4, _col7
Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
Select Operator
@@ -772,31 +770,31 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
1 false
-40 false
-34 false
-42 false
+10 false
+12 false
+14 false
14 false
+17 true
+18 false
19 false
2 false
-6 false
2 false
2 false
-39 false
-10 false
-27 false
-6 false
-18 false
-45 false
-7 false
-46 false
-23 false
-17 true
-31 false
-14 false
2 false
+23 false
25 false
+27 false
28 false
-12 false
+31 false
+34 false
+39 false
+40 false
+42 false
+45 false
+46 false
+6 false
+6 false
+7 false
Warning: Shuffle Join JOIN[11][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
PREHOOK: query: explain SELECT p_size, EXISTS(SELECT p_size FROM part)
FROM part
@@ -903,32 +901,32 @@ FROM part
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
+1 true
+10 true
+12 true
+14 true
+14 true
+17 true
+18 true
+19 true
2 true
2 true
-34 true
-6 true
-28 true
-42 true
-14 true
-40 true
2 true
+2 true
+23 true
25 true
-18 true
-17 true
-14 true
-19 true
-1 true
-45 true
-10 true
-39 true
27 true
-7 true
-12 true
+28 true
31 true
-6 true
-2 true
+34 true
+39 true
+40 true
+42 true
+45 true
46 true
-23 true
+6 true
+6 true
+7 true
PREHOOK: query: explain SELECT p_size, EXISTS(SELECT p_size FROM part pp where pp.p_type = part.p_type)
FROM part
PREHOOK: type: QUERY
@@ -1043,32 +1041,32 @@ FROM part
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-46 true
1 true
-34 true
-42 true
-19 true
-2 true
+10 true
+12 true
14 true
-6 true
-39 true
-2 true
-27 true
-6 true
-18 true
-7 true
-40 true
-23 true
14 true
17 true
-31 true
+18 true
+19 true
2 true
2 true
-10 true
+2 true
+2 true
+23 true
25 true
+27 true
28 true
-12 true
+31 true
+34 true
+39 true
+40 true
+42 true
45 true
+46 true
+6 true
+6 true
+7 true
Warning: Shuffle Join JOIN[11][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
PREHOOK: query: explain SELECT p_size, NOT EXISTS(SELECT p_size FROM part)
FROM part
@@ -1175,32 +1173,32 @@ FROM part
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
+1 false
+10 false
+12 false
+14 false
+14 false
+17 false
+18 false
+19 false
2 false
2 false
-34 false
-6 false
-28 false
-42 false
-14 false
-40 false
2 false
+2 false
+23 false
25 false
-18 false
-17 false
-14 false
-19 false
-1 false
-45 false
-10 false
-39 false
27 false
-7 false
-12 false
+28 false
31 false
-6 false
-2 false
+34 false
+39 false
+40 false
+42 false
+45 false
46 false
-23 false
+6 false
+6 false
+7 false
PREHOOK: query: explain SELECT p_size, NOT EXISTS(SELECT p_size FROM part pp where pp.p_type = part.p_type)
FROM part
PREHOOK: type: QUERY
@@ -1315,32 +1313,32 @@ FROM part
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-46 false
1 false
-34 false
-42 false
-19 false
-2 false
+10 false
+12 false
14 false
-6 false
-39 false
-2 false
-27 false
-6 false
-18 false
-7 false
-40 false
-23 false
14 false
17 false
-31 false
+18 false
+19 false
2 false
2 false
-10 false
+2 false
+2 false
+23 false
25 false
+27 false
28 false
-12 false
+31 false
+34 false
+39 false
+40 false
+42 false
45 false
+46 false
+6 false
+6 false
+7 false
PREHOOK: query: explain select p_size, (select count(p_name) from part p where p.p_type = part.p_name) from part
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -1454,31 +1452,31 @@ POSTHOOK: query: select p_size, (select count(p_name) from part p where p.p_type
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-31 0
-2 0
-2 0
-17 0
+1 0
10 0
-6 0
+12 0
+14 0
+14 0
+17 0
+18 0
19 0
-6 0
2 0
-39 0
-28 0
-46 0
-18 0
-12 0
+2 0
+2 0
+2 0
23 0
+25 0
+27 0
+28 0
+31 0
34 0
-14 0
-1 0
-45 0
-14 0
+39 0
40 0
-27 0
-2 0
42 0
-25 0
+45 0
+46 0
+6 0
+6 0
7 0
PREHOOK: query: explain select p_size, (select max(p_name) from part p where p.p_type = part.p_name) from part
PREHOOK: type: QUERY
@@ -1593,31 +1591,31 @@ POSTHOOK: query: select p_size, (select max(p_name) from part p where p.p_type =
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-31 NULL
-2 NULL
-2 NULL
-17 NULL
+1 NULL
10 NULL
-6 NULL
+12 NULL
+14 NULL
+14 NULL
+17 NULL
+18 NULL
19 NULL
-6 NULL
2 NULL
-39 NULL
-28 NULL
-46 NULL
-18 NULL
-12 NULL
+2 NULL
+2 NULL
+2 NULL
23 NULL
+25 NULL
+27 NULL
+28 NULL
+31 NULL
34 NULL
-14 NULL
-1 NULL
-45 NULL
-14 NULL
+39 NULL
40 NULL
-27 NULL
-2 NULL
42 NULL
-25 NULL
+45 NULL
+46 NULL
+6 NULL
+6 NULL
7 NULL
Warning: Shuffle Join JOIN[10][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
PREHOOK: query: explain SELECT p_size, (SELECT max(p_size) FROM part)
@@ -1722,32 +1720,32 @@ POSTHOOK: query: SELECT p_size, (SELECT max(p_size) FROM part)
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
+1 46
+10 46
+12 46
+14 46
+14 46
+17 46
+18 46
+19 46
2 46
2 46
-34 46
-6 46
-28 46
-42 46
-14 46
-40 46
2 46
+2 46
+23 46
25 46
-18 46
-17 46
-14 46
-19 46
-1 46
-45 46
-10 46
-39 46
27 46
-7 46
-12 46
+28 46
31 46
-6 46
-2 46
+34 46
+39 46
+40 46
+42 46
+45 46
46 46
-23 46
+6 46
+6 46
+7 46
Warning: Shuffle Join JOIN[16][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 4' is a cross product
PREHOOK: query: explain
select *
@@ -1956,9 +1954,9 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Map 5 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
+ Reducer 4 <- Map 3 (GROUP, 2)
+ Reducer 5 <- Map 6 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -1983,25 +1981,6 @@ STAGE PLANS:
Map 3
Map Operator Tree:
TableScan
- alias: a
- filterExpr: ((key > '9') and value is not null) (type: boolean)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: ((key > '9') and value is not null) (type: boolean)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: value (type: string)
- outputColumnNames: _col1
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: string)
- sort order: +
- Map-reduce partition columns: _col1 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Execution mode: vectorized
- Map 5
- Map Operator Tree:
- TableScan
alias: sc
filterExpr: value is not null (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
@@ -2021,10 +2000,29 @@ STAGE PLANS:
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: string)
Execution mode: vectorized
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
+ Map 6
+ Map Operator Tree:
+ TableScan
+ alias: a
+ filterExpr: ((key > '9') and value is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: ((key > '9') and value is not null) (type: boolean)
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: value (type: string)
+ outputColumnNames: _col1
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col1 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col1 (type: string)
+ Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: vectorized
+ Reducer 2
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
Left Semi Join 0 to 1
keys:
0 _col0 (type: string), _col1 (type: string)
@@ -2039,30 +2037,6 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Reducer 4
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col1 (type: string)
- 1 _col1 (type: string)
- outputColumnNames: _col1, _col2
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reducer 6
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -2084,6 +2058,30 @@ STAGE PLANS:
Map-reduce partition columns: _col1 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string)
+ Reducer 5
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col1 (type: string)
+ 1 _col1 (type: string)
+ outputColumnNames: _col0, _col3
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col3 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
Stage: Stage-0
Fetch Operator
@@ -2112,14 +2110,14 @@ POSTHOOK: Input: default@src
90 val_90
90 val_90
90 val_90
-97 val_97
-97 val_97
-98 val_98
-98 val_98
92 val_92
95 val_95
95 val_95
96 val_96
+97 val_97
+97 val_97
+98 val_98
+98 val_98
PREHOOK: query: CREATE table tnull_n3(i int)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
@@ -2137,7 +2135,7 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: _dummy_database@_dummy_table
POSTHOOK: Output: default@tnull_n3
POSTHOOK: Lineage: tnull_n3.i EXPRESSION []
-Warning: Shuffle Join JOIN[16][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+Warning: Shuffle Join JOIN[19][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: explain select p_size, p_size IN (select i from tnull_n3) from part
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -2156,10 +2154,10 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Map 4 (GROUP, 1)
- Reducer 7 <- Map 6 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
+ Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 1), Reducer 7 (PARTITION-LEVEL SORT, 1)
+ Reducer 5 <- Map 4 (GROUP, 2)
+ Reducer 7 <- Map 6 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2172,9 +2170,10 @@ STAGE PLANS:
outputColumnNames: _col0
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
Execution mode: vectorized
Map 4
Map Operator Tree:
@@ -2186,14 +2185,15 @@ STAGE PLANS:
outputColumnNames: i
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: count(), count(i)
+ keys: i (type: int)
mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
Map 6
Map Operator Tree:
@@ -2205,49 +2205,46 @@ STAGE PLANS:
outputColumnNames: i
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: i (type: int)
+ aggregations: count(), count(i)
mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col2
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ sort order:
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int), _col2 (type: boolean)
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col1, _col2, _col4
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ 0
+ 1
+ outputColumnNames: _col0, _col2, _col3, _col4
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: int), CASE WHEN ((_col1 = 0L)) THEN (false) WHEN (_col4 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col2 < _col1)) THEN (null) ELSE (false) END (type: boolean)
+ expressions: _col0 (type: int), CASE WHEN ((_col3 = 0L)) THEN (false) WHEN (_col2 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (null) ELSE (false) END (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -2256,18 +2253,6 @@ STAGE PLANS:
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
- aggregations: count(VALUE._col0), count(VALUE._col1)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
- Reducer 7
- Execution mode: vectorized
- Reduce Operator Tree:
- Group By Operator
keys: KEY._col0 (type: int)
mode: mergepartial
outputColumnNames: _col0
@@ -2282,6 +2267,18 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 1 Data size: 2 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: boolean)
+ Reducer 7
+ Execution mode: vectorized
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0), count(VALUE._col1)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Stage: Stage-0
Fetch Operator
@@ -2289,7 +2286,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[16][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+Warning: Shuffle Join JOIN[19][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: select p_size, p_size IN (select i from tnull_n3) from part
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -2303,29 +2300,29 @@ POSTHOOK: Input: default@tnull_n3
1 NULL
10 NULL
12 NULL
+14 NULL
+14 NULL
+17 NULL
+18 NULL
19 NULL
-28 NULL
-34 NULL
-42 NULL
2 NULL
2 NULL
2 NULL
2 NULL
-6 NULL
-6 NULL
-7 NULL
-14 NULL
-14 NULL
-17 NULL
-18 NULL
23 NULL
25 NULL
27 NULL
+28 NULL
31 NULL
+34 NULL
39 NULL
40 NULL
+42 NULL
45 NULL
46 NULL
+6 NULL
+6 NULL
+7 NULL
PREHOOK: query: CREATE TABLE tempty_n3(i int)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
@@ -2435,32 +2432,32 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
POSTHOOK: Input: default@tempty_n3
#### A masked pattern was here ####
+1 0
+10 0
+12 0
+14 0
+14 0
+17 0
+18 0
+19 0
+2 0
2 0
2 0
-34 0
-6 0
-28 0
-42 0
-14 0
-40 0
2 0
+23 0
25 0
-18 0
-17 0
-14 0
-19 0
-1 0
-45 0
-10 0
-39 0
27 0
-7 0
-12 0
+28 0
31 0
-6 0
-2 0
+34 0
+39 0
+40 0
+42 0
+45 0
46 0
-23 0
+6 0
+6 0
+7 0
Warning: Shuffle Join JOIN[10][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
PREHOOK: query: explain select p_size, (select max(i) from tempty_n3) from part
PREHOOK: type: QUERY
@@ -2564,32 +2561,32 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
POSTHOOK: Input: default@tempty_n3
#### A masked pattern was here ####
+1 NULL
+10 NULL
+12 NULL
+14 NULL
+14 NULL
+17 NULL
+18 NULL
+19 NULL
2 NULL
2 NULL
-34 NULL
-6 NULL
-28 NULL
-42 NULL
-14 NULL
-40 NULL
2 NULL
+2 NULL
+23 NULL
25 NULL
-18 NULL
-17 NULL
-14 NULL
-19 NULL
-1 NULL
-45 NULL
-10 NULL
-39 NULL
27 NULL
-7 NULL
-12 NULL
+28 NULL
31 NULL
-6 NULL
-2 NULL
+34 NULL
+39 NULL
+40 NULL
+42 NULL
+45 NULL
46 NULL
-23 NULL
+6 NULL
+6 NULL
+7 NULL
PREHOOK: query: DROP table tempty_n3
PREHOOK: type: DROPTABLE
PREHOOK: Input: default@tempty_n3
@@ -2723,32 +2720,32 @@ POSTHOOK: query: SELECT p_size, 1+(SELECT max(p_size) FROM part p
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-46 47
1 2
-34 35
-42 43
-19 20
-2 3
+10 11
+12 13
14 15
-6 7
-39 40
-2 3
-27 28
-6 7
-18 19
-7 8
-40 41
-23 24
14 15
17 32
-31 32
+18 19
+19 20
2 3
2 3
-10 11
+2 3
+2 3
+23 24
25 26
+27 28
28 29
-12 13
+31 32
+34 35
+39 40
+40 41
+42 43
45 46
+46 47
+6 7
+6 7
+7 8
PREHOOK: query: explain SELECT p_size, (SELECT count(p_size) FROM part p
WHERE p.p_type = part.p_type) IS NULL from part
PREHOOK: type: QUERY
@@ -2866,34 +2863,34 @@ POSTHOOK: query: SELECT p_size, (SELECT count(p_size) FROM part p
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-46 false
1 false
-34 false
-42 false
-19 false
-2 false
+10 false
+12 false
14 false
-6 false
-39 false
-2 false
-27 false
-6 false
-18 false
-7 false
-40 false
-23 false
14 false
17 false
-31 false
+18 false
+19 false
2 false
2 false
-10 false
+2 false
+2 false
+23 false
25 false
+27 false
28 false
-12 false
+31 false
+34 false
+39 false
+40 false
+42 false
45 false
-Warning: Shuffle Join JOIN[23][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
-Warning: Shuffle Join JOIN[26][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
+46 false
+6 false
+6 false
+7 false
+Warning: Shuffle Join JOIN[23][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 4' is a cross product
+Warning: Shuffle Join JOIN[26][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Work 'Reducer 5' is a cross product
PREHOOK: query: explain select p_type, (select p_size from part order by p_size limit 1) = 1 from part order by p_type
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -2910,12 +2907,12 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 7 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 1), Reducer 9 (PARTITION-LEVEL SORT, 1)
- Reducer 4 <- Reducer 3 (SORT, 1)
- Reducer 6 <- Map 5 (SORT, 1)
- Reducer 7 <- Reducer 6 (GROUP, 1)
- Reducer 9 <- Map 8 (SORT, 1)
+ Reducer 2 <- Map 1 (SORT, 1)
+ Reducer 3 <- Reducer 2 (GROUP, 1)
+ Reducer 4 <- Reducer 3 (PARTITION-LEVEL SORT, 1), Reducer 8 (PARTITION-LEVEL SORT, 1)
+ Reducer 5 <- Map 9 (PARTITION-LEVEL SORT, 1), Reducer 4 (PARTITION-LEVEL SORT, 1)
+ Reducer 6 <- Reducer 5 (SORT, 1)
+ Reducer 8 <- Map 7 (SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -2924,15 +2921,15 @@ STAGE PLANS:
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: p_type (type: string)
+ expressions: p_size (type: int)
outputColumnNames: _col0
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: int)
+ sort order: +
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
Execution mode: vectorized
- Map 5
+ Map 7
Map Operator Tree:
TableScan
alias: part
@@ -2945,48 +2942,81 @@ STAGE PLANS:
key expressions: _col0 (type: int)
sort order: +
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ TopN Hash Memory Usage: 0.1
Execution mode: vectorized
- Map 8
+ Map 9
Map Operator Tree:
TableScan
alias: part
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: p_size (type: int)
+ expressions: p_type (type: string)
outputColumnNames: _col0
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
+ sort order:
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- TopN Hash Memory Usage: 0.1
+ value expressions: _col0 (type: string)
Execution mode: vectorized
Reducer 2
+ Execution mode: vectorized
+ Reduce Operator Tree:
+ Select Operator
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Limit
+ Number of rows: 1
+ Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Reducer 3
+ Execution mode: vectorized
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0)
+ mode: mergepartial
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (sq_count_check(_col0) <= 1) (type: boolean)
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reducer 4
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
0
1
- outputColumnNames: _col0
- Statistics: Num rows: 26 Data size: 3381 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col1
+ Statistics: Num rows: 1 Data size: 130 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
- Statistics: Num rows: 26 Data size: 3381 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string)
- Reducer 3
+ Statistics: Num rows: 1 Data size: 130 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: int)
+ Reducer 5
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
0
1
- outputColumnNames: _col0, _col2
+ outputColumnNames: _col1, _col2
Statistics: Num rows: 26 Data size: 6553 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: string), (_col2 = 1) (type: boolean)
+ expressions: _col2 (type: string), (_col1 = 1) (type: boolean)
outputColumnNames: _col0, _col1
Statistics: Num rows: 26 Data size: 6553 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
@@ -2994,7 +3024,7 @@ STAGE PLANS:
sort order: +
Statistics: Num rows: 26 Data size: 6553 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: boolean)
- Reducer 4
+ Reducer 6
Execution mode: vectorized
Reduce Operator Tree:
Select Operator
@@ -3008,40 +3038,7 @@ STAGE PLANS:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 6
- Execution mode: vectorized
- Reduce Operator Tree:
- Select Operator
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Limit
- Number of rows: 1
- Statistics: Num rows: 1 Data size: 121 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint)
- Reducer 7
- Execution mode: vectorized
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0)
- mode: mergepartial
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (sq_count_check(_col0) <= 1) (type: boolean)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reducer 9
+ Reducer 8
Execution mode: vectorized
Reduce Operator Tree:
Select Operator
@@ -3062,8 +3059,8 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[23][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
-Warning: Shuffle Join JOIN[26][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
+Warning: Shuffle Join JOIN[23][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 4' is a cross product
+Warning: Shuffle Join JOIN[26][tables = [$hdt$_1, $hdt$_2, $hdt$_0]] in Work 'Reducer 5' is a cross product
PREHOOK: query: select p_type, (select p_size from part order by p_size limit 1) = 1 from part order by p_type
PREHOOK: type: QUERY
PREHOOK: Input: default@part
@@ -3098,7 +3095,7 @@ STANDARD ANODIZED TIN true
STANDARD BURNISHED TIN true
STANDARD PLATED TIN true
STANDARD POLISHED STEEL true
-Warning: Shuffle Join JOIN[47][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Work 'Reducer 4' is a cross product
+Warning: Shuffle Join JOIN[50][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3, $hdt$_4]] in Work 'Reducer 5' is a cross product
PREHOOK: query: EXPLAIN SELECT p_size, p_size IN (
SELECT MAX(p_size) FROM part p where p.p_type = part.p_type) AND
p_name IN (SELECT min(p_name) from part)
@@ -3125,8 +3122,8 @@ STAGE PLANS:
Reducer 13 <- Map 10 (GROUP, 1)
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 7 (PARTITION-LEVEL SORT, 2)
Reducer 3 <- Reducer 2 (PARTITION-LEVEL SORT, 2), Reducer 9 (PARTITION-LEVEL SORT, 2)
- Reducer 4 <- Reducer 11 (PARTITION-LEVEL SORT, 1), Reducer 3 (PARTITION-LEVEL SORT, 1)
- Reducer 5 <- Reducer 13 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
+ Reducer 4 <- Reducer 11 (PARTITION-LEVEL SORT, 2), Reducer 3 (PARTITION-LEVEL SORT, 2)
+ Reducer 5 <- Reducer 13 (PARTITION-LEVEL SORT, 1), Reducer 4 (PARTITION-LEVEL SORT, 1)
Reducer 7 <- Map 6 (GROUP PARTITION-LEVEL SORT, 2)
Reducer 9 <- Map 6 (GROUP, 2)
#### A masked pattern was here ####
@@ -3196,15 +3193,16 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col0)
- mode: complete
+ Select Operator
+ expressions: _col0 (type: string), true (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 192 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
+ Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: boolean)
Reducer 13
Execution mode: vectorized
Reduce Operator Tree:
@@ -3213,16 +3211,15 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), true (type: boolean)
+ Group By Operator
+ aggregations: count(), count(_col0)
+ mode: complete
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 192 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 1 Data size: 184 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: boolean)
+ sort order:
+ Statistics: Num rows: 1 Data size: 192 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -3250,37 +3247,37 @@ STAGE PLANS:
outputColumnNames: _col0, _col2, _col4, _col5, _col8
Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: string)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: string), _col2 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col8 (type: boolean)
+ value expressions: _col2 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col8 (type: boolean)
Reducer 4
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
- 0
- 1
- outputColumnNames: _col0, _col2, _col4, _col5, _col8, _col9, _col10
- Statistics: Num rows: 30 Data size: 9597 Basic stats: COMPLETE Column stats: NONE
+ 0 _col0 (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col2, _col4, _col5, _col8, _col10
+ Statistics: Num rows: 33 Data size: 4187 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 30 Data size: 9597 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col8 (type: boolean), _col9 (type: bigint), _col10 (type: bigint)
+ sort order:
+ Statistics: Num rows: 33 Data size: 4187 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: string), _col2 (type: int), _col4 (type: bigint), _col5 (type: bigint), _col8 (type: boolean), _col10 (type: boolean)
Reducer 5
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
- 0 _col0 (type: string)
- 1 _col0 (type: string)
- outputColumnNames: _col0, _col2, _col4, _col5, _col8, _col9, _col10, _col12
+ 0
+ 1
+ outputColumnNames: _col0, _col2, _col4, _col5, _col8, _col10, _col11, _col12
Statistics: Num rows: 33 Data size: 10556 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col2 (type: int), (CASE WHEN ((_col4 = 0L)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col2 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (null) ELSE (false) END and CASE WHEN ((_col9 = 0L)) THEN (false) WHEN (_col12 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col10 < _col9)) THEN (null) ELSE (false) END) (type: boolean)
+ expressions: _col2 (type: int), (CASE WHEN ((_col4 = 0L)) THEN (false) WHEN (_col4 is null) THEN (false) WHEN (_col8 is not null) THEN (true) WHEN (_col2 is null) THEN (null) WHEN ((_col5 < _col4)) THEN (null) ELSE (false) END and CASE WHEN ((_col11 = 0L)) THEN (false) WHEN (_col10 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col12 < _col11)) THEN (null) ELSE (false) END) (type: boolean)
outputColumnNames: _col0, _col1
Statistics: Num rows: 33 Data size: 10556 Basic stats: COMPLETE Column stats: NONE
File Output Operator
@@ -3340,7 +3337,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[47][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Work 'Reducer 4' is a cross product
+Warning: Shuffle Join JOIN[50][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3, $hdt$_4]] in Work 'Reducer 5' is a cross product
PREHOOK: query: SELECT p_size, p_size IN (
SELECT MAX(p_size) FROM part p where p.p_type = part.p_type) AND
p_name IN (SELECT min(p_name) from part)
@@ -3355,31 +3352,31 @@ FROM part
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-31 true
-2 false
-2 false
-17 false
+1 false
10 false
-6 false
+12 false
+14 false
+14 false
+17 false
+18 false
19 false
-6 false
2 false
-39 false
-28 false
-46 false
-18 false
-12 false
+2 false
+2 false
+2 false
23 false
+25 false
+27 false
+28 false
+31 true
34 false
-14 false
-1 false
-45 false
-14 false
+39 false
40 false
-27 false
-2 false
42 false
-25 false
+45 false
+46 false
+6 false
+6 false
7 false
PREHOOK: query: explain SELECT p_size, NOT EXISTS(SELECT p_size FROM part pp where pp.p_type = part.p_type)
FROM part
@@ -3495,35 +3492,35 @@ FROM part
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-46 false
1 false
-34 false
-42 false
-19 false
-2 false
+10 false
+12 false
14 false
-6 false
-39 false
-2 false
-27 false
-6 false
-18 false
-7 false
-40 false
-23 false
14 false
17 false
-31 false
+18 false
+19 false
2 false
2 false
-10 false
+2 false
+2 false
+23 false
25 false
+27 false
28 false
-12 false
+31 false
+34 false
+39 false
+40 false
+42 false
45 false
+46 false
+6 false
+6 false
+7 false
Warning: Shuffle Join JOIN[12][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 5' is a cross product
-Warning: Shuffle Join JOIN[41][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
-Warning: Shuffle Join JOIN[33][tables = [$hdt$_2, $hdt$_3]] in Work 'Reducer 11' is a cross product
+Warning: Shuffle Join JOIN[28][tables = [$hdt$_2, $hdt$_3]] in Work 'Reducer 10' is a cross product
+Warning: Shuffle Join JOIN[44][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: explain select p_size, (p_size IN
(select (select max(p_size) from part) as sb from part order by sb limit 1)) = true
from part
@@ -3544,15 +3541,15 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 11 <- Map 10 (PARTITION-LEVEL SORT, 1), Reducer 14 (PARTITION-LEVEL SORT, 1)
- Reducer 12 <- Reducer 11 (SORT, 1)
+ Reducer 10 <- Map 4 (PARTITION-LEVEL SORT, 1), Reducer 14 (PARTITION-LEVEL SORT, 1)
+ Reducer 11 <- Reducer 10 (SORT, 1)
+ Reducer 12 <- Reducer 11 (GROUP, 1)
Reducer 14 <- Map 13 (GROUP, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 7 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 12 (PARTITION-LEVEL SORT, 2), Reducer 2 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Map 10 (PARTITION-LEVEL SORT, 1), Reducer 9 (PARTITION-LEVEL SORT, 1)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 2), Reducer 6 (PARTITION-LEVEL SORT, 2)
+ Reducer 3 <- Reducer 12 (PARTITION-LEVEL SORT, 1), Reducer 2 (PARTITION-LEVEL SORT, 1)
+ Reducer 5 <- Map 4 (PARTITION-LEVEL SORT, 1), Reducer 8 (PARTITION-LEVEL SORT, 1)
Reducer 6 <- Reducer 5 (SORT, 1)
- Reducer 7 <- Reducer 6 (GROUP, 1)
- Reducer 9 <- Map 13 (GROUP, 1)
+ Reducer 8 <- Map 13 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -3565,20 +3562,10 @@ STAGE PLANS:
outputColumnNames: _col0
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
- Execution mode: vectorized
- Map 10
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: COMPLETE
- Select Operator
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
Execution mode: vectorized
Map 13
Map Operator Tree:
@@ -3599,7 +3586,18 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 4 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Execution mode: vectorized
- Reducer 11
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: part
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: COMPLETE
+ Select Operator
+ Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 26 Data size: 104 Basic stats: COMPLETE Column stats: COMPLETE
+ Execution mode: vectorized
+ Reducer 10
Reduce Operator Tree:
Join Operator
condition map:
@@ -3617,8 +3615,7 @@ STAGE PLANS:
key expressions: _col0 (type: int)
sort order: +
Statistics: Num rows: 26 Data size: 234 Basic stats: COMPLETE Column stats: NONE
- TopN Hash Memory Usage: 0.1
- Reducer 12
+ Reducer 11
Execution mode: vectorized
Reduce Operator Tree:
Select Operator
@@ -3628,16 +3625,27 @@ STAGE PLANS:
Limit
Number of rows: 1
Statistics: Num rows: 1 Data size: 9 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), true (type: boolean)
+ Group By Operator
+ aggregations: count(), count(_col0)
+ mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 9 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 1 Data size: 9 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: boolean)
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
+ Reducer 12
+ Execution mode: vectorized
+ Reduce Operator Tree:
+ Group By Operator
+ aggregations: count(VALUE._col0), count(VALUE._col1)
+ mode: mergepartial
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: bigint)
Reducer 14
Execution mode: vectorized
Reduce Operator Tree:
@@ -3654,35 +3662,33 @@ STAGE PLANS:
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Outer Join 0 to 1
keys:
- 0
- 1
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col2
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 26 Data size: 3589 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: bigint), _col2 (type: bigint)
+ sort order:
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int), _col2 (type: boolean)
Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
+ Inner Join 0 to 1
keys:
- 0 _col0 (type: int)
- 1 _col0 (type: int)
- outputColumnNames: _col0, _col1, _col2, _col4
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ 0
+ 1
+ outputColumnNames: _col0, _col2, _col3, _col4
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: int), CASE WHEN ((_col1 = 0L)) THEN (false) WHEN (_col4 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col2 < _col1)) THEN (null) ELSE (false) END (type: boolean)
+ expressions: _col0 (type: int), CASE WHEN ((_col3 = 0L)) THEN (false) WHEN (_col2 is not null) THEN (true) WHEN (_col0 is null) THEN (null) WHEN ((_col4 < _col3)) THEN (null) ELSE (false) END (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 28 Data size: 3947 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 28 Data size: 3937 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -3705,6 +3711,7 @@ STAGE PLANS:
key expressions: _col0 (type: int)
sort order: +
Statistics: Num rows: 26 Data size: 234 Basic stats: COMPLETE Column stats: NONE
+ TopN Hash Memory Usage: 0.1
Reducer 6
Execution mode: vectorized
Reduce Operator Tree:
@@ -3715,28 +3722,17 @@ STAGE PLANS:
Limit
Number of rows: 1
Statistics: Num rows: 1 Data size: 9 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(), count(_col0)
- mode: hash
+ Select Operator
+ expressions: _col0 (type: int), true (type: boolean)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 1 Data size: 9 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
- Reducer 7
- Execution mode: vectorized
- Reduce Operator Tree:
- Group By Operator
- aggregations: count(VALUE._col0), count(VALUE._col1)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: bigint)
- Reducer 9
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 1 Data size: 9 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: boolean)
+ Reducer 8
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -3756,8 +3752,8 @@ STAGE PLANS:
ListSink
Warning: Shuffle Join JOIN[12][tables = [$hdt$_1, $hdt$_2]] in Work 'Reducer 5' is a cross product
-Warning: Shuffle Join JOIN[41][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
-Warning: Shuffle Join JOIN[33][tables = [$hdt$_2, $hdt$_3]] in Work 'Reducer 11' is a cross product
+Warning: Shuffle Join JOIN[28][tables = [$hdt$_2, $hdt$_3]] in Work 'Reducer 10' is a cross product
+Warning: Shuffle Join JOIN[44][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
PREHOOK: query: select p_size, (p_size IN
(select (select max(p_size) from part) as sb from part order by sb limit 1)) = true
from part
@@ -3773,29 +3769,29 @@ POSTHOOK: Input: default@part
1 false
10 false
12 false
+14 false
+14 false
+17 false
+18 false
19 false
-28 false
-34 false
-42 false
2 false
2 false
2 false
2 false
-6 false
-6 false
-7 false
-14 false
-14 false
-17 false
-18 false
23 false
25 false
27 false
+28 false
31 false
+34 false
39 false
40 false
+42 false
45 false
46 true
+6 false
+6 false
+7 false
Warning: Shuffle Join JOIN[25][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
Warning: Shuffle Join JOIN[28][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
Warning: Shuffle Join JOIN[31][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Work 'Reducer 4' is a cross product
@@ -4387,17 +4383,17 @@ POSTHOOK: query: select o.p_size, (select count(distinct p_type) from part p whe
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
-46 1
-27 1
-39 1
23 1
+25 1
+27 1
28 1
-45 1
-40 1
31 1
-42 1
34 1
-25 1
+39 1
+40 1
+42 1
+45 1
+46 1
Warning: Shuffle Join JOIN[28][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
Warning: Shuffle Join JOIN[31][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Work 'Reducer 3' is a cross product
Warning: Shuffle Join JOIN[34][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Work 'Reducer 4' is a cross product
@@ -4828,33 +4824,33 @@ POSTHOOK: query: select t1.p_size,
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
+1 28
+10 28
+12 28
+14 28
+14 28
+17 28
+18 28
+19 28
+2 28
2 28
2 28
-34 28
-6 28
-28 28
-42 28
-14 28
-40 28
2 28
+23 28
25 28
-18 28
-17 28
-14 28
-19 28
-1 28
-45 28
-10 28
-39 28
27 28
-7 28
-12 28
+28 28
31 28
-6 28
-2 28
+34 28
+39 28
+40 28
+42 28
+45 28
46 28
-23 28
-Warning: Shuffle Join JOIN[44][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+6 28
+6 28
+7 28
+Warning: Shuffle Join JOIN[47][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
PREHOOK: query: explain select t1.p_size,
(select count(*) from part p, part pp where p.p_size = pp.p_size and p.p_type = pp.p_type
and (select sum(p_size) from part a1 where a1.p_partkey = p.p_partkey
@@ -4879,11 +4875,12 @@ STAGE PLANS:
Stage: Stage-1
Spark
Edges:
- Reducer 11 <- Map 10 (GROUP, 2)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 6 (PARTITION-LEVEL SORT, 1)
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Map 7 (PARTITION-LEVEL SORT, 2)
- Reducer 5 <- Reducer 11 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2), Reducer 9 (PARTITION-LEVEL SORT, 2)
- Reducer 6 <- Reducer 5 (GROUP, 1)
+ Reducer 12 <- Map 11 (GROUP, 2)
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 7 (PARTITION-LEVEL SORT, 1)
+ Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 2), Reducer 9 (PARTITION-LEVEL SORT, 2)
+ Reducer 5 <- Map 10 (PARTITION-LEVEL SORT, 2), Reducer 4 (PARTITION-LEVEL SORT, 2)
+ Reducer 6 <- Reducer 12 (PARTITION-LEVEL SORT, 2), Reducer 5 (PARTITION-LEVEL SORT, 2)
+ Reducer 7 <- Reducer 6 (GROUP, 1)
Reducer 9 <- Map 8 (GROUP PARTITION-LEVEL SORT, 2)
#### A masked pattern was here ####
Vertices:
@@ -4904,6 +4901,25 @@ STAGE PLANS:
Map 10
Map Operator Tree:
TableScan
+ alias: pp
+ filterExpr: (p_size is not null and p_type is not null) (type: boolean)
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_size is not null and p_type is not null) (type: boolean)
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: p_type (type: string), p_size (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Execution mode: vectorized
+ Map 11
+ Map Operator Tree:
+ TableScan
alias: a1
filterExpr: p_partkey is not null (type: boolean)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
@@ -4937,30 +4953,11 @@ STAGE PLANS:
outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col1 (type: string), _col2 (type: int)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col2 (type: int)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int)
- Execution mode: vectorized
- Map 7
- Map Operator Tree:
- TableScan
- alias: pp
- filterExpr: (p_size is not null and p_type is not null) (type: boolean)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (p_size is not null and p_type is not null) (type: boolean)
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: p_type (type: string), p_size (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: int)
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: string), _col2 (type: int)
Execution mode: vectorized
Map 8
Map Operator Tree:
@@ -4982,7 +4979,7 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
- Reducer 11
+ Reducer 12
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -5028,28 +5025,42 @@ STAGE PLANS:
Reduce Operator Tree:
Join Operator
condition map:
+ Left Outer Join 0 to 1
+ keys:
+ 0 _col0 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col1 (type: string), _col2 (type: int)
+ sort order: ++
+ Map-reduce partition columns: _col1 (type: string), _col2 (type: int)
+ Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: int)
+ Reducer 5
+ Reduce Operator Tree:
+ Join Operator
+ condition map:
Inner Join 0 to 1
keys:
0 _col1 (type: string), _col2 (type: int)
1 _col0 (type: string), _col1 (type: int)
outputColumnNames: _col0
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 28 Data size: 3461 Basic stats: COMPLETE Column stats: NONE
- Reducer 5
+ Statistics: Num rows: 30 Data size: 3807 Basic stats: COMPLETE Column stats: NONE
+ Reducer 6
Reduce Operator Tree:
Join Operator
condition map:
- Left Outer Join 0 to 1
- Inner Join 0 to 2
+ Inner Join 0 to 1
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- 2 _col0 (type: int)
- Statistics: Num rows: 61 Data size: 7614 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 33 Data size: 4187 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: count()
mode: hash
@@ -5059,7 +5070,7 @@ STAGE PLANS:
sort order:
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint)
- Reducer 6
+ Reducer 7
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
@@ -5104,7 +5115,7 @@ STAGE PLANS:
Processor Tree:
ListSink
-Warning: Shuffle Join JOIN[44][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
+Warning: Shuffle Join JOIN[47][tables = [$hdt$_0, $hdt$_1]] in Work 'Reducer 2' is a cross product
PREHOOK: query: select t1.p_size,
(select count(*) from part p, part pp where p.p_size = pp.p_size and p.p_type = pp.p_type
and (select sum(p_size) from part a1 where a1.p_partkey = p.p_partkey
@@ -5121,33 +5132,33 @@ POSTHOOK: query: select t1.p_size,
POSTHOOK: type: QUERY
POSTHOOK: Input: default@part
#### A masked pattern was here ####
+1 28
+10 28
+12 28
+14 28
+14 28
+17 28
+18 28
+19 28
2 28
<TRUNCATED>