You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by xu...@apache.org on 2014/11/25 14:03:23 UTC
svn commit: r1641614 [41/43] - in /hive/branches/spark:
itests/src/test/resources/
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/ ql/sr...
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_3.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_3.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_3.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_3.q.out Tue Nov 25 13:03:18 2014
@@ -63,15 +63,13 @@ select /*+ mapjoin(b) */ count(*) from t
on a.key=b.key and a.value=b.value where a.key < 10
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -82,12 +80,23 @@ STAGE PLANS:
Filter Operator
predicate: ((key is not null and value is not null) and (key < 10)) (type: boolean)
Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value (type: string)
- Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Map 4
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 key (type: string), value (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -95,31 +104,31 @@ STAGE PLANS:
Filter Operator
predicate: ((key is not null and value is not null) and (key < 10)) (type: boolean)
Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value (type: string)
- Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0
- 1
- Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 key (type: string), value (type: string)
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Local Work:
+ Map Reduce Local Work
Reducer 3
Reduce Operator Tree:
Group By Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_4.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_4.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_4.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_4.q.out Tue Nov 25 13:03:18 2014
@@ -61,15 +61,13 @@ select /*+ mapjoin(b) */ count(*) from t
on a.key=b.key and a.value=b.value where a.key < 10
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -80,12 +78,23 @@ STAGE PLANS:
Filter Operator
predicate: ((key is not null and value is not null) and (key < 10)) (type: boolean)
Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value (type: string)
- Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Map 4
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 key (type: string), value (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -93,31 +102,31 @@ STAGE PLANS:
Filter Operator
predicate: ((key is not null and value is not null) and (key < 10)) (type: boolean)
Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value (type: string)
- Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0
- 1
- Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 key (type: string), value (type: string)
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 45 Data size: 478 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Local Work:
+ Map Reduce Local Work
Reducer 3
Reduce Operator Tree:
Group By Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
Files hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out (original) and hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_5.q.out Tue Nov 25 13:03:18 2014 differ
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
Files hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out (original) and hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_6.q.out Tue Nov 25 13:03:18 2014 differ
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
Files hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out (original) and hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_7.q.out Tue Nov 25 13:03:18 2014 differ
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_8.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_8.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_8.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/sort_merge_join_desc_8.q.out Tue Nov 25 13:03:18 2014
@@ -108,15 +108,13 @@ select /*+ mapjoin(b) */ count(*) from t
on a.key=b.key where a.key < 10
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -127,12 +125,23 @@ STAGE PLANS:
Filter Operator
predicate: (key < 10) (type: boolean)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string)
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Map 4
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string)
+ 1 key (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -140,31 +149,31 @@ STAGE PLANS:
Filter Operator
predicate: (key < 10) (type: boolean)
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string)
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0
- 1
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string)
+ 1 key (type: string)
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 182 Data size: 1939 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Local Work:
+ Map Reduce Local Work
Reducer 3
Reduce Operator Tree:
Group By Operator
@@ -216,15 +225,13 @@ select /*+ mapjoin(b) */ count(*) from t
on a.key=b.key and a.value2=b.value2 where a.key < 10
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -235,12 +242,23 @@ STAGE PLANS:
Filter Operator
predicate: ((key is not null and value2 is not null) and (key < 10)) (type: boolean)
Statistics: Num rows: 41 Data size: 517 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value2 (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value2 (type: string)
- Statistics: Num rows: 41 Data size: 517 Basic stats: COMPLETE Column stats: NONE
- Map 4
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string), value2 (type: string)
+ 1 key (type: string), value2 (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -248,31 +266,31 @@ STAGE PLANS:
Filter Operator
predicate: ((key is not null and value2 is not null) and (key < 10)) (type: boolean)
Statistics: Num rows: 41 Data size: 837 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value2 (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value2 (type: string)
- Statistics: Num rows: 41 Data size: 837 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0
- 1
- Statistics: Num rows: 45 Data size: 920 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- Statistics: Num rows: 45 Data size: 920 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count()
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0
+ 1
+ keys:
+ 0 key (type: string), value2 (type: string)
+ 1 key (type: string), value2 (type: string)
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 45 Data size: 920 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ Statistics: Num rows: 45 Data size: 920 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count()
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint)
+ Local Work:
+ Map Reduce Local Work
Reducer 3
Reduce Operator Tree:
Group By Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_exists.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_exists.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_exists.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_exists.q.out Tue Nov 25 13:03:18 2014
@@ -1,4 +1,6 @@
-PREHOOK: query: -- no agg, corr
+PREHOOK: query: -- SORT_QUERY_RESULTS
+
+-- no agg, corr
explain
select *
from src b
@@ -8,7 +10,9 @@ where exists
where b.value = a.value and a.key = b.key and a.value > 'val_9'
)
PREHOOK: type: QUERY
-POSTHOOK: query: -- no agg, corr
+POSTHOOK: query: -- SORT_QUERY_RESULTS
+
+-- no agg, corr
explain
select *
from src b
@@ -19,30 +23,16 @@ where exists
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 3 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
- Map 1
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (value is not null and key is not null) (type: boolean)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: value (type: string), key (type: string)
- sort order: ++
- Map-reduce partition columns: value (type: string), key (type: string)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -59,32 +49,54 @@ STAGE PLANS:
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey1} {KEY.reducesinkkey0}
- 1
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 value (type: string), key (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (value is not null and key is not null) (type: boolean)
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 value (type: string), key (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 2
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Stage: Stage-0
Fetch Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_in.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_in.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_in.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/subquery_in.q.out Tue Nov 25 13:03:18 2014
@@ -15,14 +15,13 @@ from src
where src.key in (select key from src s1 where s1.key > '9')
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 3 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -42,12 +41,21 @@ STAGE PLANS:
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string)
- sort order: +
- Map-reduce partition columns: _col0 (type: string)
- Statistics: Num rows: 166 Data size: 1763 Basic stats: COMPLETE Column stats: NONE
- Map 3
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 key (type: string)
+ 1 _col0 (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: src
@@ -55,33 +63,32 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string)
- sort order: +
- Map-reduce partition columns: key (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {VALUE._col0}
- 1
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 key (type: string)
+ 1 _col0 (type: string)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Stage: Stage-0
Fetch Operator
@@ -133,30 +140,16 @@ where b.key in
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 3 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
- Map 1
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (key is not null and value is not null) (type: boolean)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value (type: string)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -173,32 +166,54 @@ STAGE PLANS:
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {KEY.reducesinkkey1}
- 1
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Map 2
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Stage: Stage-0
Fetch Operator
@@ -258,15 +273,15 @@ part where part.p_size in
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
Edges:
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1)
- Reducer 4 <- Map 5 (PARTITION-LEVEL SORT, 1), Reducer 3 (PARTITION-LEVEL SORT, 1)
Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
@@ -281,20 +296,6 @@ STAGE PLANS:
Map-reduce partition columns: p_mfgr (type: string)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
value expressions: p_mfgr (type: string), p_size (type: int)
- Map 5
- Map Operator Tree:
- TableScan
- alias: part
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: UDFToDouble(p_size) is not null (type: boolean)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: UDFToDouble(p_size) (type: double)
- sort order: +
- Map-reduce partition columns: UDFToDouble(p_size) (type: double)
- Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
- value expressions: p_name (type: string), p_size (type: int)
Reducer 2
Reduce Operator Tree:
Extract
@@ -318,6 +319,8 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 0 Basic stats: PARTIAL Column stats: NONE
value expressions: _col0 (type: struct<count:bigint,sum:double,input:int>)
Reducer 3
+ Local Work:
+ Map Reduce Local Work
Reduce Operator Tree:
Group By Operator
aggregations: avg(VALUE._col0)
@@ -336,32 +339,52 @@ STAGE PLANS:
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: double)
- sort order: +
- Map-reduce partition columns: _col0 (type: double)
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reducer 4
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {VALUE._col1} {VALUE._col5}
- 1
- outputColumnNames: _col1, _col5
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: string), _col5 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {p_name} {p_size}
+ 1
+ keys:
+ 0 UDFToDouble(p_size) (type: double)
+ 1 _col0 (type: double)
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 4
+ Map Operator Tree:
+ TableScan
+ alias: part
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: UDFToDouble(p_size) is not null (type: boolean)
+ Statistics: Num rows: 13 Data size: 1573 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {p_name} {p_size}
+ 1
+ keys:
+ 0 UDFToDouble(p_size) (type: double)
+ 1 _col0 (type: double)
+ outputColumnNames: _col1, _col5
+ input vertices:
+ 1 Reducer 3
+ Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: string), _col5 (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 14 Data size: 1730 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Stage: Stage-0
Fetch Operator
@@ -410,33 +433,19 @@ from part b where b.p_size in
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
Edges:
- Reducer 4 <- Map 3 (PARTITION-LEVEL SORT, 1)
- Reducer 5 <- Reducer 4 (GROUP, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
+ Reducer 3 <- Map 2 (PARTITION-LEVEL SORT, 1)
+ Reducer 4 <- Reducer 3 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
- Map 1
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (p_size is not null and p_mfgr is not null) (type: boolean)
- Statistics: Num rows: 7 Data size: 847 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: p_size (type: int), p_mfgr (type: string)
- sort order: ++
- Map-reduce partition columns: p_size (type: int), p_mfgr (type: string)
- Statistics: Num rows: 7 Data size: 847 Basic stats: COMPLETE Column stats: NONE
- value expressions: p_name (type: string)
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: part
@@ -447,28 +456,7 @@ STAGE PLANS:
Map-reduce partition columns: p_mfgr (type: string)
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
value expressions: p_mfgr (type: string), p_size (type: int)
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {VALUE._col1} {KEY.reducesinkkey1} {KEY.reducesinkkey0}
- 1
- outputColumnNames: _col1, _col2, _col5
- Statistics: Num rows: 7 Data size: 931 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int)
- outputColumnNames: _col0, _col1, _col2
- Statistics: Num rows: 7 Data size: 931 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 7 Data size: 931 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
+ Reducer 3
Reduce Operator Tree:
Extract
Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
@@ -493,7 +481,9 @@ STAGE PLANS:
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 4 Data size: 484 Basic stats: COMPLETE Column stats: NONE
value expressions: _col1 (type: int)
- Reducer 5
+ Reducer 4
+ Local Work:
+ Map Reduce Local Work
Reduce Operator Tree:
Group By Operator
aggregations: min(VALUE._col0)
@@ -513,11 +503,52 @@ STAGE PLANS:
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: string)
- Statistics: Num rows: 2 Data size: 242 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {p_name} {p_mfgr} {p_size}
+ 1
+ keys:
+ 0 p_size (type: int), p_mfgr (type: string)
+ 1 _col0 (type: int), _col1 (type: string)
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (p_size is not null and p_mfgr is not null) (type: boolean)
+ Statistics: Num rows: 7 Data size: 847 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {p_name} {p_mfgr} {p_size}
+ 1
+ keys:
+ 0 p_size (type: int), p_mfgr (type: string)
+ 1 _col0 (type: int), _col1 (type: string)
+ outputColumnNames: _col1, _col2, _col5
+ input vertices:
+ 1 Reducer 4
+ Statistics: Num rows: 7 Data size: 931 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col2 (type: string), _col1 (type: string), _col5 (type: int)
+ outputColumnNames: _col0, _col1, _col2
+ Statistics: Num rows: 7 Data size: 931 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 7 Data size: 931 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Stage: Stage-0
Fetch Operator
@@ -570,31 +601,18 @@ where b.key in
)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
Edges:
- Reducer 4 <- Map 3 (GROUP, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 4 (PARTITION-LEVEL SORT, 1)
+ Reducer 3 <- Map 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
- Map 1
- Map Operator Tree:
- TableScan
- alias: b
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (key is not null and value is not null) (type: boolean)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: string), value (type: string)
- sort order: ++
- Map-reduce partition columns: key (type: string), value (type: string)
- Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
- Map 3
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -616,28 +634,9 @@ STAGE PLANS:
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
Statistics: Num rows: 83 Data size: 881 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {KEY.reducesinkkey1}
- 1
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: string), _col1 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 4
+ Reducer 3
+ Local Work:
+ Map Reduce Local Work
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
@@ -653,11 +652,52 @@ STAGE PLANS:
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 41 Data size: 435 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: b
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 125 Data size: 1328 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1
+ keys:
+ 0 key (type: string), value (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ input vertices:
+ 1 Reducer 3
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: string), _col1 (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 137 Data size: 1460 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Stage: Stage-0
Fetch Operator
@@ -733,16 +773,13 @@ where li.l_linenumber = 1 and
li.l_orderkey in (select l_orderkey from lineitem where l_shipmode = 'AIR')
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 3 <- Map 6 (PARTITION-LEVEL SORT, 1), Reducer 2 (PARTITION-LEVEL SORT, 1)
- Reducer 5 <- Map 4 (GROUP, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -753,94 +790,73 @@ STAGE PLANS:
Filter Operator
predicate: ((l_partkey is not null and l_orderkey is not null) and (l_linenumber = 1)) (type: boolean)
Statistics: Num rows: 12 Data size: 1439 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: l_partkey (type: int)
- sort order: +
- Map-reduce partition columns: l_partkey (type: int)
- Statistics: Num rows: 12 Data size: 1439 Basic stats: COMPLETE Column stats: NONE
- value expressions: l_orderkey (type: int), l_suppkey (type: int)
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col0}
+ 1 {l_orderkey} {l_suppkey}
+ keys:
+ 0 _col0 (type: int)
+ 1 l_partkey (type: int)
+ Local Work:
+ Map Reduce Local Work
Map 4
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((l_shipmode = 'AIR') and l_orderkey is not null) (type: boolean)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: l_partkey (type: int)
- outputColumnNames: l_partkey
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ expressions: l_orderkey (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: l_partkey (type: int)
+ keys: _col0 (type: int)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Map 6
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col0} {_col3}
+ 1
+ keys:
+ 0 _col1 (type: int)
+ 1 _col0 (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((l_shipmode = 'AIR') and l_orderkey is not null) (type: boolean)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ predicate: l_partkey is not null (type: boolean)
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: l_orderkey (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ expressions: l_partkey (type: int)
+ outputColumnNames: l_partkey
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: l_partkey (type: int)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0}
- 1 {VALUE._col0} {VALUE._col1}
- outputColumnNames: _col0, _col1, _col3
- Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: int)
- sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int), _col3 (type: int)
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reducer 3
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {VALUE._col0} {VALUE._col2}
- 1
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
+ Local Work:
+ Map Reduce Local Work
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -851,11 +867,43 @@ STAGE PLANS:
expressions: _col0 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {_col0}
+ 1 {l_orderkey} {l_suppkey}
+ keys:
+ 0 _col0 (type: int)
+ 1 l_partkey (type: int)
+ outputColumnNames: _col0, _col1, _col3
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {_col0} {_col3}
+ 1
+ keys:
+ 0 _col1 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col3
+ input vertices:
+ 1 Map 4
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col3 (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator