You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by xu...@apache.org on 2014/11/25 14:03:23 UTC
svn commit: r1641614 [43/43] - in /hive/branches/spark:
itests/src/test/resources/
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/ ql/sr...
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out Tue Nov 25 13:03:18 2014
@@ -23,16 +23,13 @@ where li.l_linenumber = 1 and
li.l_orderkey in (select l_orderkey from lineitem where l_shipmode = 'AIR')
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 3 <- Map 6 (PARTITION-LEVEL SORT, 1), Reducer 2 (PARTITION-LEVEL SORT, 1)
- Reducer 5 <- Map 4 (GROUP, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -43,94 +40,73 @@ STAGE PLANS:
Filter Operator
predicate: ((l_partkey is not null and l_orderkey is not null) and (l_linenumber = 1)) (type: boolean)
Statistics: Num rows: 12 Data size: 1439 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: l_partkey (type: int)
- sort order: +
- Map-reduce partition columns: l_partkey (type: int)
- Statistics: Num rows: 12 Data size: 1439 Basic stats: COMPLETE Column stats: NONE
- value expressions: l_orderkey (type: int), l_suppkey (type: int)
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col0}
+ 1 {l_orderkey} {l_suppkey}
+ keys:
+ 0 _col0 (type: int)
+ 1 l_partkey (type: int)
+ Local Work:
+ Map Reduce Local Work
Map 4
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: l_partkey is not null (type: boolean)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((l_shipmode = 'AIR') and l_orderkey is not null) (type: boolean)
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: l_partkey (type: int)
- outputColumnNames: l_partkey
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
+ expressions: l_orderkey (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: l_partkey (type: int)
+ keys: _col0 (type: int)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Map 6
+ Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col0} {_col3}
+ 1
+ keys:
+ 0 _col1 (type: int)
+ 1 _col0 (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((l_shipmode = 'AIR') and l_orderkey is not null) (type: boolean)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ predicate: l_partkey is not null (type: boolean)
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: l_orderkey (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ expressions: l_partkey (type: int)
+ outputColumnNames: l_partkey
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- keys: _col0 (type: int)
+ keys: l_partkey (type: int)
mode: hash
outputColumnNames: _col0
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0}
- 1 {VALUE._col0} {VALUE._col1}
- outputColumnNames: _col0, _col1, _col3
- Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: int)
- sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int), _col3 (type: int)
+ Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Reducer 3
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {VALUE._col0} {VALUE._col2}
- 1
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
+ Local Work:
+ Map Reduce Local Work
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -141,11 +117,43 @@ STAGE PLANS:
expressions: _col0 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {_col0}
+ 1 {l_orderkey} {l_suppkey}
+ keys:
+ 0 _col0 (type: int)
+ 1 l_partkey (type: int)
+ outputColumnNames: _col0, _col1, _col3
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {_col0} {_col3}
+ 1
+ keys:
+ 0 _col1 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col0, _col3
+ input vertices:
+ 1 Map 4
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col3 (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: vectorized
Stage: Stage-0
@@ -193,16 +201,13 @@ where li.l_linenumber = 1 and
li.l_orderkey in (select l_orderkey from lineitem where l_shipmode = 'AIR' and l_linenumber = li.l_linenumber)
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 3 <- Map 6 (PARTITION-LEVEL SORT, 1), Reducer 2 (PARTITION-LEVEL SORT, 1)
- Reducer 5 <- Map 4 (GROUP, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -213,18 +218,54 @@ STAGE PLANS:
Filter Operator
predicate: (((l_partkey is not null and l_orderkey is not null) and l_linenumber is not null) and (l_linenumber = 1)) (type: boolean)
Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: l_partkey (type: int)
- sort order: +
- Map-reduce partition columns: l_partkey (type: int)
- Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
- value expressions: l_orderkey (type: int), l_suppkey (type: int)
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col0}
+ 1 {l_orderkey} {l_suppkey}
+ keys:
+ 0 _col0 (type: int)
+ 1 l_partkey (type: int)
+ Local Work:
+ Map Reduce Local Work
Map 4
Map Operator Tree:
TableScan
alias: lineitem
Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
Filter Operator
+ predicate: ((((l_shipmode = 'AIR') and l_orderkey is not null) and l_linenumber is not null) and (l_linenumber = 1)) (type: boolean)
+ Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: l_orderkey (type: int), 1 (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: int), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col0} {_col3}
+ 1
+ keys:
+ 0 _col1 (type: int), 1 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
+ Map Operator Tree:
+ TableScan
+ alias: lineitem
+ Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
predicate: l_partkey is not null (type: boolean)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
Select Operator
@@ -241,66 +282,9 @@ STAGE PLANS:
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 50 Data size: 5999 Basic stats: COMPLETE Column stats: NONE
- Map 6
- Map Operator Tree:
- TableScan
- alias: lineitem
- Statistics: Num rows: 100 Data size: 11999 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: ((((l_shipmode = 'AIR') and l_orderkey is not null) and l_linenumber is not null) and (l_linenumber = 1)) (type: boolean)
- Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: l_orderkey (type: int), 1 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: int), _col1 (type: int)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int), _col1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col0 (type: int), _col1 (type: int)
- Statistics: Num rows: 6 Data size: 719 Basic stats: COMPLETE Column stats: NONE
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0}
- 1 {VALUE._col0} {VALUE._col1}
- outputColumnNames: _col0, _col1, _col3
- Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: int), 1 (type: int)
- sort order: ++
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: int), _col3 (type: int)
Reducer 3
- Reduce Operator Tree:
- Join Operator
- condition map:
- Left Semi Join 0 to 1
- condition expressions:
- 0 {VALUE._col0} {VALUE._col2}
- 1
- outputColumnNames: _col0, _col3
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col3 (type: int)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
- Reducer 5
+ Local Work:
+ Map Reduce Local Work
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: int)
@@ -311,11 +295,43 @@ STAGE PLANS:
expressions: _col0 (type: int)
outputColumnNames: _col0
Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: int)
- sort order: +
- Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 25 Data size: 2999 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {_col0}
+ 1 {l_orderkey} {l_suppkey}
+ keys:
+ 0 _col0 (type: int)
+ 1 l_partkey (type: int)
+ outputColumnNames: _col0, _col1, _col3
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 27 Data size: 3298 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Left Semi Join 0 to 1
+ condition expressions:
+ 0 {_col0} {_col3}
+ 1
+ keys:
+ 0 _col1 (type: int), 1 (type: int)
+ 1 _col0 (type: int), _col1 (type: int)
+ outputColumnNames: _col0, _col3
+ input vertices:
+ 1 Map 4
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col3 (type: int)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 29 Data size: 3627 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: vectorized
Stage: Stage-0
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_bucketmapjoin1.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_bucketmapjoin1.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_bucketmapjoin1.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_bucketmapjoin1.q.out Tue Nov 25 13:03:18 2014
@@ -101,14 +101,13 @@ POSTHOOK: query: explain
select /*+MAPJOIN(a)*/ * from vsmb_bucket_1 a join vsmb_bucket_2 b on a.key = b.key
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 3 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -119,14 +118,21 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: int)
- sort order: +
- Map-reduce partition columns: key (type: int)
- Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
- Execution mode: vectorized
- Map 3
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1 {value}
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -134,34 +140,33 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: int)
- sort order: +
- Map-reduce partition columns: key (type: int)
- Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1 {key} {value}
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0, _col1, _col5, _col6
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Execution mode: vectorized
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {VALUE._col0}
- 1 {KEY.reducesinkkey0} {VALUE._col0}
- outputColumnNames: _col0, _col1, _col5, _col6
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
@@ -190,14 +195,13 @@ POSTHOOK: query: explain
select /*+MAPJOIN(b)*/ * from vsmb_bucket_1 a join vsmb_bucket_RC b on a.key = b.key
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 3 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -208,13 +212,21 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 25 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: int)
- sort order: +
- Map-reduce partition columns: key (type: int)
- Statistics: Num rows: 1 Data size: 25 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
- Map 3
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1 {value}
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -222,34 +234,33 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: int)
- sort order: +
- Map-reduce partition columns: key (type: int)
- Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1 {key} {value}
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0, _col1, _col5, _col6
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Execution mode: vectorized
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {VALUE._col0}
- 1 {KEY.reducesinkkey0} {VALUE._col0}
- outputColumnNames: _col0, _col1, _col5, _col6
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
@@ -288,14 +299,13 @@ explain
select /*+MAPJOIN(b)*/ * from vsmb_bucket_1 a join vsmb_bucket_TXT b on a.key = b.key
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 3 (PARTITION-LEVEL SORT, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -306,13 +316,21 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 26 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: int)
- sort order: +
- Map-reduce partition columns: key (type: int)
- Statistics: Num rows: 1 Data size: 26 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
- Map 3
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {key} {value}
+ 1 {value}
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: a
@@ -320,34 +338,33 @@ STAGE PLANS:
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: key (type: int)
- sort order: +
- Map-reduce partition columns: key (type: int)
- Statistics: Num rows: 1 Data size: 104 Basic stats: COMPLETE Column stats: NONE
- value expressions: value (type: string)
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {key} {value}
+ 1 {key} {value}
+ keys:
+ 0 key (type: int)
+ 1 key (type: int)
+ outputColumnNames: _col0, _col1, _col5, _col6
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.TextInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ Local Work:
+ Map Reduce Local Work
Execution mode: vectorized
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {VALUE._col0}
- 1 {KEY.reducesinkkey0} {VALUE._col0}
- outputColumnNames: _col0, _col1, _col5, _col6
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), _col5 (type: int), _col6 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- Statistics: Num rows: 1 Data size: 114 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.TextInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out Tue Nov 25 13:03:18 2014
@@ -7,15 +7,13 @@ POSTHOOK: query: EXPLAIN SELECT COUNT(t1
JOIN alltypesorc t2 ON t1.cint = t2.cint
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -26,13 +24,23 @@ STAGE PLANS:
Filter Operator
predicate: cint is not null (type: boolean)
Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: cint (type: int)
- sort order: +
- Map-reduce partition columns: cint (type: int)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Execution mode: vectorized
- Map 4
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {cint}
+ 1
+ keys:
+ 0 cint (type: int)
+ 1 cint (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: t1
@@ -40,35 +48,35 @@ STAGE PLANS:
Filter Operator
predicate: cint is not null (type: boolean)
Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: cint (type: int)
- sort order: +
- Map-reduce partition columns: cint (type: int)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {cint}
+ 1 {cint}
+ keys:
+ 0 cint (type: int)
+ 1 cint (type: int)
+ outputColumnNames: _col2, _col17
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col2 (type: int), _col17 (type: int)
+ outputColumnNames: _col2, _col17
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count(_col2), max(_col17), min(_col2), avg((_col2 + _col17))
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: int), _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>)
+ Local Work:
+ Map Reduce Local Work
Execution mode: vectorized
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0}
- 1 {KEY.reducesinkkey0}
- outputColumnNames: _col2, _col17
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: int), _col17 (type: int)
- outputColumnNames: _col2, _col17
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(_col2), max(_col17), min(_col2), avg((_col2 + _col17))
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: int), _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>)
Reducer 3
Reduce Operator Tree:
Group By Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out Tue Nov 25 13:03:18 2014
@@ -3,33 +3,17 @@ PREHOOK: type: QUERY
POSTHOOK: query: explain select sum(t1.td) from (select v1.csmallint as tsi, v1.cdouble as td from alltypesorc v1, alltypesorc v2 where v1.ctinyint=v2.ctinyint) t1 join alltypesorc v3 on t1.tsi=v3.csmallint
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-4 is a root stage
+ Stage-3 depends on stages: Stage-4
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-4
Spark
- Edges:
- Reducer 5 <- Map 4 (PARTITION-LEVEL SORT, 1), Map 6 (PARTITION-LEVEL SORT, 1)
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Reducer 5 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
- Map 1
- Map Operator Tree:
- TableScan
- alias: v3
- Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: csmallint is not null (type: boolean)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: csmallint (type: smallint)
- sort order: +
- Map-reduce partition columns: csmallint (type: smallint)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Execution mode: vectorized
- Map 4
+ Map 3
Map Operator Tree:
TableScan
alias: v1
@@ -37,14 +21,21 @@ STAGE PLANS:
Filter Operator
predicate: (ctinyint is not null and csmallint is not null) (type: boolean)
Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: ctinyint (type: tinyint)
- sort order: +
- Map-reduce partition columns: ctinyint (type: tinyint)
- Statistics: Num rows: 3072 Data size: 94309 Basic stats: COMPLETE Column stats: NONE
- value expressions: csmallint (type: smallint), cdouble (type: double)
- Execution mode: vectorized
- Map 6
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {csmallint} {cdouble}
+ 1 {ctinyint}
+ keys:
+ 0 ctinyint (type: tinyint)
+ 1 ctinyint (type: tinyint)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-3
+ Spark
+#### A masked pattern was here ####
+ Vertices:
+ Map 4
Map Operator Tree:
TableScan
alias: v2
@@ -52,37 +43,81 @@ STAGE PLANS:
Filter Operator
predicate: ctinyint is not null (type: boolean)
Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: ctinyint (type: tinyint)
- sort order: +
- Map-reduce partition columns: ctinyint (type: tinyint)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {ctinyint} {csmallint} {cdouble}
+ 1 {ctinyint}
+ keys:
+ 0 ctinyint (type: tinyint)
+ 1 ctinyint (type: tinyint)
+ outputColumnNames: _col0, _col1, _col5, _col15
+ input vertices:
+ 0 Map 3
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (_col0 = _col15) (type: boolean)
+ Statistics: Num rows: 3379 Data size: 103739 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: smallint), _col5 (type: double)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 3379 Data size: 103739 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {_col1}
+ 1
+ keys:
+ 0 _col0 (type: smallint)
+ 1 csmallint (type: smallint)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 2 <- Map 1 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 1
+ Map Operator Tree:
+ TableScan
+ alias: v3
+ Statistics: Num rows: 12288 Data size: 377237 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: csmallint is not null (type: boolean)
+ Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {_col1}
+ 1
+ keys:
+ 0 _col0 (type: smallint)
+ 1 csmallint (type: smallint)
+ outputColumnNames: _col1
+ input vertices:
+ 0 Map 4
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col1 (type: double)
+ outputColumnNames: _col1
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: sum(_col1)
+ mode: hash
+ outputColumnNames: _col0
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: double)
+ Local Work:
+ Map Reduce Local Work
Execution mode: vectorized
Reducer 2
Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {VALUE._col0}
- 1
- outputColumnNames: _col1
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: double)
- outputColumnNames: _col1
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col1)
- mode: hash
- outputColumnNames: _col0
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: double)
- Reducer 3
- Reduce Operator Tree:
Group By Operator
aggregations: sum(VALUE._col0)
mode: mergepartial
@@ -100,29 +135,6 @@ STAGE PLANS:
output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Execution mode: vectorized
- Reducer 5
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0} {VALUE._col0} {VALUE._col4}
- 1 {KEY.reducesinkkey0}
- outputColumnNames: _col0, _col1, _col5, _col15
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (_col0 = _col15) (type: boolean)
- Statistics: Num rows: 3379 Data size: 103739 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: smallint), _col5 (type: double)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 3379 Data size: 103739 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: smallint)
- sort order: +
- Map-reduce partition columns: _col0 (type: smallint)
- Statistics: Num rows: 3379 Data size: 103739 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: double)
Stage: Stage-0
Fetch Operator
@@ -138,4 +150,4 @@ POSTHOOK: query: select sum(t1.td) from
POSTHOOK: type: QUERY
POSTHOOK: Input: default@alltypesorc
#### A masked pattern was here ####
-6.065190932485957E11
+6.06519093248863E11
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
Files hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out (original) and hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out Tue Nov 25 13:03:18 2014 differ
Modified: hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_shufflejoin.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_shufflejoin.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_shufflejoin.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/spark/vectorized_shufflejoin.q.out Tue Nov 25 13:03:18 2014
@@ -7,15 +7,13 @@ POSTHOOK: query: EXPLAIN SELECT COUNT(t1
JOIN alltypesorc t2 ON t1.cint = t2.cint
POSTHOOK: type: QUERY
STAGE DEPENDENCIES:
- Stage-1 is a root stage
+ Stage-3 is a root stage
+ Stage-1 depends on stages: Stage-3
Stage-0 depends on stages: Stage-1
STAGE PLANS:
- Stage: Stage-1
+ Stage: Stage-3
Spark
- Edges:
- Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 1), Map 4 (PARTITION-LEVEL SORT, 1)
- Reducer 3 <- Reducer 2 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
@@ -26,13 +24,23 @@ STAGE PLANS:
Filter Operator
predicate: cint is not null (type: boolean)
Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: cint (type: int)
- sort order: +
- Map-reduce partition columns: cint (type: int)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Execution mode: vectorized
- Map 4
+ Spark HashTable Sink Operator
+ condition expressions:
+ 0 {cint}
+ 1
+ keys:
+ 0 cint (type: int)
+ 1 cint (type: int)
+ Local Work:
+ Map Reduce Local Work
+
+ Stage: Stage-1
+ Spark
+ Edges:
+ Reducer 3 <- Map 2 (GROUP, 1)
+#### A masked pattern was here ####
+ Vertices:
+ Map 2
Map Operator Tree:
TableScan
alias: t1
@@ -40,35 +48,35 @@ STAGE PLANS:
Filter Operator
predicate: cint is not null (type: boolean)
Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: cint (type: int)
- sort order: +
- Map-reduce partition columns: cint (type: int)
- Statistics: Num rows: 6144 Data size: 188618 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ condition expressions:
+ 0 {cint}
+ 1 {cint}
+ keys:
+ 0 cint (type: int)
+ 1 cint (type: int)
+ outputColumnNames: _col2, _col17
+ input vertices:
+ 1 Map 1
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: _col2 (type: int), _col17 (type: int)
+ outputColumnNames: _col2, _col17
+ Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ aggregations: count(_col2), max(_col17), min(_col2), avg((_col2 + _col17))
+ mode: hash
+ outputColumnNames: _col0, _col1, _col2, _col3
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ sort order:
+ Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col0 (type: bigint), _col1 (type: int), _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>)
+ Local Work:
+ Map Reduce Local Work
Execution mode: vectorized
- Reducer 2
- Reduce Operator Tree:
- Join Operator
- condition map:
- Inner Join 0 to 1
- condition expressions:
- 0 {KEY.reducesinkkey0}
- 1 {KEY.reducesinkkey0}
- outputColumnNames: _col2, _col17
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col2 (type: int), _col17 (type: int)
- outputColumnNames: _col2, _col17
- Statistics: Num rows: 6758 Data size: 207479 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: count(_col2), max(_col17), min(_col2), avg((_col2 + _col17))
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- sort order:
- Statistics: Num rows: 1 Data size: 16 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col0 (type: bigint), _col1 (type: int), _col2 (type: int), _col3 (type: struct<count:bigint,sum:double,input:int>)
Reducer 3
Reduce Operator Tree:
Group By Operator
Modified: hive/branches/spark/ql/src/test/results/clientpositive/subquery_exists.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/subquery_exists.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/subquery_exists.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/subquery_exists.q.out Tue Nov 25 13:03:18 2014
@@ -1,4 +1,6 @@
-PREHOOK: query: -- no agg, corr
+PREHOOK: query: -- SORT_QUERY_RESULTS
+
+-- no agg, corr
explain
select *
from src b
@@ -8,7 +10,9 @@ where exists
where b.value = a.value and a.key = b.key and a.value > 'val_9'
)
PREHOOK: type: QUERY
-POSTHOOK: query: -- no agg, corr
+POSTHOOK: query: -- SORT_QUERY_RESULTS
+
+-- no agg, corr
explain
select *
from src b
Modified: hive/branches/spark/ql/src/test/results/clientpositive/subquery_multiinsert.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/subquery_multiinsert.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/subquery_multiinsert.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/subquery_multiinsert.q.out Tue Nov 25 13:03:18 2014
@@ -1,11 +1,15 @@
-PREHOOK: query: CREATE TABLE src_4(
+PREHOOK: query: -- SORT_QUERY_RESULTS
+
+CREATE TABLE src_4(
key STRING,
value STRING
)
PREHOOK: type: CREATETABLE
PREHOOK: Output: database:default
PREHOOK: Output: default@src_4
-POSTHOOK: query: CREATE TABLE src_4(
+POSTHOOK: query: -- SORT_QUERY_RESULTS
+
+CREATE TABLE src_4(
key STRING,
value STRING
)
@@ -881,17 +885,17 @@ POSTHOOK: query: select * from src_4
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src_4
#### A masked pattern was here ####
-98 val_98
+90 val_90
+90 val_90
+90 val_90
92 val_92
-96 val_96
95 val_95
-98 val_98
-90 val_90
95 val_95
-90 val_90
+96 val_96
97 val_97
-90 val_90
97 val_97
+98 val_98
+98 val_98
PREHOOK: query: select * from src_5
PREHOOK: type: QUERY
PREHOOK: Input: default@src_5
Modified: hive/branches/spark/ql/src/test/results/clientpositive/tez/subquery_exists.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/tez/subquery_exists.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
--- hive/branches/spark/ql/src/test/results/clientpositive/tez/subquery_exists.q.out (original)
+++ hive/branches/spark/ql/src/test/results/clientpositive/tez/subquery_exists.q.out Tue Nov 25 13:03:18 2014
@@ -1,4 +1,6 @@
-PREHOOK: query: -- no agg, corr
+PREHOOK: query: -- SORT_QUERY_RESULTS
+
+-- no agg, corr
explain
select *
from src b
@@ -8,7 +10,9 @@ where exists
where b.value = a.value and a.key = b.key and a.value > 'val_9'
)
PREHOOK: type: QUERY
-POSTHOOK: query: -- no agg, corr
+POSTHOOK: query: -- SORT_QUERY_RESULTS
+
+-- no agg, corr
explain
select *
from src b
Modified: hive/branches/spark/ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
Files hive/branches/spark/ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out (original) and hive/branches/spark/ql/src/test/results/clientpositive/tez/vectorized_ptf.q.out Tue Nov 25 13:03:18 2014 differ
Modified: hive/branches/spark/ql/src/test/results/clientpositive/vectorized_ptf.q.out
URL: http://svn.apache.org/viewvc/hive/branches/spark/ql/src/test/results/clientpositive/vectorized_ptf.q.out?rev=1641614&r1=1641613&r2=1641614&view=diff
==============================================================================
Files hive/branches/spark/ql/src/test/results/clientpositive/vectorized_ptf.q.out (original) and hive/branches/spark/ql/src/test/results/clientpositive/vectorized_ptf.q.out Tue Nov 25 13:03:18 2014 differ