You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by jc...@apache.org on 2018/11/12 18:04:50 UTC
[30/51] [partial] hive git commit: HIVE-20850: Push case conditional
from projections to dimension tables if possible (Zoltan Haindrich via Jesus
Camacho Rodriguez)
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/masking_4.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/masking_4.q.out b/ql/src/test/results/clientpositive/masking_4.q.out
index 60cbd0f..54861b0 100644
--- a/ql/src/test/results/clientpositive/masking_4.q.out
+++ b/ql/src/test/results/clientpositive/masking_4.q.out
@@ -210,12 +210,16 @@ STAGE PLANS:
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Select Operator
+ expressions: _col0 (type: int), UDFToDouble(_col0) (type: double)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-2
Map Reduce
@@ -228,20 +232,20 @@ STAGE PLANS:
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: key (type: string)
- outputColumnNames: _col0
+ expressions: key (type: string), UDFToDouble(key) (type: double)
+ outputColumnNames: _col0, _col1
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: UDFToDouble(_col0) (type: double)
+ key expressions: _col1 (type: double)
sort order: +
- Map-reduce partition columns: UDFToDouble(_col0) (type: double)
+ Map-reduce partition columns: _col1 (type: double)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: string)
TableScan
Reduce Output Operator
- key expressions: UDFToDouble(_col0) (type: double)
+ key expressions: _col1 (type: double)
sort order: +
- Map-reduce partition columns: UDFToDouble(_col0) (type: double)
+ Map-reduce partition columns: _col1 (type: double)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: int)
Reduce Operator Tree:
@@ -249,21 +253,25 @@ STAGE PLANS:
condition map:
Inner Join 0 to 1
keys:
- 0 UDFToDouble(_col0) (type: double)
- 1 UDFToDouble(_col0) (type: double)
- outputColumnNames: _col0, _col1
+ 0 _col1 (type: double)
+ 1 _col1 (type: double)
+ outputColumnNames: _col0, _col2
Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: _col0 (type: string), _col1 (type: int)
- mode: hash
+ Select Operator
+ expressions: _col0 (type: string), _col2 (type: int)
outputColumnNames: _col0, _col1
Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: int)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
+ File Output Operator
+ compressed: false
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
Stage: Stage-1
Map Reduce
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/mergejoin.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/mergejoin.q.out b/ql/src/test/results/clientpositive/mergejoin.q.out
index b20ea45..2108434 100644
--- a/ql/src/test/results/clientpositive/mergejoin.q.out
+++ b/ql/src/test/results/clientpositive/mergejoin.q.out
@@ -3399,8 +3399,8 @@ NULL NULL NULL 98 val_98 2008-04-08 98 val_98 2008-04-08
NULL NULL NULL 98 val_98 2008-04-08 98 val_98 2008-04-08
NULL NULL NULL 98 val_98 2008-04-08 98 val_98 2008-04-08
NULL NULL NULL 98 val_98 2008-04-08 98 val_98 2008-04-08
-Warning: Shuffle Join JOIN[13][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
-Warning: Shuffle Join JOIN[10][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[14][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-2:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[8][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
PREHOOK: query: select * from
(select * from tab_n9 where tab_n9.key = 0)a
join
@@ -3927,8 +3927,8 @@ NULL NULL NULL NULL NULL NULL 97 val_97 2008-04-08
NULL NULL NULL NULL NULL NULL 97 val_97 2008-04-08
NULL NULL NULL NULL NULL NULL 98 val_98 2008-04-08
NULL NULL NULL NULL NULL NULL 98 val_98 2008-04-08
-Warning: Shuffle Join JOIN[13][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
-Warning: Shuffle Join JOIN[10][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[14][tables = [$hdt$_0, $hdt$_1]] in Stage 'Stage-1:MAPRED' is a cross product
+Warning: Shuffle Join JOIN[10][tables = [$hdt$_1, $hdt$_2]] in Stage 'Stage-2:MAPRED' is a cross product
PREHOOK: query: select * from
(select * from tab_n9 where tab_n9.key = 0)a
join
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/mergejoins.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/mergejoins.q.out b/ql/src/test/results/clientpositive/mergejoins.q.out
index dcae432..1841e63 100644
--- a/ql/src/test/results/clientpositive/mergejoins.q.out
+++ b/ql/src/test/results/clientpositive/mergejoins.q.out
@@ -218,24 +218,24 @@ STAGE PLANS:
Map Reduce
Map Operator Tree:
TableScan
- alias: a_n0
+ alias: b_n0
filterExpr: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
predicate: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: key (type: string), value (type: string)
- outputColumnNames: _col0, _col1
+ expressions: key (type: string), value (type: string), (UDFToDouble(key) < 10.0D) (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ value expressions: _col1 (type: string), _col2 (type: boolean)
TableScan
- alias: b_n0
+ alias: a_n0
filterExpr: key is not null (type: boolean)
Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Filter Operator
@@ -272,24 +272,28 @@ STAGE PLANS:
Join Operator
condition map:
Inner Join 0 to 1
- Left Outer Join 1 to 2
+ Left Outer Join 0 to 2
filter predicates:
- 0
- 1 {(UDFToDouble(KEY.reducesinkkey0) < 10.0D)}
+ 0 {VALUE._col1}
+ 1
2
keys:
0 _col0 (type: string)
1 _col0 (type: string)
2 _col0 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
+ outputColumnNames: _col0, _col1, _col3, _col4, _col5, _col6
Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
+ Select Operator
+ expressions: _col3 (type: string), _col4 (type: string), _col0 (type: string), _col1 (type: string), _col5 (type: string), _col6 (type: string)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
+ File Output Operator
+ compressed: false
+ Statistics: Num rows: 1100 Data size: 11686 Basic stats: COMPLETE Column stats: NONE
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/multi_insert_with_join2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/multi_insert_with_join2.q.out b/ql/src/test/results/clientpositive/multi_insert_with_join2.q.out
index 50d407f..761e7c0 100644
--- a/ql/src/test/results/clientpositive/multi_insert_with_join2.q.out
+++ b/ql/src/test/results/clientpositive/multi_insert_with_join2.q.out
@@ -92,12 +92,12 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: val (type: string)
- outputColumnNames: _col1
+ outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ value expressions: _col0 (type: string)
TableScan
alias: b
filterExpr: ((id = 'Id_1') and (val = 'val_103')) (type: boolean)
@@ -117,10 +117,10 @@ STAGE PLANS:
keys:
0
1
- outputColumnNames: _col1
+ outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 25 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: 'Id_1' (type: string), _col1 (type: string), 'Id_1' (type: string), 'val_103' (type: string)
+ expressions: 'Id_1' (type: string), _col0 (type: string), 'Id_1' (type: string), 'val_103' (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 1 Data size: 25 Basic stats: COMPLETE Column stats: NONE
File Output Operator
@@ -226,12 +226,12 @@ STAGE PLANS:
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: val (type: string)
- outputColumnNames: _col1
+ outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
sort order:
Statistics: Num rows: 1 Data size: 12 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ value expressions: _col0 (type: string)
TableScan
alias: b
filterExpr: ((val = 'val_104') and (id = 'Id_2')) (type: boolean)
@@ -251,10 +251,10 @@ STAGE PLANS:
keys:
0
1
- outputColumnNames: _col1
+ outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 25 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: 'Id_2' (type: string), _col1 (type: string), 'Id_2' (type: string), 'val_104' (type: string)
+ expressions: 'Id_2' (type: string), _col0 (type: string), 'Id_2' (type: string), 'val_104' (type: string)
outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 1 Data size: 25 Basic stats: COMPLETE Column stats: NONE
File Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/outer_join_ppr.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/outer_join_ppr.q.out b/ql/src/test/results/clientpositive/outer_join_ppr.q.out
index ea85034..fc0ea34 100644
--- a/ql/src/test/results/clientpositive/outer_join_ppr.q.out
+++ b/ql/src/test/results/clientpositive/outer_join_ppr.q.out
@@ -26,11 +26,11 @@ POSTHOOK: Input: default@srcpart
POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
#### A masked pattern was here ####
-OPTIMIZED SQL: SELECT `t0`.`key`, `t0`.`value`, `t2`.`key` AS `key1`, `t2`.`value` AS `value1`
+OPTIMIZED SQL: SELECT *
FROM (SELECT `key`, `value`
FROM `default`.`src`
WHERE `key` < 20 AND `key` > 15) AS `t0`
-INNER JOIN (SELECT `key`, `value`, CAST('2008-04-08' AS STRING) AS `ds`
+INNER JOIN (SELECT `key`, `value`
FROM `default`.`srcpart`
WHERE `key` > 15 AND `ds` = '2008-04-08' AND `key` < 20) AS `t2` ON `t0`.`key` = `t2`.`key`
STAGE DEPENDENCIES:
@@ -343,11 +343,11 @@ POSTHOOK: Input: default@srcpart
POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=11
POSTHOOK: Input: default@srcpart@ds=2008-04-08/hr=12
#### A masked pattern was here ####
-OPTIMIZED SQL: SELECT `t0`.`key`, `t0`.`value`, `t2`.`key` AS `key1`, `t2`.`value` AS `value1`
+OPTIMIZED SQL: SELECT *
FROM (SELECT `key`, `value`
FROM `default`.`src`
WHERE `key` < 20 AND `key` > 15) AS `t0`
-INNER JOIN (SELECT `key`, `value`, CAST('2008-04-08' AS STRING) AS `ds`
+INNER JOIN (SELECT `key`, `value`
FROM `default`.`srcpart`
WHERE `key` > 15 AND `ds` = '2008-04-08' AND `key` < 20) AS `t2` ON `t0`.`key` = `t2`.`key`
STAGE DEPENDENCIES:
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/pcr.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/pcr.q.out b/ql/src/test/results/clientpositive/pcr.q.out
index 95f560f..d71ef67 100644
--- a/ql/src/test/results/clientpositive/pcr.q.out
+++ b/ql/src/test/results/clientpositive/pcr.q.out
@@ -2430,15 +2430,15 @@ POSTHOOK: type: QUERY
POSTHOOK: Input: default@pcr_t1
POSTHOOK: Input: default@pcr_t1@ds=2000-04-08
#### A masked pattern was here ####
-OPTIMIZED SQL: SELECT `t4`.`key`, `t4`.`value`, CAST('2000-04-08' AS STRING) AS `ds`, `t4`.`key1`, `t4`.`value1`, CAST('2000-04-08' AS STRING) AS `ds1`
-FROM (SELECT `t0`.`key`, `t0`.`value`, `t2`.`key` AS `key1`, `t2`.`value` AS `value1`
-FROM (SELECT `key`, `value`, CAST('2000-04-08' AS STRING) AS `ds`
+OPTIMIZED SQL: SELECT `t3`.`key`, `t3`.`value`, CAST('2000-04-08' AS STRING) AS `ds`, `t3`.`key0` AS `key1`, `t3`.`value0` AS `value1`, CAST('2000-04-08' AS STRING) AS `ds1`
+FROM (SELECT *
+FROM (SELECT `key`, `value`
FROM `default`.`pcr_t1`
WHERE `ds` = '2000-04-08' AND `key` IS NOT NULL) AS `t0`
-INNER JOIN (SELECT `key`, `value`, CAST('2000-04-08' AS STRING) AS `ds`
+INNER JOIN (SELECT `key`, `value`
FROM `default`.`pcr_t1`
WHERE `ds` = '2000-04-08' AND `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
-ORDER BY `t0`.`key`) AS `t4`
+ORDER BY `t0`.`key`) AS `t3`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
@@ -2553,30 +2553,26 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col0, _col1, _col3, _col4
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 22 Data size: 176 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), _col3 (type: int), _col4 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 22 Data size: 176 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- GlobalTableId: 0
+ File Output Operator
+ compressed: false
+ GlobalTableId: 0
#### A masked pattern was here ####
- NumFilesPerFileSink: 1
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_col2,_col3
- columns.types int,string,int,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- TotalFiles: 1
- GatherStats: false
- MultiFileSpray: false
+ NumFilesPerFileSink: 1
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3
+ columns.types int,string,int,string
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ TotalFiles: 1
+ GatherStats: false
+ MultiFileSpray: false
Stage: Stage-2
Map Reduce
@@ -2714,15 +2710,15 @@ POSTHOOK: Input: default@pcr_t1
POSTHOOK: Input: default@pcr_t1@ds=2000-04-08
POSTHOOK: Input: default@pcr_t1@ds=2000-04-09
#### A masked pattern was here ####
-OPTIMIZED SQL: SELECT `t4`.`key`, `t4`.`value`, CAST('2000-04-08' AS STRING) AS `ds`, `t4`.`key1`, `t4`.`value1`, CAST('2000-04-09' AS STRING) AS `ds1`
-FROM (SELECT `t0`.`key`, `t0`.`value`, `t2`.`key` AS `key1`, `t2`.`value` AS `value1`
-FROM (SELECT `key`, `value`, CAST('2000-04-08' AS STRING) AS `ds`
+OPTIMIZED SQL: SELECT `t3`.`key`, `t3`.`value`, CAST('2000-04-08' AS STRING) AS `ds`, `t3`.`key0` AS `key1`, `t3`.`value0` AS `value1`, CAST('2000-04-09' AS STRING) AS `ds1`
+FROM (SELECT *
+FROM (SELECT `key`, `value`
FROM `default`.`pcr_t1`
WHERE `ds` = '2000-04-08' AND `key` IS NOT NULL) AS `t0`
-INNER JOIN (SELECT `key`, `value`, CAST('2000-04-09' AS STRING) AS `ds`
+INNER JOIN (SELECT `key`, `value`
FROM `default`.`pcr_t1`
WHERE `ds` = '2000-04-09' AND `key` IS NOT NULL) AS `t2` ON `t0`.`key` = `t2`.`key`
-ORDER BY `t0`.`key`) AS `t4`
+ORDER BY `t0`.`key`) AS `t3`
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
@@ -2886,30 +2882,26 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col0, _col1, _col3, _col4
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 22 Data size: 176 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col0 (type: int), _col1 (type: string), _col3 (type: int), _col4 (type: string)
- outputColumnNames: _col0, _col1, _col2, _col3
- Statistics: Num rows: 22 Data size: 176 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- GlobalTableId: 0
+ File Output Operator
+ compressed: false
+ GlobalTableId: 0
#### A masked pattern was here ####
- NumFilesPerFileSink: 1
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- properties:
- column.name.delimiter ,
- columns _col0,_col1,_col2,_col3
- columns.types int,string,int,string
- escape.delim \
- serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
- TotalFiles: 1
- GatherStats: false
- MultiFileSpray: false
+ NumFilesPerFileSink: 1
+ table:
+ input format: org.apache.hadoop.mapred.SequenceFileInputFormat
+ output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
+ properties:
+ column.name.delimiter ,
+ columns _col0,_col1,_col2,_col3
+ columns.types int,string,int,string
+ escape.delim \
+ serialization.lib org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
+ TotalFiles: 1
+ GatherStats: false
+ MultiFileSpray: false
Stage: Stage-2
Map Reduce
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/perf/spark/query1.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query1.q.out b/ql/src/test/results/clientpositive/perf/spark/query1.q.out
index e0fd19e..75da726 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query1.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query1.q.out
@@ -309,14 +309,14 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col6
+ outputColumnNames: _col1, _col2, _col5
Statistics: Num rows: 88000001 Data size: 75681779077 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 88000001 Data size: 75681779077 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: decimal(17,2)), _col6 (type: string)
+ value expressions: _col2 (type: decimal(17,2)), _col5 (type: string)
Reducer 5
Reduce Operator Tree:
Join Operator
@@ -325,13 +325,13 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col1 (type: int)
- outputColumnNames: _col2, _col6, _col7
+ outputColumnNames: _col2, _col5, _col6
Statistics: Num rows: 96800003 Data size: 83249958789 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (_col2 > _col7) (type: boolean)
+ predicate: (_col2 > _col6) (type: boolean)
Statistics: Num rows: 32266667 Data size: 27749985689 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col6 (type: string)
+ expressions: _col5 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 32266667 Data size: 27749985689 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/perf/spark/query11.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query11.q.out b/ql/src/test/results/clientpositive/perf/spark/query11.q.out
index 823ad71..28ce6d0 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query11.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query11.q.out
@@ -190,15 +190,15 @@ STAGE PLANS:
predicate: (ws_bill_customer_sk is not null and ws_sold_date_sk is not null) (type: boolean)
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ws_sold_date_sk (type: int), ws_bill_customer_sk (type: int), ws_ext_discount_amt (type: decimal(7,2)), ws_ext_list_price (type: decimal(7,2))
- outputColumnNames: _col0, _col1, _col2, _col3
+ expressions: ws_sold_date_sk (type: int), ws_bill_customer_sk (type: int), (ws_ext_list_price - ws_ext_discount_amt) (type: decimal(8,2))
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int), _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col1 (type: int), _col2 (type: decimal(8,2))
Execution mode: vectorized
Map 13
Map Operator Tree:
@@ -249,15 +249,15 @@ STAGE PLANS:
predicate: (ws_bill_customer_sk is not null and ws_sold_date_sk is not null) (type: boolean)
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ws_sold_date_sk (type: int), ws_bill_customer_sk (type: int), ws_ext_discount_amt (type: decimal(7,2)), ws_ext_list_price (type: decimal(7,2))
- outputColumnNames: _col0, _col1, _col2, _col3
+ expressions: ws_sold_date_sk (type: int), ws_bill_customer_sk (type: int), (ws_ext_list_price - ws_ext_discount_amt) (type: decimal(8,2))
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 144002668 Data size: 19580198212 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int), _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col1 (type: int), _col2 (type: decimal(8,2))
Execution mode: vectorized
Map 19
Map Operator Tree:
@@ -308,15 +308,15 @@ STAGE PLANS:
predicate: (ss_customer_sk is not null and ss_sold_date_sk is not null) (type: boolean)
Statistics: Num rows: 575995635 Data size: 50814502088 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_sold_date_sk (type: int), ss_customer_sk (type: int), ss_ext_discount_amt (type: decimal(7,2)), ss_ext_list_price (type: decimal(7,2))
- outputColumnNames: _col0, _col1, _col2, _col3
+ expressions: ss_sold_date_sk (type: int), ss_customer_sk (type: int), (ss_ext_list_price - ss_ext_discount_amt) (type: decimal(8,2))
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 575995635 Data size: 50814502088 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 575995635 Data size: 50814502088 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int), _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col1 (type: int), _col2 (type: decimal(8,2))
Execution mode: vectorized
Map 25
Map Operator Tree:
@@ -406,15 +406,15 @@ STAGE PLANS:
predicate: (ss_customer_sk is not null and ss_sold_date_sk is not null) (type: boolean)
Statistics: Num rows: 575995635 Data size: 50814502088 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_sold_date_sk (type: int), ss_customer_sk (type: int), ss_ext_discount_amt (type: decimal(7,2)), ss_ext_list_price (type: decimal(7,2))
- outputColumnNames: _col0, _col1, _col2, _col3
+ expressions: ss_sold_date_sk (type: int), ss_customer_sk (type: int), (ss_ext_list_price - ss_ext_discount_amt) (type: decimal(8,2))
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 575995635 Data size: 50814502088 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 575995635 Data size: 50814502088 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int), _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col1 (type: int), _col2 (type: decimal(8,2))
Execution mode: vectorized
Reducer 10
Reduce Operator Tree:
@@ -424,14 +424,14 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
+ outputColumnNames: _col1, _col2
Statistics: Num rows: 633595212 Data size: 55895953508 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 633595212 Data size: 55895953508 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col2 (type: decimal(8,2))
Reducer 11
Reduce Operator Tree:
Join Operator
@@ -440,24 +440,20 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col3, _col7, _col8, _col9, _col10, _col11, _col12, _col13
+ outputColumnNames: _col2, _col5, _col6, _col7, _col8, _col9, _col10, _col11
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string), _col12 (type: string), _col13 (type: string), (_col3 - _col2) (type: decimal(8,2))
+ Group By Operator
+ aggregations: sum(_col2)
+ keys: _col5 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col7)
- keys: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
+ sort order: +++++++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- sort order: +++++++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col7 (type: decimal(18,2))
+ value expressions: _col7 (type: decimal(18,2))
Reducer 12
Execution mode: vectorized
Reduce Operator Tree:
@@ -492,14 +488,14 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
+ outputColumnNames: _col1, _col2
Statistics: Num rows: 158402938 Data size: 21538218500 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 158402938 Data size: 21538218500 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col2 (type: decimal(8,2))
Reducer 17
Reduce Operator Tree:
Join Operator
@@ -508,24 +504,20 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col3, _col7, _col8, _col9, _col10, _col11, _col12, _col13
+ outputColumnNames: _col2, _col5, _col6, _col7, _col8, _col9, _col10, _col11
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string), _col12 (type: string), _col13 (type: string), (_col3 - _col2) (type: decimal(8,2))
+ Group By Operator
+ aggregations: sum(_col2)
+ keys: _col5 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col7)
- keys: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
+ sort order: +++++++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- sort order: +++++++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col7 (type: decimal(18,2))
+ value expressions: _col7 (type: decimal(18,2))
Reducer 18
Execution mode: vectorized
Reduce Operator Tree:
@@ -543,15 +535,15 @@ STAGE PLANS:
predicate: (_col7 > 0) (type: boolean)
Statistics: Num rows: 29040539 Data size: 3948673454 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col0 (type: string), _col7 (type: decimal(18,2))
- outputColumnNames: _col0, _col1
+ expressions: _col0 (type: string), _col7 (type: decimal(18,2)), _col7 is not null (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 29040539 Data size: 3948673454 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 29040539 Data size: 3948673454 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: decimal(18,2))
+ value expressions: _col1 (type: decimal(18,2)), _col2 (type: boolean)
Reducer 2
Reduce Operator Tree:
Join Operator
@@ -560,14 +552,14 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
+ outputColumnNames: _col1, _col2
Statistics: Num rows: 158402938 Data size: 21538218500 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 158402938 Data size: 21538218500 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col2 (type: decimal(8,2))
Reducer 22
Reduce Operator Tree:
Join Operator
@@ -576,14 +568,14 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3
+ outputColumnNames: _col1, _col2
Statistics: Num rows: 633595212 Data size: 55895953508 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 633595212 Data size: 55895953508 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: decimal(7,2)), _col3 (type: decimal(7,2))
+ value expressions: _col2 (type: decimal(8,2))
Reducer 23
Reduce Operator Tree:
Join Operator
@@ -592,24 +584,20 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col3, _col7, _col8, _col9, _col10, _col11, _col12, _col13
+ outputColumnNames: _col2, _col5, _col6, _col7, _col8, _col9, _col10, _col11
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string), _col12 (type: string), _col13 (type: string), (_col3 - _col2) (type: decimal(8,2))
+ Group By Operator
+ aggregations: sum(_col2)
+ keys: _col5 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col7)
- keys: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
+ sort order: +++++++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- sort order: +++++++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col7 (type: decimal(18,2))
+ value expressions: _col7 (type: decimal(18,2))
Reducer 24
Execution mode: vectorized
Reduce Operator Tree:
@@ -637,24 +625,20 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col3, _col7, _col8, _col9, _col10, _col11, _col12, _col13
+ outputColumnNames: _col2, _col5, _col6, _col7, _col8, _col9, _col10, _col11
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string), _col12 (type: string), _col13 (type: string), (_col3 - _col2) (type: decimal(8,2))
+ Group By Operator
+ aggregations: sum(_col2)
+ keys: _col5 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type: string), _col9 (type: string), _col10 (type: string), _col11 (type: string)
+ mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- aggregations: sum(_col7)
- keys: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- mode: hash
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
+ sort order: +++++++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- sort order: +++++++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string), _col2 (type: string), _col3 (type: string), _col4 (type: string), _col5 (type: string), _col6 (type: string)
- Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col7 (type: decimal(18,2))
+ value expressions: _col7 (type: decimal(18,2))
Reducer 4
Execution mode: vectorized
Reduce Operator Tree:
@@ -686,13 +670,13 @@ STAGE PLANS:
1 _col0 (type: string)
2 _col0 (type: string)
3 _col0 (type: string)
- outputColumnNames: _col1, _col3, _col5, _col7, _col8
+ outputColumnNames: _col1, _col3, _col5, _col6, _col8, _col9
Statistics: Num rows: 1149975359 Data size: 101451160012 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: CASE WHEN (_col3 is not null) THEN (CASE WHEN (_col5 is not null) THEN (((_col1 / _col5) > (_col8 / _col3))) ELSE ((null > (_col8 / _col3))) END) ELSE (CASE WHEN (_col5 is not null) THEN (((_col1 / _col5) > null)) ELSE (null) END) END (type: boolean)
+ predicate: CASE WHEN (_col3 is not null) THEN (CASE WHEN (_col6) THEN (((_col1 / _col5) > (_col9 / _col3))) ELSE ((null > (_col9 / _col3))) END) ELSE (CASE WHEN (_col6) THEN (((_col1 / _col5) > null)) ELSE (null) END) END (type: boolean)
Statistics: Num rows: 574987679 Data size: 50725579961 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col7 (type: string)
+ expressions: _col8 (type: string)
outputColumnNames: _col0
Statistics: Num rows: 574987679 Data size: 50725579961 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/perf/spark/query12.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query12.q.out b/ql/src/test/results/clientpositive/perf/spark/query12.q.out
index 903fcee..6ec6b73 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query12.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query12.q.out
@@ -170,11 +170,11 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col6, _col7, _col8, _col9, _col10
+ outputColumnNames: _col2, _col5, _col6, _col7, _col8, _col9
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
Group By Operator
aggregations: sum(_col2)
- keys: _col10 (type: string), _col9 (type: string), _col6 (type: string), _col7 (type: string), _col8 (type: decimal(7,2))
+ keys: _col9 (type: string), _col8 (type: string), _col5 (type: string), _col6 (type: string), _col7 (type: decimal(7,2))
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 174243235 Data size: 23692040863 Basic stats: COMPLETE Column stats: NONE
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/perf/spark/query13.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query13.q.out b/ql/src/test/results/clientpositive/perf/spark/query13.q.out
index 4d530fc..2b6c19d 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query13.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query13.q.out
@@ -114,8 +114,7 @@ POSTHOOK: Input: default@store_sales
#### A masked pattern was here ####
STAGE DEPENDENCIES:
Stage-2 is a root stage
- Stage-3 depends on stages: Stage-2
- Stage-1 depends on stages: Stage-3
+ Stage-1 depends on stages: Stage-2
Stage-0 depends on stages: Stage-1
STAGE PLANS:
@@ -123,6 +122,26 @@ STAGE PLANS:
Spark
#### A masked pattern was here ####
Vertices:
+ Map 7
+ Map Operator Tree:
+ TableScan
+ alias: store
+ filterExpr: s_store_sk is not null (type: boolean)
+ Statistics: Num rows: 1704 Data size: 3256276 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: s_store_sk is not null (type: boolean)
+ Statistics: Num rows: 1704 Data size: 3256276 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: s_store_sk (type: int)
+ outputColumnNames: _col0
+ Statistics: Num rows: 1704 Data size: 3256276 Basic stats: COMPLETE Column stats: NONE
+ Spark HashTable Sink Operator
+ keys:
+ 0 _col5 (type: int)
+ 1 _col0 (type: int)
+ Execution mode: vectorized
+ Local Work:
+ Map Reduce Local Work
Map 8
Map Operator Tree:
TableScan
@@ -133,8 +152,8 @@ STAGE PLANS:
predicate: ((hd_dep_count) IN (3, 1) and hd_demo_sk is not null) (type: boolean)
Statistics: Num rows: 7200 Data size: 770400 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: hd_demo_sk (type: int), hd_dep_count (type: int)
- outputColumnNames: _col0, _col1
+ expressions: hd_demo_sk (type: int), (hd_dep_count = 3) (type: boolean), (hd_dep_count = 1) (type: boolean)
+ outputColumnNames: _col0, _col1, _col2
Statistics: Num rows: 7200 Data size: 770400 Basic stats: COMPLETE Column stats: NONE
Spark HashTable Sink Operator
keys:
@@ -144,40 +163,34 @@ STAGE PLANS:
Local Work:
Map Reduce Local Work
- Stage: Stage-3
+ Stage: Stage-1
Spark
+ Edges:
+ Reducer 2 <- Map 1 (PARTITION-LEVEL SORT, 45), Map 6 (PARTITION-LEVEL SORT, 45)
+ Reducer 3 <- Map 9 (PARTITION-LEVEL SORT, 218), Reducer 2 (PARTITION-LEVEL SORT, 218)
+ Reducer 4 <- Map 10 (PARTITION-LEVEL SORT, 55), Reducer 3 (PARTITION-LEVEL SORT, 55)
+ Reducer 5 <- Reducer 4 (GROUP, 1)
#### A masked pattern was here ####
Vertices:
Map 1
Map Operator Tree:
TableScan
- alias: store
- filterExpr: s_store_sk is not null (type: boolean)
- Statistics: Num rows: 1704 Data size: 3256276 Basic stats: COMPLETE Column stats: NONE
+ alias: date_dim
+ filterExpr: ((d_year = 2001) and d_date_sk is not null) (type: boolean)
+ Statistics: Num rows: 73049 Data size: 81741831 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: s_store_sk is not null (type: boolean)
- Statistics: Num rows: 1704 Data size: 3256276 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((d_year = 2001) and d_date_sk is not null) (type: boolean)
+ Statistics: Num rows: 36524 Data size: 40870356 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: s_store_sk (type: int)
+ expressions: d_date_sk (type: int)
outputColumnNames: _col0
- Statistics: Num rows: 1704 Data size: 3256276 Basic stats: COMPLETE Column stats: NONE
- Spark HashTable Sink Operator
- keys:
- 0 _col0 (type: int)
- 1 _col4 (type: int)
+ Statistics: Num rows: 36524 Data size: 40870356 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col0 (type: int)
+ Statistics: Num rows: 36524 Data size: 40870356 Basic stats: COMPLETE Column stats: NONE
Execution mode: vectorized
- Local Work:
- Map Reduce Local Work
-
- Stage: Stage-1
- Spark
- Edges:
- Reducer 3 <- Map 2 (PARTITION-LEVEL SORT, 49), Map 7 (PARTITION-LEVEL SORT, 49)
- Reducer 4 <- Map 9 (PARTITION-LEVEL SORT, 218), Reducer 3 (PARTITION-LEVEL SORT, 218)
- Reducer 5 <- Map 10 (PARTITION-LEVEL SORT, 28), Reducer 4 (PARTITION-LEVEL SORT, 28)
- Reducer 6 <- Reducer 5 (GROUP, 1)
-#### A masked pattern was here ####
- Vertices:
Map 10
Map Operator Tree:
TableScan
@@ -188,17 +201,17 @@ STAGE PLANS:
predicate: ((cd_education_status) IN ('4 yr Degree', 'Primary', 'Advanced Degree') and (cd_marital_status) IN ('M', 'D', 'U') and cd_demo_sk is not null) (type: boolean)
Statistics: Num rows: 1861800 Data size: 717186159 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cd_demo_sk (type: int), cd_marital_status (type: string), cd_education_status (type: string)
- outputColumnNames: _col0, _col1, _col2
+ expressions: cd_demo_sk (type: int), (cd_marital_status = 'M') (type: boolean), (cd_education_status = '4 yr Degree') (type: boolean), (cd_marital_status = 'D') (type: boolean), (cd_education_status = 'Primary') (type: boolean), (cd_marital_status = 'U') (type: boolean), (cd_education_status = 'Advanced Degree') (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6
Statistics: Num rows: 1861800 Data size: 717186159 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 1861800 Data size: 717186159 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string), _col2 (type: string)
+ value expressions: _col1 (type: boolean), _col2 (type: boolean), _col3 (type: boolean), _col4 (type: boolean), _col5 (type: boolean), _col6 (type: boolean)
Execution mode: vectorized
- Map 2
+ Map 6
Map Operator Tree:
TableScan
alias: store_sales
@@ -208,46 +221,15 @@ STAGE PLANS:
predicate: ((ss_net_profit BETWEEN 100 AND 200 or ss_net_profit BETWEEN 150 AND 300 or ss_net_profit BETWEEN 50 AND 250) and (ss_sales_price BETWEEN 100 AND 150 or ss_sales_price BETWEEN 50 AND 100 or ss_sales_price BETWEEN 150 AND 200) and ss_addr_sk is not null and ss_cdemo_sk is not null and ss_hdemo_sk is not null and ss_sold_date_sk is not null and ss_store_sk is not null) (type: boolean)
Statistics: Num rows: 63999513 Data size: 5646055611 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ss_sold_date_sk (type: int), ss_cdemo_sk (type: int), ss_hdemo_sk (type: int), ss_addr_sk (type: int), ss_store_sk (type: int), ss_quantity (type: int), ss_sales_price (type: decimal(7,2)), ss_ext_sales_price (type: decimal(7,2)), ss_ext_wholesale_cost (type: decimal(7,2)), ss_net_profit (type: decimal(7,2))
- outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9
+ expressions: ss_sold_date_sk (type: int), ss_cdemo_sk (type: int), ss_hdemo_sk (type: int), ss_addr_sk (type: int), ss_store_sk (type: int), ss_quantity (type: int), ss_ext_sales_price (type: decimal(7,2)), ss_ext_wholesale_cost (type: decimal(7,2)), ss_net_profit BETWEEN 100 AND 200 (type: boolean), ss_net_profit BETWEEN 150 AND 300 (type: boolean), ss_net_profit BETWEEN 50 AND 250 (type: boolean), ss_sales_price BETWEEN 100 AND 150 (type: boolean), ss_sales_price BETWEEN 50 AND 100 (type: boolean), ss_sales_price BETWEEN 150 AND 200 (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13
Statistics: Num rows: 63999513 Data size: 5646055611 Basic stats: COMPLETE Column stats: NONE
- Map Join Operator
- condition map:
- Inner Join 0 to 1
- keys:
- 0 _col0 (type: int)
- 1 _col4 (type: int)
- outputColumnNames: _col1, _col2, _col3, _col4, _col6, _col7, _col8, _col9, _col10
- input vertices:
- 0 Map 1
- Statistics: Num rows: 70399465 Data size: 6210661306 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col1 (type: int)
- sort order: +
- Map-reduce partition columns: _col1 (type: int)
- Statistics: Num rows: 70399465 Data size: 6210661306 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: int), _col3 (type: int), _col4 (type: int), _col6 (type: int), _col7 (type: decimal(7,2)), _col8 (type: decimal(7,2)), _col9 (type: decimal(7,2)), _col10 (type: decimal(7,2))
- Execution mode: vectorized
- Local Work:
- Map Reduce Local Work
- Map 7
- Map Operator Tree:
- TableScan
- alias: date_dim
- filterExpr: ((d_year = 2001) and d_date_sk is not null) (type: boolean)
- Statistics: Num rows: 73049 Data size: 81741831 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: ((d_year = 2001) and d_date_sk is not null) (type: boolean)
- Statistics: Num rows: 36524 Data size: 40870356 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: d_date_sk (type: int)
- outputColumnNames: _col0
- Statistics: Num rows: 36524 Data size: 40870356 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
- Statistics: Num rows: 36524 Data size: 40870356 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 63999513 Data size: 5646055611 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col1 (type: int), _col2 (type: int), _col3 (type: int), _col4 (type: int), _col5 (type: int), _col6 (type: decimal(7,2)), _col7 (type: decimal(7,2)), _col8 (type: boolean), _col9 (type: boolean), _col10 (type: boolean), _col11 (type: boolean), _col12 (type: boolean), _col13 (type: boolean)
Execution mode: vectorized
Map 9
Map Operator Tree:
@@ -259,17 +241,17 @@ STAGE PLANS:
predicate: ((ca_country = 'United States') and (ca_state) IN ('KY', 'GA', 'NM', 'MT', 'OR', 'IN', 'WI', 'MO', 'WV') and ca_address_sk is not null) (type: boolean)
Statistics: Num rows: 20000000 Data size: 20297597642 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ca_address_sk (type: int), ca_state (type: string)
- outputColumnNames: _col0, _col1
+ expressions: ca_address_sk (type: int), (ca_state) IN ('KY', 'GA', 'NM') (type: boolean), (ca_state) IN ('MT', 'OR', 'IN') (type: boolean), (ca_state) IN ('WI', 'MO', 'WV') (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 20000000 Data size: 20297597642 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 20000000 Data size: 20297597642 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string)
+ value expressions: _col1 (type: boolean), _col2 (type: boolean), _col3 (type: boolean)
Execution mode: vectorized
- Reducer 3
+ Reducer 2
Local Work:
Map Reduce Local Work
Reduce Operator Tree:
@@ -277,27 +259,37 @@ STAGE PLANS:
condition map:
Inner Join 0 to 1
keys:
- 0 _col1 (type: int)
+ 0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col3, _col4, _col6, _col7, _col8, _col9, _col10
- Statistics: Num rows: 77439413 Data size: 6831727584 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14
+ Statistics: Num rows: 70399465 Data size: 6210661306 Basic stats: COMPLETE Column stats: NONE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:
- 0 _col3 (type: int)
+ 0 _col5 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col4, _col6, _col7, _col8, _col9, _col10, _col14
+ outputColumnNames: _col2, _col3, _col4, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14
input vertices:
- 1 Map 8
- Statistics: Num rows: 85183356 Data size: 7514900505 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col4 (type: int)
- sort order: +
- Map-reduce partition columns: _col4 (type: int)
+ 1 Map 7
+ Statistics: Num rows: 77439413 Data size: 6831727584 Basic stats: COMPLETE Column stats: NONE
+ Map Join Operator
+ condition map:
+ Inner Join 0 to 1
+ keys:
+ 0 _col3 (type: int)
+ 1 _col0 (type: int)
+ outputColumnNames: _col2, _col4, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col17, _col18
+ input vertices:
+ 1 Map 8
Statistics: Num rows: 85183356 Data size: 7514900505 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: int), _col6 (type: int), _col7 (type: decimal(7,2)), _col8 (type: decimal(7,2)), _col9 (type: decimal(7,2)), _col10 (type: decimal(7,2)), _col14 (type: int)
- Reducer 4
+ Reduce Output Operator
+ key expressions: _col4 (type: int)
+ sort order: +
+ Map-reduce partition columns: _col4 (type: int)
+ Statistics: Num rows: 85183356 Data size: 7514900505 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col2 (type: int), _col6 (type: int), _col7 (type: decimal(7,2)), _col8 (type: decimal(7,2)), _col9 (type: boolean), _col10 (type: boolean), _col11 (type: boolean), _col12 (type: boolean), _col13 (type: boolean), _col14 (type: boolean), _col17 (type: boolean), _col18 (type: boolean)
+ Reducer 3
Reduce Operator Tree:
Join Operator
condition map:
@@ -305,18 +297,18 @@ STAGE PLANS:
keys:
0 _col4 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col2, _col6, _col7, _col8, _col9, _col10, _col14, _col16
+ outputColumnNames: _col2, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col17, _col18, _col20, _col21, _col22
Statistics: Num rows: 93701693 Data size: 8266390734 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (((_col16) IN ('KY', 'GA', 'NM') and _col10 BETWEEN 100 AND 200) or ((_col16) IN ('MT', 'OR', 'IN') and _col10 BETWEEN 150 AND 300) or ((_col16) IN ('WI', 'MO', 'WV') and _col10 BETWEEN 50 AND 250)) (type: boolean)
- Statistics: Num rows: 31233897 Data size: 2755463519 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((_col20 and _col9) or (_col21 and _col10) or (_col22 and _col11)) (type: boolean)
+ Statistics: Num rows: 70276269 Data size: 6199792984 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col2 (type: int)
sort order: +
Map-reduce partition columns: _col2 (type: int)
- Statistics: Num rows: 31233897 Data size: 2755463519 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col6 (type: int), _col7 (type: decimal(7,2)), _col8 (type: decimal(7,2)), _col9 (type: decimal(7,2)), _col14 (type: int)
- Reducer 5
+ Statistics: Num rows: 70276269 Data size: 6199792984 Basic stats: COMPLETE Column stats: NONE
+ value expressions: _col6 (type: int), _col7 (type: decimal(7,2)), _col8 (type: decimal(7,2)), _col12 (type: boolean), _col13 (type: boolean), _col14 (type: boolean), _col17 (type: boolean), _col18 (type: boolean)
+ Reducer 4
Reduce Operator Tree:
Join Operator
condition map:
@@ -324,17 +316,17 @@ STAGE PLANS:
keys:
0 _col2 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col6, _col7, _col8, _col9, _col14, _col19, _col20
- Statistics: Num rows: 34357287 Data size: 3031009936 Basic stats: COMPLETE Column stats: NONE
+ outputColumnNames: _col6, _col7, _col8, _col12, _col13, _col14, _col17, _col18, _col24, _col25, _col26, _col27, _col28, _col29
+ Statistics: Num rows: 77303897 Data size: 6819772430 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: (((_col19 = 'D') and (_col20 = 'Primary') and _col7 BETWEEN 50 AND 100 and (_col14 = 1)) or ((_col19 = 'M') and (_col20 = '4 yr Degree') and _col7 BETWEEN 100 AND 150 and (_col14 = 3)) or ((_col19 = 'U') and (_col20 = 'Advanced Degree') and _col7 BETWEEN 150 AND 200 and (_col14 = 1))) (type: boolean)
- Statistics: Num rows: 1431552 Data size: 126291937 Basic stats: COMPLETE Column stats: NONE
+ predicate: ((_col24 and _col25 and _col12 and _col17) or (_col26 and _col27 and _col13 and _col18) or (_col28 and _col29 and _col14 and _col18)) (type: boolean)
+ Statistics: Num rows: 14494479 Data size: 1278707181 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col6 (type: int), _col8 (type: decimal(7,2)), _col9 (type: decimal(7,2))
- outputColumnNames: _col6, _col8, _col9
- Statistics: Num rows: 1431552 Data size: 126291937 Basic stats: COMPLETE Column stats: NONE
+ expressions: _col6 (type: int), _col7 (type: decimal(7,2)), _col8 (type: decimal(7,2))
+ outputColumnNames: _col6, _col7, _col8
+ Statistics: Num rows: 14494479 Data size: 1278707181 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col6), count(_col6), sum(_col8), count(_col8), sum(_col9), count(_col9)
+ aggregations: sum(_col6), count(_col6), sum(_col7), count(_col7), sum(_col8), count(_col8)
mode: hash
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5
Statistics: Num rows: 1 Data size: 256 Basic stats: COMPLETE Column stats: NONE
@@ -342,7 +334,7 @@ STAGE PLANS:
sort order:
Statistics: Num rows: 1 Data size: 256 Basic stats: COMPLETE Column stats: NONE
value expressions: _col0 (type: bigint), _col1 (type: bigint), _col2 (type: decimal(17,2)), _col3 (type: bigint), _col4 (type: decimal(17,2)), _col5 (type: bigint)
- Reducer 6
+ Reducer 5
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/perf/spark/query15.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query15.q.out b/ql/src/test/results/clientpositive/perf/spark/query15.q.out
index d0017e4..8b45053 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query15.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query15.q.out
@@ -91,15 +91,15 @@ STAGE PLANS:
predicate: ca_address_sk is not null (type: boolean)
Statistics: Num rows: 40000000 Data size: 40595195284 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: ca_address_sk (type: int), ca_state (type: string), ca_zip (type: string)
- outputColumnNames: _col0, _col1, _col2
+ expressions: ca_address_sk (type: int), ca_zip (type: string), (substr(ca_zip, 1, 5)) IN ('85669', '86197', '88274', '83405', '86475', '85392', '85460', '80348', '81792') (type: boolean), (ca_state) IN ('CA', 'WA', 'GA') (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 40000000 Data size: 40595195284 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 40000000 Data size: 40595195284 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: string), _col2 (type: string)
+ value expressions: _col1 (type: string), _col2 (type: boolean), _col3 (type: boolean)
Execution mode: vectorized
Map 7
Map Operator Tree:
@@ -111,15 +111,15 @@ STAGE PLANS:
predicate: (cs_bill_customer_sk is not null and cs_sold_date_sk is not null) (type: boolean)
Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: cs_sold_date_sk (type: int), cs_bill_customer_sk (type: int), cs_sales_price (type: decimal(7,2))
- outputColumnNames: _col0, _col1, _col2
+ expressions: cs_sold_date_sk (type: int), cs_bill_customer_sk (type: int), cs_sales_price (type: decimal(7,2)), (cs_sales_price > 500) (type: boolean)
+ outputColumnNames: _col0, _col1, _col2, _col3
Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 287989836 Data size: 38999608952 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col1 (type: int), _col2 (type: decimal(7,2))
+ value expressions: _col1 (type: int), _col2 (type: decimal(7,2)), _col3 (type: boolean)
Execution mode: vectorized
Map 9
Map Operator Tree:
@@ -148,14 +148,14 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col0, _col3, _col4
+ outputColumnNames: _col0, _col3, _col4, _col5
Statistics: Num rows: 88000001 Data size: 75681779077 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: int)
sort order: +
Map-reduce partition columns: _col0 (type: int)
Statistics: Num rows: 88000001 Data size: 75681779077 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: string), _col4 (type: string)
+ value expressions: _col3 (type: string), _col4 (type: boolean), _col5 (type: boolean)
Reducer 3
Reduce Operator Tree:
Join Operator
@@ -164,18 +164,18 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col1 (type: int)
- outputColumnNames: _col3, _col4, _col7
+ outputColumnNames: _col3, _col4, _col5, _col8, _col9
Statistics: Num rows: 348467716 Data size: 47189528877 Basic stats: COMPLETE Column stats: NONE
Filter Operator
- predicate: ((_col3) IN ('CA', 'WA', 'GA') or (_col7 > 500) or (substr(_col4, 1, 5)) IN ('85669', '86197', '88274', '83405', '86475', '85392', '85460', '80348', '81792')) (type: boolean)
+ predicate: (_col4 or _col5 or _col9) (type: boolean)
Statistics: Num rows: 348467716 Data size: 47189528877 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col4 (type: string), _col7 (type: decimal(7,2))
- outputColumnNames: _col4, _col7
+ expressions: _col3 (type: string), _col8 (type: decimal(7,2))
+ outputColumnNames: _col3, _col8
Statistics: Num rows: 348467716 Data size: 47189528877 Basic stats: COMPLETE Column stats: NONE
Group By Operator
- aggregations: sum(_col7)
- keys: _col4 (type: string)
+ aggregations: sum(_col8)
+ keys: _col3 (type: string)
mode: hash
outputColumnNames: _col0, _col1
Statistics: Num rows: 348467716 Data size: 47189528877 Basic stats: COMPLETE Column stats: NONE
@@ -226,14 +226,14 @@ STAGE PLANS:
keys:
0 _col0 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2
+ outputColumnNames: _col1, _col2, _col3
Statistics: Num rows: 316788826 Data size: 42899570777 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int)
sort order: +
Map-reduce partition columns: _col1 (type: int)
Statistics: Num rows: 316788826 Data size: 42899570777 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col2 (type: decimal(7,2))
+ value expressions: _col2 (type: decimal(7,2)), _col3 (type: boolean)
Stage: Stage-0
Fetch Operator
http://git-wip-us.apache.org/repos/asf/hive/blob/55887646/ql/src/test/results/clientpositive/perf/spark/query17.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/spark/query17.q.out b/ql/src/test/results/clientpositive/perf/spark/query17.q.out
index ba78752..7505449 100644
--- a/ql/src/test/results/clientpositive/perf/spark/query17.q.out
+++ b/ql/src/test/results/clientpositive/perf/spark/query17.q.out
@@ -305,14 +305,14 @@ STAGE PLANS:
keys:
0 _col2 (type: int), _col1 (type: int)
1 _col1 (type: int), _col2 (type: int)
- outputColumnNames: _col3, _col7, _col8, _col9, _col10
+ outputColumnNames: _col3, _col6, _col7, _col8, _col9
Statistics: Num rows: 348467716 Data size: 47189528877 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
- key expressions: _col7 (type: int), _col8 (type: int), _col9 (type: int)
+ key expressions: _col6 (type: int), _col7 (type: int), _col8 (type: int)
sort order: +++
- Map-reduce partition columns: _col7 (type: int), _col8 (type: int), _col9 (type: int)
+ Map-reduce partition columns: _col6 (type: int), _col7 (type: int), _col8 (type: int)
Statistics: Num rows: 348467716 Data size: 47189528877 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: int), _col10 (type: int)
+ value expressions: _col3 (type: int), _col9 (type: int)
Reducer 14
Reduce Operator Tree:
Join Operator
@@ -353,14 +353,14 @@ STAGE PLANS:
keys:
0 _col1 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col1, _col2, _col3, _col4, _col5, _col9, _col10
+ outputColumnNames: _col1, _col2, _col3, _col4, _col5, _col8, _col9
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: int), _col2 (type: int), _col4 (type: int)
sort order: +++
Map-reduce partition columns: _col1 (type: int), _col2 (type: int), _col4 (type: int)
Statistics: Num rows: 696954748 Data size: 61485550191 Basic stats: COMPLETE Column stats: NONE
- value expressions: _col3 (type: int), _col5 (type: int), _col9 (type: string), _col10 (type: string)
+ value expressions: _col3 (type: int), _col5 (type: int), _col8 (type: string), _col9 (type: string)
Reducer 4
Local Work:
Map Reduce Local Work
@@ -370,8 +370,8 @@ STAGE PLANS:
Inner Join 0 to 1
keys:
0 _col1 (type: int), _col2 (type: int), _col4 (type: int)
- 1 _col7 (type: int), _col8 (type: int), _col9 (type: int)
- outputColumnNames: _col3, _col5, _col9, _col10, _col14, _col21
+ 1 _col6 (type: int), _col7 (type: int), _col8 (type: int)
+ outputColumnNames: _col3, _col5, _col8, _col9, _col13, _col19
Statistics: Num rows: 766650239 Data size: 67634106676 Basic stats: COMPLETE Column stats: NONE
Map Join Operator
condition map:
@@ -379,12 +379,12 @@ STAGE PLANS:
keys:
0 _col3 (type: int)
1 _col0 (type: int)
- outputColumnNames: _col5, _col9, _col10, _col14, _col21, _col25
+ outputColumnNames: _col5, _col8, _col9, _col13, _col19, _col22
input vertices:
1 Map 16
Statistics: Num rows: 843315281 Data size: 74397518956 Basic stats: COMPLETE Column stats: NONE
Select Operator
- expressions: _col9 (type: string), _col10 (type: string), _col25 (type: string), _col5 (type: int), _col21 (type: int), _col14 (type: int), UDFToDouble(_col5) (type: double), (UDFToDouble(_col5) * UDFToDouble(_col5)) (type: double), UDFToDouble(_col21) (type: double), (UDFToDouble(_col21) * UDFToDouble(_col21)) (type: double), UDFToDouble(_col14) (type: double), (UDFToDouble(_col14) * UDFToDouble(_col14)) (type: double)
+ expressions: _col8 (type: string), _col9 (type: string), _col22 (type: string), _col5 (type: int), _col19 (type: int), _col13 (type: int), UDFToDouble(_col5) (type: double), (UDFToDouble(_col5) * UDFToDouble(_col5)) (type: double), UDFToDouble(_col19) (type: double), (UDFToDouble(_col19) * UDFToDouble(_col19)) (type: double), UDFToDouble(_col13) (type: double), (UDFToDouble(_col13) * UDFToDouble(_col13)) (type: double)
outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11
Statistics: Num rows: 843315281 Data size: 74397518956 Basic stats: COMPLETE Column stats: NONE
Group By Operator