You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hive.apache.org by jc...@apache.org on 2018/03/29 05:42:44 UTC
[02/13] hive git commit: HIVE-18770: Additional tests and fixes for
materialized view rewriting (Jesus Camacho Rodriguez,
reviewed by Ashutosh Chauhan)
http://git-wip-us.apache.org/repos/asf/hive/blob/1b01f9c4/ql/src/test/results/clientpositive/perf/tez/query87.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/tez/query87.q.out b/ql/src/test/results/clientpositive/perf/tez/query87.q.out
index 19f0e00..0455411 100644
--- a/ql/src/test/results/clientpositive/perf/tez/query87.q.out
+++ b/ql/src/test/results/clientpositive/perf/tez/query87.q.out
@@ -61,143 +61,143 @@ Stage-0
limit:-1
Stage-1
Reducer 9
- File Output Operator [FS_104]
- Group By Operator [GBY_102] (rows=1 width=24)
+ File Output Operator [FS_101]
+ Group By Operator [GBY_99] (rows=1 width=24)
Output:["_col0"],aggregations:["count(VALUE._col0)"]
<-Reducer 8 [CUSTOM_SIMPLE_EDGE]
- PARTITION_ONLY_SHUFFLE [RS_101]
- Group By Operator [GBY_100] (rows=1 width=24)
+ PARTITION_ONLY_SHUFFLE [RS_98]
+ Group By Operator [GBY_97] (rows=1 width=24)
Output:["_col0"],aggregations:["count()"]
- Select Operator [SEL_98] (rows=4537552 width=129)
- Filter Operator [FIL_97] (rows=4537552 width=129)
+ Select Operator [SEL_95] (rows=4537552 width=129)
+ Filter Operator [FIL_94] (rows=4537552 width=129)
predicate:(((_col3 * 2) = _col4) and (_col3 > 0L))
- Select Operator [SEL_129] (rows=27225312 width=129)
+ Select Operator [SEL_126] (rows=27225312 width=129)
Output:["_col3","_col4"]
- Group By Operator [GBY_96] (rows=27225312 width=129)
+ Group By Operator [GBY_93] (rows=27225312 width=129)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0, KEY._col1, KEY._col2
<-Union 7 [SIMPLE_EDGE]
<-Reducer 16 [CONTAINS]
- Reduce Output Operator [RS_95]
+ Reduce Output Operator [RS_92]
PartitionCols:_col0, _col1, _col2
- Group By Operator [GBY_94] (rows=54450625 width=129)
+ Group By Operator [GBY_91] (rows=54450625 width=129)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(_col3)","sum(_col4)"],keys:_col0, _col1, _col2
- Select Operator [SEL_92] (rows=54450625 width=129)
+ Select Operator [SEL_89] (rows=54450625 width=129)
Output:["_col0","_col1","_col2","_col3","_col4"]
- Select Operator [SEL_89] (rows=43560808 width=135)
+ Select Operator [SEL_87] (rows=43560808 width=135)
Output:["_col0","_col1","_col2","_col3","_col4"]
- Group By Operator [GBY_88] (rows=43560808 width=135)
+ Group By Operator [GBY_86] (rows=43560808 width=135)
Output:["_col0","_col1","_col2","_col3"],aggregations:["count()"],keys:_col1, _col0, _col2
- Select Operator [SEL_84] (rows=87121617 width=135)
+ Select Operator [SEL_82] (rows=87121617 width=135)
Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_83] (rows=87121617 width=135)
+ Group By Operator [GBY_81] (rows=87121617 width=135)
Output:["_col0","_col1","_col2"],keys:KEY._col0, KEY._col1, KEY._col2
<-Reducer 15 [SIMPLE_EDGE]
- SHUFFLE [RS_82]
+ SHUFFLE [RS_80]
PartitionCols:_col0, _col1, _col2
- Group By Operator [GBY_81] (rows=174243235 width=135)
+ Group By Operator [GBY_79] (rows=174243235 width=135)
Output:["_col0","_col1","_col2"],keys:_col7, _col6, _col3
- Merge Join Operator [MERGEJOIN_135] (rows=174243235 width=135)
- Conds:RS_77._col1=RS_78._col0(Inner),Output:["_col3","_col6","_col7"]
+ Merge Join Operator [MERGEJOIN_132] (rows=174243235 width=135)
+ Conds:RS_75._col1=RS_76._col0(Inner),Output:["_col3","_col6","_col7"]
<-Map 17 [SIMPLE_EDGE]
- SHUFFLE [RS_78]
+ SHUFFLE [RS_76]
PartitionCols:_col0
Select Operator [SEL_8] (rows=80000000 width=860)
Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_122] (rows=80000000 width=860)
+ Filter Operator [FIL_119] (rows=80000000 width=860)
predicate:c_customer_sk is not null
TableScan [TS_6] (rows=80000000 width=860)
default@customer,customer,Tbl:COMPLETE,Col:NONE,Output:["c_customer_sk","c_first_name","c_last_name"]
<-Reducer 14 [SIMPLE_EDGE]
- SHUFFLE [RS_77]
+ SHUFFLE [RS_75]
PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_134] (rows=158402938 width=135)
- Conds:RS_74._col0=RS_75._col0(Inner),Output:["_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_131] (rows=158402938 width=135)
+ Conds:RS_72._col0=RS_73._col0(Inner),Output:["_col1","_col3"]
<-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_75]
+ SHUFFLE [RS_73]
PartitionCols:_col0
Select Operator [SEL_5] (rows=8116 width=1119)
Output:["_col0","_col1"]
- Filter Operator [FIL_121] (rows=8116 width=1119)
+ Filter Operator [FIL_118] (rows=8116 width=1119)
predicate:(d_date_sk is not null and d_month_seq BETWEEN 1212 AND 1223)
TableScan [TS_3] (rows=73049 width=1119)
default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_date","d_month_seq"]
<-Map 19 [SIMPLE_EDGE]
- SHUFFLE [RS_74]
+ SHUFFLE [RS_72]
PartitionCols:_col0
- Select Operator [SEL_67] (rows=144002668 width=135)
+ Select Operator [SEL_65] (rows=144002668 width=135)
Output:["_col0","_col1"]
- Filter Operator [FIL_126] (rows=144002668 width=135)
+ Filter Operator [FIL_123] (rows=144002668 width=135)
predicate:(ws_bill_customer_sk is not null and ws_sold_date_sk is not null)
- TableScan [TS_65] (rows=144002668 width=135)
+ TableScan [TS_63] (rows=144002668 width=135)
default@web_sales,web_sales,Tbl:COMPLETE,Col:NONE,Output:["ws_sold_date_sk","ws_bill_customer_sk"]
<-Reducer 6 [CONTAINS]
- Reduce Output Operator [RS_95]
+ Reduce Output Operator [RS_92]
PartitionCols:_col0, _col1, _col2
- Group By Operator [GBY_94] (rows=54450625 width=129)
+ Group By Operator [GBY_91] (rows=54450625 width=129)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(_col3)","sum(_col4)"],keys:_col0, _col1, _col2
- Select Operator [SEL_92] (rows=54450625 width=129)
+ Select Operator [SEL_89] (rows=54450625 width=129)
Output:["_col0","_col1","_col2","_col3","_col4"]
- Select Operator [SEL_64] (rows=10889817 width=103)
+ Select Operator [SEL_62] (rows=10889817 width=103)
Output:["_col0","_col1","_col2","_col3","_col4"]
- Group By Operator [GBY_63] (rows=10889817 width=103)
+ Group By Operator [GBY_61] (rows=10889817 width=103)
Output:["_col0","_col1","_col2","_col3"],aggregations:["count()"],keys:_col0, _col1, _col2
- Select Operator [SEL_59] (rows=21779634 width=103)
+ Select Operator [SEL_57] (rows=21779634 width=103)
Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_58] (rows=21779634 width=103)
+ Filter Operator [FIL_56] (rows=21779634 width=103)
predicate:(((_col3 * 2) = _col4) and (_col3 > 0L))
- Group By Operator [GBY_57] (rows=130677808 width=103)
+ Group By Operator [GBY_55] (rows=130677808 width=103)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(VALUE._col0)","sum(VALUE._col1)"],keys:KEY._col0, KEY._col1, KEY._col2
<-Union 5 [SIMPLE_EDGE]
<-Reducer 13 [CONTAINS]
- Reduce Output Operator [RS_56]
+ Reduce Output Operator [RS_54]
PartitionCols:_col0, _col1, _col2
- Group By Operator [GBY_55] (rows=261355616 width=103)
+ Group By Operator [GBY_53] (rows=261355616 width=103)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(_col3)","sum(_col4)"],keys:_col0, _col1, _col2
- Select Operator [SEL_53] (rows=261355616 width=103)
+ Select Operator [SEL_51] (rows=261355616 width=103)
Output:["_col0","_col1","_col2","_col3","_col4"]
- Select Operator [SEL_50] (rows=87116929 width=135)
+ Select Operator [SEL_49] (rows=87116929 width=135)
Output:["_col0","_col1","_col2","_col3","_col4"]
- Group By Operator [GBY_49] (rows=87116929 width=135)
+ Group By Operator [GBY_48] (rows=87116929 width=135)
Output:["_col0","_col1","_col2","_col3"],aggregations:["count()"],keys:_col1, _col0, _col2
- Select Operator [SEL_45] (rows=174233858 width=135)
+ Select Operator [SEL_44] (rows=174233858 width=135)
Output:["_col0","_col1","_col2"]
- Group By Operator [GBY_44] (rows=174233858 width=135)
+ Group By Operator [GBY_43] (rows=174233858 width=135)
Output:["_col0","_col1","_col2"],keys:KEY._col0, KEY._col1, KEY._col2
<-Reducer 12 [SIMPLE_EDGE]
- SHUFFLE [RS_43]
+ SHUFFLE [RS_42]
PartitionCols:_col0, _col1, _col2
- Group By Operator [GBY_42] (rows=348467716 width=135)
+ Group By Operator [GBY_41] (rows=348467716 width=135)
Output:["_col0","_col1","_col2"],keys:_col7, _col6, _col3
- Merge Join Operator [MERGEJOIN_133] (rows=348467716 width=135)
- Conds:RS_38._col1=RS_39._col0(Inner),Output:["_col3","_col6","_col7"]
+ Merge Join Operator [MERGEJOIN_130] (rows=348467716 width=135)
+ Conds:RS_37._col1=RS_38._col0(Inner),Output:["_col3","_col6","_col7"]
<-Map 17 [SIMPLE_EDGE]
- SHUFFLE [RS_39]
+ SHUFFLE [RS_38]
PartitionCols:_col0
Please refer to the previous Select Operator [SEL_8]
<-Reducer 11 [SIMPLE_EDGE]
- SHUFFLE [RS_38]
+ SHUFFLE [RS_37]
PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_132] (rows=316788826 width=135)
- Conds:RS_35._col0=RS_36._col0(Inner),Output:["_col1","_col3"]
+ Merge Join Operator [MERGEJOIN_129] (rows=316788826 width=135)
+ Conds:RS_34._col0=RS_35._col0(Inner),Output:["_col1","_col3"]
<-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_36]
+ SHUFFLE [RS_35]
PartitionCols:_col0
Please refer to the previous Select Operator [SEL_5]
<-Map 18 [SIMPLE_EDGE]
- SHUFFLE [RS_35]
+ SHUFFLE [RS_34]
PartitionCols:_col0
- Select Operator [SEL_28] (rows=287989836 width=135)
+ Select Operator [SEL_27] (rows=287989836 width=135)
Output:["_col0","_col1"]
- Filter Operator [FIL_123] (rows=287989836 width=135)
+ Filter Operator [FIL_120] (rows=287989836 width=135)
predicate:(cs_bill_customer_sk is not null and cs_sold_date_sk is not null)
- TableScan [TS_26] (rows=287989836 width=135)
+ TableScan [TS_25] (rows=287989836 width=135)
default@catalog_sales,catalog_sales,Tbl:COMPLETE,Col:NONE,Output:["cs_sold_date_sk","cs_bill_customer_sk"]
<-Reducer 4 [CONTAINS]
- Reduce Output Operator [RS_56]
+ Reduce Output Operator [RS_54]
PartitionCols:_col0, _col1, _col2
- Group By Operator [GBY_55] (rows=261355616 width=103)
+ Group By Operator [GBY_53] (rows=261355616 width=103)
Output:["_col0","_col1","_col2","_col3","_col4"],aggregations:["sum(_col3)","sum(_col4)"],keys:_col0, _col1, _col2
- Select Operator [SEL_53] (rows=261355616 width=103)
+ Select Operator [SEL_51] (rows=261355616 width=103)
Output:["_col0","_col1","_col2","_col3","_col4"]
Select Operator [SEL_24] (rows=174238687 width=88)
Output:["_col0","_col1","_col2","_col3","_col4"]
@@ -212,7 +212,7 @@ Stage-0
PartitionCols:_col0, _col1, _col2
Group By Operator [GBY_16] (rows=696954748 width=88)
Output:["_col0","_col1","_col2"],keys:_col7, _col6, _col3
- Merge Join Operator [MERGEJOIN_131] (rows=696954748 width=88)
+ Merge Join Operator [MERGEJOIN_128] (rows=696954748 width=88)
Conds:RS_12._col1=RS_13._col0(Inner),Output:["_col3","_col6","_col7"]
<-Map 17 [SIMPLE_EDGE]
SHUFFLE [RS_13]
@@ -221,7 +221,7 @@ Stage-0
<-Reducer 2 [SIMPLE_EDGE]
SHUFFLE [RS_12]
PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_130] (rows=633595212 width=88)
+ Merge Join Operator [MERGEJOIN_127] (rows=633595212 width=88)
Conds:RS_9._col0=RS_10._col0(Inner),Output:["_col1","_col3"]
<-Map 10 [SIMPLE_EDGE]
SHUFFLE [RS_10]
@@ -232,7 +232,7 @@ Stage-0
PartitionCols:_col0
Select Operator [SEL_2] (rows=575995635 width=88)
Output:["_col0","_col1"]
- Filter Operator [FIL_120] (rows=575995635 width=88)
+ Filter Operator [FIL_117] (rows=575995635 width=88)
predicate:(ss_customer_sk is not null and ss_sold_date_sk is not null)
TableScan [TS_0] (rows=575995635 width=88)
default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_customer_sk"]
http://git-wip-us.apache.org/repos/asf/hive/blob/1b01f9c4/ql/src/test/results/clientpositive/perf/tez/query89.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/tez/query89.q.out b/ql/src/test/results/clientpositive/perf/tez/query89.q.out
index 1beb703..8d3f6c8 100644
--- a/ql/src/test/results/clientpositive/perf/tez/query89.q.out
+++ b/ql/src/test/results/clientpositive/perf/tez/query89.q.out
@@ -81,67 +81,65 @@ Stage-0
Select Operator [SEL_29] (rows=383325119 width=88)
Output:["avg_window_0","_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
PTF Operator [PTF_28] (rows=383325119 width=88)
- Function definitions:[{},{"name:":"windowingtablefunction","order by:":"_col0 ASC NULLS FIRST, _col2 ASC NULLS FIRST, _col3 ASC NULLS FIRST, _col4 ASC NULLS FIRST","partition by:":"_col0, _col2, _col3, _col4"}]
+ Function definitions:[{},{"name:":"windowingtablefunction","order by:":"_col2 ASC NULLS FIRST, _col0 ASC NULLS FIRST, _col4 ASC NULLS FIRST, _col5 ASC NULLS FIRST","partition by:":"_col2, _col0, _col4, _col5"}]
Select Operator [SEL_27] (rows=383325119 width=88)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
<-Reducer 5 [SIMPLE_EDGE]
SHUFFLE [RS_26]
- PartitionCols:_col0, _col2, _col3, _col4
- Select Operator [SEL_25] (rows=383325119 width=88)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
- Group By Operator [GBY_24] (rows=383325119 width=88)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4, KEY._col5
- <-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_23]
- PartitionCols:_col0, _col1, _col2, _col3, _col4, _col5
- Group By Operator [GBY_22] (rows=766650239 width=88)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["sum(_col3)"],keys:_col5, _col6, _col7, _col10, _col12, _col13
- Merge Join Operator [MERGEJOIN_54] (rows=766650239 width=88)
- Conds:RS_18._col2=RS_19._col0(Inner),Output:["_col3","_col5","_col6","_col7","_col10","_col12","_col13"]
- <-Map 10 [SIMPLE_EDGE]
- SHUFFLE [RS_19]
- PartitionCols:_col0
- Select Operator [SEL_11] (rows=1704 width=1910)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_51] (rows=1704 width=1910)
- predicate:s_store_sk is not null
- TableScan [TS_9] (rows=1704 width=1910)
- default@store,store,Tbl:COMPLETE,Col:NONE,Output:["s_store_sk","s_store_name","s_company_name"]
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_18]
- PartitionCols:_col2
- Merge Join Operator [MERGEJOIN_53] (rows=696954748 width=88)
- Conds:RS_15._col0=RS_16._col0(Inner),Output:["_col2","_col3","_col5","_col6","_col7","_col10"]
- <-Map 9 [SIMPLE_EDGE]
- SHUFFLE [RS_16]
- PartitionCols:_col0
- Select Operator [SEL_8] (rows=36524 width=1119)
- Output:["_col0","_col2"]
- Filter Operator [FIL_50] (rows=36524 width=1119)
- predicate:((d_year = 2000) and d_date_sk is not null)
- TableScan [TS_6] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
- <-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_15]
- PartitionCols:_col0
- Merge Join Operator [MERGEJOIN_52] (rows=633595212 width=88)
- Conds:RS_12._col1=RS_13._col0(Inner),Output:["_col0","_col2","_col3","_col5","_col6","_col7"]
- <-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_12]
- PartitionCols:_col1
- Select Operator [SEL_2] (rows=575995635 width=88)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_48] (rows=575995635 width=88)
- predicate:(ss_item_sk is not null and ss_sold_date_sk is not null and ss_store_sk is not null)
- TableScan [TS_0] (rows=575995635 width=88)
- default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_item_sk","ss_store_sk","ss_sales_price"]
- <-Map 8 [SIMPLE_EDGE]
- SHUFFLE [RS_13]
- PartitionCols:_col0
- Select Operator [SEL_5] (rows=231000 width=1436)
- Output:["_col0","_col1","_col2","_col3"]
- Filter Operator [FIL_49] (rows=231000 width=1436)
- predicate:((((i_category) IN ('Home', 'Books', 'Electronics') and (i_class) IN ('wallpaper', 'parenting', 'musical')) or ((i_category) IN ('Shoes', 'Jewelry', 'Men') and (i_class) IN ('womens', 'birdal', 'pants'))) and ((i_category) IN ('Home', 'Books', 'Electronics') or (i_category) IN ('Shoes', 'Jewelry', 'Men')) and ((i_class) IN ('wallpaper', 'parenting', 'musical') or (i_class) IN ('womens', 'birdal', 'pants')) and i_item_sk is not null)
- TableScan [TS_3] (rows=462000 width=1436)
- default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_brand","i_class","i_category"]
+ PartitionCols:_col2, _col0, _col4, _col5
+ Group By Operator [GBY_24] (rows=383325119 width=88)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4, KEY._col5
+ <-Reducer 4 [SIMPLE_EDGE]
+ SHUFFLE [RS_23]
+ PartitionCols:_col0, _col1, _col2, _col3, _col4, _col5
+ Group By Operator [GBY_22] (rows=766650239 width=88)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"],aggregations:["sum(_col3)"],keys:_col5, _col6, _col7, _col10, _col12, _col13
+ Merge Join Operator [MERGEJOIN_54] (rows=766650239 width=88)
+ Conds:RS_18._col2=RS_19._col0(Inner),Output:["_col3","_col5","_col6","_col7","_col10","_col12","_col13"]
+ <-Map 10 [SIMPLE_EDGE]
+ SHUFFLE [RS_19]
+ PartitionCols:_col0
+ Select Operator [SEL_11] (rows=1704 width=1910)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_51] (rows=1704 width=1910)
+ predicate:s_store_sk is not null
+ TableScan [TS_9] (rows=1704 width=1910)
+ default@store,store,Tbl:COMPLETE,Col:NONE,Output:["s_store_sk","s_store_name","s_company_name"]
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_18]
+ PartitionCols:_col2
+ Merge Join Operator [MERGEJOIN_53] (rows=696954748 width=88)
+ Conds:RS_15._col0=RS_16._col0(Inner),Output:["_col2","_col3","_col5","_col6","_col7","_col10"]
+ <-Map 9 [SIMPLE_EDGE]
+ SHUFFLE [RS_16]
+ PartitionCols:_col0
+ Select Operator [SEL_8] (rows=36524 width=1119)
+ Output:["_col0","_col2"]
+ Filter Operator [FIL_50] (rows=36524 width=1119)
+ predicate:((d_year = 2000) and d_date_sk is not null)
+ TableScan [TS_6] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_year","d_moy"]
+ <-Reducer 2 [SIMPLE_EDGE]
+ SHUFFLE [RS_15]
+ PartitionCols:_col0
+ Merge Join Operator [MERGEJOIN_52] (rows=633595212 width=88)
+ Conds:RS_12._col1=RS_13._col0(Inner),Output:["_col0","_col2","_col3","_col5","_col6","_col7"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_12]
+ PartitionCols:_col1
+ Select Operator [SEL_2] (rows=575995635 width=88)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_48] (rows=575995635 width=88)
+ predicate:(ss_item_sk is not null and ss_sold_date_sk is not null and ss_store_sk is not null)
+ TableScan [TS_0] (rows=575995635 width=88)
+ default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_item_sk","ss_store_sk","ss_sales_price"]
+ <-Map 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_13]
+ PartitionCols:_col0
+ Select Operator [SEL_5] (rows=231000 width=1436)
+ Output:["_col0","_col1","_col2","_col3"]
+ Filter Operator [FIL_49] (rows=231000 width=1436)
+ predicate:((((i_category) IN ('Home', 'Books', 'Electronics') and (i_class) IN ('wallpaper', 'parenting', 'musical')) or ((i_category) IN ('Shoes', 'Jewelry', 'Men') and (i_class) IN ('womens', 'birdal', 'pants'))) and ((i_category) IN ('Home', 'Books', 'Electronics') or (i_category) IN ('Shoes', 'Jewelry', 'Men')) and ((i_class) IN ('wallpaper', 'parenting', 'musical') or (i_class) IN ('womens', 'birdal', 'pants')) and i_item_sk is not null)
+ TableScan [TS_3] (rows=462000 width=1436)
+ default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_brand","i_class","i_category"]
http://git-wip-us.apache.org/repos/asf/hive/blob/1b01f9c4/ql/src/test/results/clientpositive/perf/tez/query98.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/perf/tez/query98.q.out b/ql/src/test/results/clientpositive/perf/tez/query98.q.out
index 19bb49c..e403384 100644
--- a/ql/src/test/results/clientpositive/perf/tez/query98.q.out
+++ b/ql/src/test/results/clientpositive/perf/tez/query98.q.out
@@ -74,61 +74,59 @@ Stage-0
limit:-1
Stage-1
Reducer 6
- File Output Operator [FS_28]
- Select Operator [SEL_26] (rows=348477374 width=88)
+ File Output Operator [FS_27]
+ Select Operator [SEL_25] (rows=348477374 width=88)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
<-Reducer 5 [SIMPLE_EDGE]
- SHUFFLE [RS_25]
- Select Operator [SEL_23] (rows=348477374 width=88)
+ SHUFFLE [RS_24]
+ Select Operator [SEL_22] (rows=348477374 width=88)
Output:["_col0","_col1","_col2","_col3","_col4","_col5","_col6"]
- PTF Operator [PTF_22] (rows=348477374 width=88)
- Function definitions:[{},{"name:":"windowingtablefunction","order by:":"_col3 ASC NULLS FIRST","partition by:":"_col3"}]
- Select Operator [SEL_21] (rows=348477374 width=88)
+ PTF Operator [PTF_21] (rows=348477374 width=88)
+ Function definitions:[{},{"name:":"windowingtablefunction","order by:":"_col1 ASC NULLS FIRST","partition by:":"_col1"}]
+ Select Operator [SEL_20] (rows=348477374 width=88)
Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
<-Reducer 4 [SIMPLE_EDGE]
- SHUFFLE [RS_20]
- PartitionCols:_col3
- Select Operator [SEL_19] (rows=348477374 width=88)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
- Group By Operator [GBY_18] (rows=348477374 width=88)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4
- <-Reducer 3 [SIMPLE_EDGE]
- SHUFFLE [RS_17]
- PartitionCols:_col0, _col1, _col2, _col3, _col4
- Group By Operator [GBY_16] (rows=696954748 width=88)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["sum(_col2)"],keys:_col10, _col9, _col6, _col7, _col8
- Merge Join Operator [MERGEJOIN_38] (rows=696954748 width=88)
- Conds:RS_12._col1=RS_13._col0(Inner),Output:["_col2","_col6","_col7","_col8","_col9","_col10"]
- <-Map 8 [SIMPLE_EDGE]
- SHUFFLE [RS_13]
- PartitionCols:_col0
- Select Operator [SEL_8] (rows=231000 width=1436)
- Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
- Filter Operator [FIL_36] (rows=231000 width=1436)
- predicate:((i_category) IN ('Jewelry', 'Sports', 'Books') and i_item_sk is not null)
- TableScan [TS_6] (rows=462000 width=1436)
- default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_id","i_item_desc","i_current_price","i_class","i_category"]
- <-Reducer 2 [SIMPLE_EDGE]
- SHUFFLE [RS_12]
- PartitionCols:_col1
- Merge Join Operator [MERGEJOIN_37] (rows=633595212 width=88)
- Conds:RS_9._col0=RS_10._col0(Inner),Output:["_col1","_col2"]
- <-Map 1 [SIMPLE_EDGE]
- SHUFFLE [RS_9]
- PartitionCols:_col0
- Select Operator [SEL_2] (rows=575995635 width=88)
- Output:["_col0","_col1","_col2"]
- Filter Operator [FIL_34] (rows=575995635 width=88)
- predicate:(ss_item_sk is not null and ss_sold_date_sk is not null)
- TableScan [TS_0] (rows=575995635 width=88)
- default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_item_sk","ss_ext_sales_price"]
- <-Map 7 [SIMPLE_EDGE]
- SHUFFLE [RS_10]
- PartitionCols:_col0
- Select Operator [SEL_5] (rows=8116 width=1119)
- Output:["_col0"]
- Filter Operator [FIL_35] (rows=8116 width=1119)
- predicate:(CAST( d_date AS TIMESTAMP) BETWEEN TIMESTAMP'2001-01-12 00:00:00.0' AND TIMESTAMP'2001-02-11 00:00:00.0' and d_date_sk is not null)
- TableScan [TS_3] (rows=73049 width=1119)
- default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_date"]
+ SHUFFLE [RS_19]
+ PartitionCols:_col1
+ Group By Operator [GBY_18] (rows=348477374 width=88)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["sum(VALUE._col0)"],keys:KEY._col0, KEY._col1, KEY._col2, KEY._col3, KEY._col4
+ <-Reducer 3 [SIMPLE_EDGE]
+ SHUFFLE [RS_17]
+ PartitionCols:_col0, _col1, _col2, _col3, _col4
+ Group By Operator [GBY_16] (rows=696954748 width=88)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5"],aggregations:["sum(_col2)"],keys:_col10, _col9, _col6, _col7, _col8
+ Merge Join Operator [MERGEJOIN_37] (rows=696954748 width=88)
+ Conds:RS_12._col1=RS_13._col0(Inner),Output:["_col2","_col6","_col7","_col8","_col9","_col10"]
+ <-Map 8 [SIMPLE_EDGE]
+ SHUFFLE [RS_13]
+ PartitionCols:_col0
+ Select Operator [SEL_8] (rows=231000 width=1436)
+ Output:["_col0","_col1","_col2","_col3","_col4","_col5"]
+ Filter Operator [FIL_35] (rows=231000 width=1436)
+ predicate:((i_category) IN ('Jewelry', 'Sports', 'Books') and i_item_sk is not null)
+ TableScan [TS_6] (rows=462000 width=1436)
+ default@item,item,Tbl:COMPLETE,Col:NONE,Output:["i_item_sk","i_item_id","i_item_desc","i_current_price","i_class","i_category"]
+ <-Reducer 2 [SIMPLE_EDGE]
+ SHUFFLE [RS_12]
+ PartitionCols:_col1
+ Merge Join Operator [MERGEJOIN_36] (rows=633595212 width=88)
+ Conds:RS_9._col0=RS_10._col0(Inner),Output:["_col1","_col2"]
+ <-Map 1 [SIMPLE_EDGE]
+ SHUFFLE [RS_9]
+ PartitionCols:_col0
+ Select Operator [SEL_2] (rows=575995635 width=88)
+ Output:["_col0","_col1","_col2"]
+ Filter Operator [FIL_33] (rows=575995635 width=88)
+ predicate:(ss_item_sk is not null and ss_sold_date_sk is not null)
+ TableScan [TS_0] (rows=575995635 width=88)
+ default@store_sales,store_sales,Tbl:COMPLETE,Col:NONE,Output:["ss_sold_date_sk","ss_item_sk","ss_ext_sales_price"]
+ <-Map 7 [SIMPLE_EDGE]
+ SHUFFLE [RS_10]
+ PartitionCols:_col0
+ Select Operator [SEL_5] (rows=8116 width=1119)
+ Output:["_col0"]
+ Filter Operator [FIL_34] (rows=8116 width=1119)
+ predicate:(CAST( d_date AS TIMESTAMP) BETWEEN TIMESTAMP'2001-01-12 00:00:00.0' AND TIMESTAMP'2001-02-11 00:00:00.0' and d_date_sk is not null)
+ TableScan [TS_3] (rows=73049 width=1119)
+ default@date_dim,date_dim,Tbl:COMPLETE,Col:NONE,Output:["d_date_sk","d_date"]
http://git-wip-us.apache.org/repos/asf/hive/blob/1b01f9c4/ql/src/test/results/clientpositive/reduce_deduplicate_extended2.q.out
----------------------------------------------------------------------
diff --git a/ql/src/test/results/clientpositive/reduce_deduplicate_extended2.q.out b/ql/src/test/results/clientpositive/reduce_deduplicate_extended2.q.out
index 9102bd4..7584d84 100644
--- a/ql/src/test/results/clientpositive/reduce_deduplicate_extended2.q.out
+++ b/ql/src/test/results/clientpositive/reduce_deduplicate_extended2.q.out
@@ -723,9 +723,8 @@ STAGE DEPENDENCIES:
Stage-3 depends on stages: Stage-2
Stage-4 depends on stages: Stage-3, Stage-8
Stage-6 is a root stage
- Stage-7 depends on stages: Stage-6, Stage-9
+ Stage-7 depends on stages: Stage-6
Stage-8 depends on stages: Stage-7
- Stage-9 is a root stage
Stage-0 depends on stages: Stage-4
STAGE PLANS:
@@ -846,26 +845,26 @@ STAGE PLANS:
Map Operator Tree:
TableScan
Union
- Statistics: Num rows: 412 Data size: 4376 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: string)
sort order: --
- Statistics: Num rows: 412 Data size: 4376 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE
TableScan
Union
- Statistics: Num rows: 412 Data size: 4376 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: _col1 (type: string), _col0 (type: string)
sort order: --
- Statistics: Num rows: 412 Data size: 4376 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Select Operator
expressions: KEY.reducesinkkey1 (type: string), KEY.reducesinkkey0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 412 Data size: 4376 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
- Statistics: Num rows: 412 Data size: 4376 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5842 Basic stats: COMPLETE Column stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
@@ -917,25 +916,39 @@ STAGE PLANS:
Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
TableScan
- Reduce Output Operator
- key expressions: _col1 (type: string), _col0 (type: string)
- sort order: ++
- Map-reduce partition columns: _col1 (type: string), _col0 (type: string)
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ alias: src
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Filter Operator
+ predicate: (key is not null and value is not null) (type: boolean)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Select Operator
+ expressions: value (type: string), key (type: string)
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Group By Operator
+ keys: _col0 (type: string), _col1 (type: string)
+ mode: hash
+ outputColumnNames: _col0, _col1
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Reduce Output Operator
+ key expressions: _col0 (type: string), _col1 (type: string)
+ sort order: ++
+ Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
+ Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
- Inner Join 0 to 1
+ Left Semi Join 0 to 1
keys:
0 _col1 (type: string), _col0 (type: string)
- 1 _col1 (type: string), _col0 (type: string)
+ 1 _col0 (type: string), _col1 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
Group By Operator
keys: _col1 (type: string), _col0 (type: string)
mode: hash
outputColumnNames: _col0, _col1
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table:
@@ -951,53 +964,17 @@ STAGE PLANS:
key expressions: _col0 (type: string), _col1 (type: string)
sort order: ++
Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
- Reduce Operator Tree:
- Group By Operator
- keys: KEY._col0 (type: string), KEY._col1 (type: string)
- mode: mergepartial
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
- Select Operator
- expressions: _col1 (type: string), _col0 (type: string)
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 137 Data size: 1455 Basic stats: COMPLETE Column stats: NONE
- File Output Operator
- compressed: false
- table:
- input format: org.apache.hadoop.mapred.SequenceFileInputFormat
- output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
- serde: org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe
-
- Stage: Stage-9
- Map Reduce
- Map Operator Tree:
- TableScan
- alias: src
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Filter Operator
- predicate: (key is not null and value is not null) (type: boolean)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Group By Operator
- keys: value (type: string), key (type: string)
- mode: hash
- outputColumnNames: _col0, _col1
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
- Reduce Output Operator
- key expressions: _col0 (type: string), _col1 (type: string)
- sort order: ++
- Map-reduce partition columns: _col0 (type: string), _col1 (type: string)
- Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 550 Data size: 5843 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string), KEY._col1 (type: string)
mode: mergepartial
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
Select Operator
expressions: _col1 (type: string), _col0 (type: string)
outputColumnNames: _col0, _col1
- Statistics: Num rows: 250 Data size: 2656 Basic stats: COMPLETE Column stats: NONE
+ Statistics: Num rows: 275 Data size: 2921 Basic stats: COMPLETE Column stats: NONE
File Output Operator
compressed: false
table: