You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by pengcheng xiong <px...@hortonworks.com> on 2015/11/23 06:59:49 UTC
Review Request 40589: Implement limit push down through union all in
CBO
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
HIVE-11775
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java c005b1a
ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68
ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4
ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64
ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486
ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff
ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d
ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
ql/src/test/results/clientpositive/merge4.q.out 031376f
ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4
ql/src/test/results/clientpositive/parquet_ppd_char.q.out defaa9d
ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9
ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e
ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7
ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80
ql/src/test/results/clientpositive/spark/temp_table.q.out 718a8a4
ql/src/test/results/clientpositive/temp_table.q.out a9f2bae
ql/src/test/results/clientpositive/tez/temp_table.q.out 200ccdd
Diff: https://reviews.apache.org/r/40589/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review108698
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 110)
<https://reviews.apache.org/r/40589/#comment168155>
what if the previous limit got pushed past project/join due to limitjointranspose rules.
Then this will fire in a loop.
Limit-Union-Proj-Limit
- John Pullokkaran
On Nov. 23, 2015, 5:59 a.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 5:59 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java c005b1a
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68
> ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4
> ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64
> ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486
> ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff
> ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
> ql/src/test/results/clientpositive/merge4.q.out 031376f
> ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4
> ql/src/test/results/clientpositive/parquet_ppd_char.q.out defaa9d
> ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9
> ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e
> ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7
> ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80
> ql/src/test/results/clientpositive/spark/temp_table.q.out 718a8a4
> ql/src/test/results/clientpositive/temp_table.q.out a9f2bae
> ql/src/test/results/clientpositive/tez/temp_table.q.out 200ccdd
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111079
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 56)
<https://reviews.apache.org/r/40589/#comment171053>
Nit Pick: extra space
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 81)
<https://reviews.apache.org/r/40589/#comment171054>
This should really be
if ((RexLiteral.intValue(sort.offset) + RexLiteral.intValue(sort.fetch)) < RelMetadataQuery.getRowCount(input)) {
}
- John Pullokkaran
On Dec. 17, 2015, 10:09 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 10:09 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111311
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 84)
<https://reviews.apache.org/r/40589/#comment171432>
The real value of pushing limit past union is to push it further down (by pushing it past sel, join).
If the sortrel ended up just below union & if sortrel also had sort in it (i.e if union's input is synthetic limit) then i think we should remove it.
May be we should enhance HiveSortRemoveRule to look for pattern SortRel-union-SortRel and remove synthetic SortRel below union if it has sort in it.
Otherwise we end up paying double sorting cost.
- John Pullokkaran
On Dec. 17, 2015, 10:47 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 10:47 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111083
-----------------------------------------------------------
Ship it!
Ship It!
- John Pullokkaran
On Dec. 17, 2015, 10:47 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2015, 10:47 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/
-----------------------------------------------------------
(Updated Dec. 17, 2015, 10:47 p.m.)
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
HIVE-11775
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/40589/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/
-----------------------------------------------------------
(Updated Dec. 17, 2015, 10:09 p.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
address John's comments
Repository: hive-git
Description
-------
HIVE-11775
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/40589/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111019
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 63)
<https://reviews.apache.org/r/40589/#comment171015>
Same as above
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 103)
<https://reviews.apache.org/r/40589/#comment171017>
May be rename the flag that signifies its use
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 106)
<https://reviews.apache.org/r/40589/#comment171019>
Shouldn't this be < instead of >=.
- John Pullokkaran
On Dec. 16, 2015, 6:46 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2015, 6:46 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by pengcheng xiong <px...@hortonworks.com>.
> On Dec. 17, 2015, 7:36 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java, line 94
> > <https://reviews.apache.org/r/40589/diff/4/?file=1166681#file1166681line94>
> >
> > In case of pure oder by fetch will be null.
> > Wouldn't "RexLiteral.intValue(sort.fetch)" result in an NPE?
we check sort.fetch != null in L92
- pengcheng
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111018
-----------------------------------------------------------
On Dec. 16, 2015, 6:46 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2015, 6:46 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review111018
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 57)
<https://reviews.apache.org/r/40589/#comment171014>
This is not needed for our Parallel execution env as there is no guarantee that set ops will preserve order.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java (line 94)
<https://reviews.apache.org/r/40589/#comment171013>
In case of pure oder by fetch will be null.
Wouldn't "RexLiteral.intValue(sort.fetch)" result in an NPE?
- John Pullokkaran
On Dec. 16, 2015, 6:46 p.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2015, 6:46 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/
-----------------------------------------------------------
(Updated Dec. 16, 2015, 6:46 p.m.)
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
HIVE-11775
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 243f281
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/40589/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/
-----------------------------------------------------------
(Updated Dec. 15, 2015, 7:21 a.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
address John's comments.
Repository: hive-git
Description
-------
HIVE-11775
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 31f0634
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 36a12bf
ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68
ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4
ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64
ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486
ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff
ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/constprog_dpp.q.out a1addb7
ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4
ql/src/test/results/clientpositive/parquet_ppd_char.q.out c1565f8
ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9
ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e
ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7
ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80
ql/src/test/results/clientpositive/spark/temp_table.q.out 119d26d
Diff: https://reviews.apache.org/r/40589/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review110361
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1071)
<https://reviews.apache.org/r/40589/#comment170171>
This should be added to previous hepplan invocation so that limit can be pushed as low as possible.
Ex: Limit-Sel-Union-Join-Proj-TS1
TS2
- John Pullokkaran
On Dec. 14, 2015, 6:59 a.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 6:59 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56a39df
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java a0d14dc
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
> ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68
> ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4
> ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64
> ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486
> ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff
> ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/constprog_dpp.q.out a1addb7
> ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4
> ql/src/test/results/clientpositive/parquet_ppd_char.q.out c1565f8
> ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9
> ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e
> ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7
> ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80
> ql/src/test/results/clientpositive/spark/temp_table.q.out 119d26d
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by pengcheng xiong <px...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 6:59 a.m.)
Review request for hive and Ashutosh Chauhan.
Repository: hive-git
Description
-------
HIVE-11775
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56a39df
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java a0d14dc
ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
ql/src/test/queries/clientpositive/limit_join_transpose.q 80430c6
ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68
ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4
ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64
ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486
ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff
ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q cecbbd7
ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/constprog_dpp.q.out a1addb7
ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4
ql/src/test/results/clientpositive/parquet_ppd_char.q.out c1565f8
ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9
ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e
ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7
ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80
ql/src/test/results/clientpositive/spark/temp_table.q.out 119d26d
Diff: https://reviews.apache.org/r/40589/diff/
Testing
-------
Thanks,
pengcheng xiong
Re: Review Request 40589: Implement limit push down through union all
in CBO
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40589/#review108692
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 1090)
<https://reviews.apache.org/r/40589/#comment168147>
This should be conditionally turned off.
Limit push through Join is protected by "hive.optimize.limitjointranspose". Rename this to "hive.optimize.limitpushdown" or something similar.
You should then add this rule to where limit push through join is.
- John Pullokkaran
On Nov. 23, 2015, 5:59 a.m., pengcheng xiong wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40589/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 5:59 a.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-11775
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortUnionReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java c005b1a
> ql/src/test/queries/clientpositive/cbo_SortUnionTransposeRule.q PRE-CREATION
> ql/src/test/queries/clientpositive/parquet_ppd_boolean.q 059da68
> ql/src/test/queries/clientpositive/parquet_ppd_char.q eaddcb4
> ql/src/test/queries/clientpositive/parquet_ppd_date.q 41d0d64
> ql/src/test/queries/clientpositive/parquet_ppd_decimal.q dfca486
> ql/src/test/queries/clientpositive/parquet_ppd_timestamp.q 1b9f6ff
> ql/src/test/queries/clientpositive/parquet_ppd_varchar.q 6449c6d
> ql/src/test/results/clientpositive/cbo_SortUnionTransposeRule.q.out PRE-CREATION
> ql/src/test/results/clientpositive/merge4.q.out 031376f
> ql/src/test/results/clientpositive/parquet_ppd_boolean.q.out 6e62ee4
> ql/src/test/results/clientpositive/parquet_ppd_char.q.out defaa9d
> ql/src/test/results/clientpositive/parquet_ppd_date.q.out 55231e9
> ql/src/test/results/clientpositive/parquet_ppd_decimal.q.out a30820e
> ql/src/test/results/clientpositive/parquet_ppd_timestamp.q.out 3b3e5b7
> ql/src/test/results/clientpositive/parquet_ppd_varchar.q.out 5a62e80
> ql/src/test/results/clientpositive/spark/temp_table.q.out 718a8a4
> ql/src/test/results/clientpositive/temp_table.q.out a9f2bae
> ql/src/test/results/clientpositive/tez/temp_table.q.out 200ccdd
>
> Diff: https://reviews.apache.org/r/40589/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> pengcheng xiong
>
>