You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jesús Camacho Rodríguez <jc...@hortonworks.com> on 2015/09/09 10:34:09 UTC
Review Request 38211: HIVE-11684
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 8a00079631b657acd79877202e9819ff282dfdd6
itests/src/test/resources/testconfiguration.properties bed621d3eb74f01e54110552f68538afd228018d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 73ae7c4ef90af22847bef9e54863d4651d9c1013
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review100456
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java (line 172)
<https://reviews.apache.org/r/38211/#comment157655>
We should seperate Limit push through Project from Limit Push through Join. Looking for Limit/Proj/Join is a hard coupling.
- John Pullokkaran
On Sept. 17, 2015, 1:55 p.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2015, 1:55 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da29f861c0e6c11065a95092d9e7e7b08aba
> itests/src/test/resources/testconfiguration.properties 0d3e1cc17df496e8b4dd8c3622017cd15384348a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 0a7ce3a5f81f3175cddc59319753be3bb0b2af04
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review101851
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java (line 80)
<https://reviews.apache.org/r/38211/#comment159364>
"reduction" as it is set is not a %.
Shouldn't you multiply by ((100-reduction)/100)
guarding for reduction to be <= 100.
- John Pullokkaran
On Sept. 28, 2015, 11:40 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2015, 11:40 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86
> itests/src/test/resources/testconfiguration.properties 4f7b25f2563683933b9d677626f0447705927966
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8c5e58570570812b525b8bf0edb0f4c0a9
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review101952
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java (line 4)
<https://reviews.apache.org/r/38211/#comment159463>
As we discussed:
we should seperate Limit push down from constraining condition.
I.e first push limit as low as possible then pull it up in the subsequent phase if it doesn't meet reduction criteria.
Also the reduction criteria probably needs to specify the minimun no of tuples to reduce for limit to be cost effective (this should go away once we have execution cost model).
- John Pullokkaran
On Sept. 28, 2015, 11:40 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Sept. 28, 2015, 11:40 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86
> itests/src/test/resources/testconfiguration.properties 4f7b25f2563683933b9d677626f0447705927966
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8c5e58570570812b525b8bf0edb0f4c0a9
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103668
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java (line 39)
<https://reviews.apache.org/r/38211/#comment161734>
We should enhance Join cardinality estimation for outer joins.
Outer Join cardinality should be: max(outer side cardinality, inner join cardinality)
- John Pullokkaran
On Oct. 11, 2015, 12:57 p.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2015, 12:57 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bf48f69dd8914020dcd4eeba8dd448f4ee8d5b24
> itests/src/test/resources/testconfiguration.properties ad47fac2e3e5716f2cdf4b9c25ee2d6d14eab5ec
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 6948fb16c6a3c9ce456a900536bee4c0d9b1614e
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 61ee2bdf21000aa20d9fff02a3d926a1031410bc
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103667
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 997)
<https://reviews.apache.org/r/38211/#comment161733>
Redundant Limit removal should happen bottom up.
- John Pullokkaran
On Oct. 11, 2015, 12:57 p.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Oct. 11, 2015, 12:57 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bf48f69dd8914020dcd4eeba8dd448f4ee8d5b24
> itests/src/test/resources/testconfiguration.properties ad47fac2e3e5716f2cdf4b9c25ee2d6d14eab5ec
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 6948fb16c6a3c9ce456a900536bee4c0d9b1614e
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 61ee2bdf21000aa20d9fff02a3d926a1031410bc
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review107412
-----------------------------------------------------------
Ship it!
Ship It!
- John Pullokkaran
On Nov. 18, 2015, 10:40 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Nov. 18, 2015, 10:40 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 953e52c4de49bbc083647033b370a1ca80f5cb97
> itests/src/test/resources/testconfiguration.properties 402914cdfb4d72998a9c219e812d561e67612897
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54a2c6cfd9bc4413ebf7f715e54c61b966f
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/join32_lessSize.q.out 5ea40241d329617220bbe775dcba91a9d7fb2f89
> ql/src/test/results/clientpositive/join_merging.q.out b42bdc5ee92e9d725b7c0b8323c7763a55f3d059
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 937e8fcb0c7b89c13548a01e182176102ebcb681
> ql/src/test/results/clientpositive/spark/join_merging.q.out 6c5089b4868e839a2e72546a39a7a5d4741464ed
> ql/src/test/results/clientpositive/subquery_notin.q.out 5563794f9761a8269a56c7e716545c5edb1e4076
> ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 9689ae37dfedc8c3becd6915e1e4cd5a5da29f9b
> ql/src/test/results/clientpositive/tez/explainuser_1.q.out fa29dfec8fe38fcbec2cfa22859821898e07dbaa
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Nov. 18, 2015, 10:40 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 953e52c4de49bbc083647033b370a1ca80f5cb97
itests/src/test/resources/testconfiguration.properties 402914cdfb4d72998a9c219e812d561e67612897
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54a2c6cfd9bc4413ebf7f715e54c61b966f
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/join32_lessSize.q.out 5ea40241d329617220bbe775dcba91a9d7fb2f89
ql/src/test/results/clientpositive/join_merging.q.out b42bdc5ee92e9d725b7c0b8323c7763a55f3d059
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 937e8fcb0c7b89c13548a01e182176102ebcb681
ql/src/test/results/clientpositive/spark/join_merging.q.out 6c5089b4868e839a2e72546a39a7a5d4741464ed
ql/src/test/results/clientpositive/subquery_notin.q.out 5563794f9761a8269a56c7e716545c5edb1e4076
ql/src/test/results/clientpositive/subquery_notin_having.q.java1.7.out 9689ae37dfedc8c3becd6915e1e4cd5a5da29f9b
ql/src/test/results/clientpositive/tez/explainuser_1.q.out fa29dfec8fe38fcbec2cfa22859821898e07dbaa
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review106333
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java (line 60)
<https://reviews.apache.org/r/38211/#comment165065>
You don't have to add collation to traitdef; HiveSortLimit does this for you automatically.
Also the traits of sortnode from above should not be used; theoretically we should use the trait from project input. Practically Once sort is introduced only ordering trait can survive.
- John Pullokkaran
On Nov. 12, 2015, 10:47 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2015, 10:47 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7a8517b166185f5c501aceacc780dfab21de1c0a
> itests/src/test/resources/testconfiguration.properties 70f96da40c3c5b74e3ad2dd640321a2967cfcb86
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 90c2067e8279808fc0040e16d99a801b5d167eec
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54a2c6cfd9bc4413ebf7f715e54c61b966f
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review106415
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java (line 60)
<https://reviews.apache.org/r/38211/#comment165155>
You are right, I need to replace
sort.getTraitSet().replace(newCollation)
by
newProject.getTraitSet()
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java (line 56)
<https://reviews.apache.org/r/38211/#comment165153>
Top HiveSortLimit operator never sorts, it can only be a limit (this is checked in the matches method). Thus, we take the collation from the bottom operator.
Currently it does not seem possible to merge two SortLimit operators if both have Sort and Limit operations (or am I missing something?)
- Jesús Camacho Rodríguez
On Nov. 12, 2015, 10:47 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2015, 10:47 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7a8517b166185f5c501aceacc780dfab21de1c0a
> itests/src/test/resources/testconfiguration.properties 70f96da40c3c5b74e3ad2dd640321a2967cfcb86
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 90c2067e8279808fc0040e16d99a801b5d167eec
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54a2c6cfd9bc4413ebf7f715e54c61b966f
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review106332
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java (line 56)
<https://reviews.apache.org/r/38211/#comment165064>
Don't you need to merge the collation from top?
Otherwise you will top OB.
- John Pullokkaran
On Nov. 12, 2015, 10:47 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2015, 10:47 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7a8517b166185f5c501aceacc780dfab21de1c0a
> itests/src/test/resources/testconfiguration.properties 70f96da40c3c5b74e3ad2dd640321a2967cfcb86
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 90c2067e8279808fc0040e16d99a801b5d167eec
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54a2c6cfd9bc4413ebf7f715e54c61b966f
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Nov. 12, 2015, 10:47 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7a8517b166185f5c501aceacc780dfab21de1c0a
itests/src/test/resources/testconfiguration.properties 70f96da40c3c5b74e3ad2dd640321a2967cfcb86
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 90c2067e8279808fc0040e16d99a801b5d167eec
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java de67b54a2c6cfd9bc4413ebf7f715e54c61b966f
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Nov. 11, 2015, 3:19 p.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7a8517b166185f5c501aceacc780dfab21de1c0a
itests/src/test/resources/testconfiguration.properties 70f96da40c3c5b74e3ad2dd640321a2967cfcb86
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveCalciteUtil.java 90c2067e8279808fc0040e16d99a801b5d167eec
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortMergeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 728c5aa91f8d6eab6bcc2696c059ae1bdbeea689
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e13356ce16626bda1d9b6553ac5f191a65243394
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103714
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java (line 1307)
<https://reviews.apache.org/r/38211/#comment161785>
May be all the config should be changed to LimitPushDown instead of using join; since limit can be pushed down below outer joins, projects, union, Map Side GB...
- John Pullokkaran
On Oct. 23, 2015, 3:21 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2015, 3:21 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 23ae0dcca405bfe5521b1638f5c7f4e720c1a493
> itests/src/test/resources/testconfiguration.properties fa1d89d2169023b3842829e762e02b896909b4b0
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 6948fb16c6a3c9ce456a900536bee4c0d9b1614e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e13356ce16626bda1d9b6553ac5f191a65243394
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review103715
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java (line 962)
<https://reviews.apache.org/r/38211/#comment161786>
Pushing just limit without pushing orderby is not sufficent.
Limit almost always will have preceeding order by.
Order by pushdown also nneds to be handled.
- John Pullokkaran
On Oct. 23, 2015, 3:21 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2015, 3:21 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 23ae0dcca405bfe5521b1638f5c7f4e720c1a493
> itests/src/test/resources/testconfiguration.properties fa1d89d2169023b3842829e762e02b896909b4b0
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 6948fb16c6a3c9ce456a900536bee4c0d9b1614e
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e13356ce16626bda1d9b6553ac5f191a65243394
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Oct. 23, 2015, 3:21 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 23ae0dcca405bfe5521b1638f5c7f4e720c1a493
itests/src/test/resources/testconfiguration.properties fa1d89d2169023b3842829e762e02b896909b4b0
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 6948fb16c6a3c9ce456a900536bee4c0d9b1614e
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdSelectivity.java 715f24fc1872a54637a47bf80bb29e4a2e053779
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e13356ce16626bda1d9b6553ac5f191a65243394
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Oct. 11, 2015, 12:57 p.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java bf48f69dd8914020dcd4eeba8dd448f4ee8d5b24
itests/src/test/resources/testconfiguration.properties ad47fac2e3e5716f2cdf4b9c25ee2d6d14eab5ec
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveSortLimit.java 82fdc0e04011aa585912ce5aae5e3245e5e5bb01
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortRemoveRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/stats/HiveRelMdRowCount.java 6948fb16c6a3c9ce456a900536bee4c0d9b1614e
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 61ee2bdf21000aa20d9fff02a3d926a1031410bc
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Sept. 28, 2015, 11:40 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86
itests/src/test/resources/testconfiguration.properties 4f7b25f2563683933b9d677626f0447705927966
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8c5e58570570812b525b8bf0edb0f4c0a9
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Sept. 28, 2015, 9:27 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86
itests/src/test/resources/testconfiguration.properties 4f7b25f2563683933b9d677626f0447705927966
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8c5e58570570812b525b8bf0edb0f4c0a9
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Sept. 28, 2015, 9:04 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java dffdb5c14114d9c2d1e4c44a36f6b60966dcfc86
itests/src/test/resources/testconfiguration.properties 4f7b25f2563683933b9d677626f0447705927966
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortProjectTransposeRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9c731b8c5e58570570812b525b8bf0edb0f4c0a9
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review100455
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java (line 73)
<https://reviews.apache.org/r/38211/#comment157654>
May be reuse HiveCalciteUtil.orderRelNode
- John Pullokkaran
On Sept. 17, 2015, 1:55 p.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Sept. 17, 2015, 1:55 p.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da29f861c0e6c11065a95092d9e7e7b08aba
> itests/src/test/resources/testconfiguration.properties 0d3e1cc17df496e8b4dd8c3622017cd15384348a
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 0a7ce3a5f81f3175cddc59319753be3bb0b2af04
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Sept. 17, 2015, 1:55 p.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da29f861c0e6c11065a95092d9e7e7b08aba
itests/src/test/resources/testconfiguration.properties 0d3e1cc17df496e8b4dd8c3622017cd15384348a
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 0a7ce3a5f81f3175cddc59319753be3bb0b2af04
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Sept. 17, 2015, 1:51 p.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f29da29f861c0e6c11065a95092d9e7e7b08aba
itests/src/test/resources/testconfiguration.properties 0d3e1cc17df496e8b4dd8c3622017cd15384348a
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 0a7ce3a5f81f3175cddc59319753be3bb0b2af04
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez
Re: Review Request 38211: HIVE-11684
Posted by John Pullokkaran <jp...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/#review99318
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java (line 49)
<https://reviews.apache.org/r/38211/#comment156224>
It would be good to implement
public boolean matches(RelOptRuleCall call)
and check if sort rel is actualy limit & not sort
- John Pullokkaran
On Sept. 10, 2015, 8:23 a.m., Jesús Camacho Rodríguez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38211/
> -----------------------------------------------------------
>
> (Updated Sept. 10, 2015, 8:23 a.m.)
>
>
> Review request for hive and John Pullokkaran.
>
>
> Bugs: HIVE-11684
> https://issues.apache.org/jira/browse/HIVE-11684
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Implement limit pushdown through outer join in CBO
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2c5885b4808335cdddc826137d95a8a065dabf7
> itests/src/test/resources/testconfiguration.properties bed621d3eb74f01e54110552f68538afd228018d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 73ae7c4ef90af22847bef9e54863d4651d9c1013
> ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
> ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
> ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/38211/diff/
>
>
> Testing
> -------
>
> Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
>
>
> Thanks,
>
> Jesús Camacho Rodríguez
>
>
Re: Review Request 38211: HIVE-11684
Posted by Jesús Camacho Rodríguez <jc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38211/
-----------------------------------------------------------
(Updated Sept. 10, 2015, 8:23 a.m.)
Review request for hive and John Pullokkaran.
Bugs: HIVE-11684
https://issues.apache.org/jira/browse/HIVE-11684
Repository: hive-git
Description
-------
Implement limit pushdown through outer join in CBO
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d2c5885b4808335cdddc826137d95a8a065dabf7
itests/src/test/resources/testconfiguration.properties bed621d3eb74f01e54110552f68538afd228018d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSortJoinReduceRule.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 73ae7c4ef90af22847bef9e54863d4651d9c1013
ql/src/test/queries/clientpositive/limit_join_transpose.q PRE-CREATION
ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_3.q PRE-CREATION
ql/src/test/results/clientpositive/limit_join_transpose.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/tez_dynpart_hashjoin_3.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/38211/diff/
Testing
-------
Two new tests added: limit_join_transpose.q and tez_dynpart_hashjoin_3.q (only Tez)
Thanks,
Jesús Camacho Rodríguez