You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vineet Garg <vg...@hortonworks.com> on 2017/06/11 19:17:39 UTC
Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/
-----------------------------------------------------------
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-16838
https://issues.apache.org/jira/browse/HIVE-16838
Repository: hive-git
Description
-------
This patch improves plans for subqueries which have not equal corelated predicates.
Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261
ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474
ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82
ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5
ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975
ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30
ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out d3586e0db2
ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2
ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00
ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d
ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 2f2609f03e
ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035
ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 82bef24230
Diff: https://reviews.apache.org/r/59984/diff/1/
Testing
-------
* Added new tests
* Pre-commit testing
Thanks,
Vineet Garg
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/
-----------------------------------------------------------
(Updated June 27, 2017, 11:12 p.m.)
Review request for hive and Ashutosh Chauhan.
Changes
-------
Addressed review comments
Bugs: HIVE-16838
https://issues.apache.org/jira/browse/HIVE-16838
Repository: hive-git
Description
-------
This patch improves plans for subqueries which have not equal corelated predicates.
Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
ql/src/test/results/clientpositive/llap/explainuser_1.q.out e78ceb3a4c
ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
ql/src/test/results/clientpositive/llap/subquery_in.q.out b3d735afb9
ql/src/test/results/clientpositive/llap/subquery_multi.q.out 718bc13ed4
ql/src/test/results/clientpositive/llap/subquery_notin.q.out d10fd84a3a
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out ceccc88b2e
ql/src/test/results/clientpositive/llap/subquery_select.q.out 440e9d7b55
ql/src/test/results/clientpositive/llap/subquery_views.q.out ad199e2336
ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 8aabb62b9e
ql/src/test/results/clientpositive/perf/query16.q.out 239f2b7236
ql/src/test/results/clientpositive/perf/query94.q.out 9cc934fd1b
ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
ql/src/test/results/clientpositive/spark/subquery_in.q.out ae4e6fb4f4
ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 433b9a2880
ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
ql/src/test/results/clientpositive/subquery_in_having.q.out 2bb812cd73
ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 0ff11df8a3
Diff: https://reviews.apache.org/r/59984/diff/3/
Changes: https://reviews.apache.org/r/59984/diff/2-3/
Testing
-------
* Added new tests
* Pre-commit testing
Thanks,
Vineet Garg
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Vineet Garg <vg...@hortonworks.com>.
> On June 27, 2017, 9:24 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_in.q.out
> > Lines 5531-5540 (patched)
> > <https://reviews.apache.org/r/59984/diff/1-2/?file=1747778#file1747778line5623>
> >
> > Change in resultset. Either earlier results were wrong or new one.
Thanks for catching this. I missed it because i was comparing it with master!
- Vineet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review179052
-----------------------------------------------------------
On June 26, 2017, 8:41 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 26, 2017, 8:41 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out e78ceb3a4c
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out b3d735afb9
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 718bc13ed4
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out d10fd84a3a
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out ceccc88b2e
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 440e9d7b55
> ql/src/test/results/clientpositive/llap/subquery_views.q.out ad199e2336
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 8aabb62b9e
> ql/src/test/results/clientpositive/perf/query16.q.out 239f2b7236
> ql/src/test/results/clientpositive/perf/query94.q.out 9cc934fd1b
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out ae4e6fb4f4
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 433b9a2880
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out 2bb812cd73
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 0ff11df8a3
>
>
> Diff: https://reviews.apache.org/r/59984/diff/2/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review179052
-----------------------------------------------------------
ql/src/test/results/clientpositive/llap/subquery_in.q.out
Lines 5531-5540 (patched)
<https://reviews.apache.org/r/59984/#comment253456>
Change in resultset. Either earlier results were wrong or new one.
- Ashutosh Chauhan
On June 26, 2017, 8:41 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 26, 2017, 8:41 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out e78ceb3a4c
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out b3d735afb9
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 718bc13ed4
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out d10fd84a3a
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out ceccc88b2e
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 440e9d7b55
> ql/src/test/results/clientpositive/llap/subquery_views.q.out ad199e2336
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 8aabb62b9e
> ql/src/test/results/clientpositive/perf/query16.q.out 239f2b7236
> ql/src/test/results/clientpositive/perf/query94.q.out 9cc934fd1b
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out ae4e6fb4f4
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 433b9a2880
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out 2bb812cd73
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 0ff11df8a3
>
>
> Diff: https://reviews.apache.org/r/59984/diff/2/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review179054
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 3129-3130 (original), 3112-3113 (patched)
<https://reviews.apache.org/r/59984/#comment253458>
If I am reading this right, than it implies we will use valueGen even when subquery is in project. Is that expected?
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 3145-3147 (original), 3128-3130 (patched)
<https://reviews.apache.org/r/59984/#comment253459>
Logic seems to be doing opposite of comment, that is dont use valueGen when grouping sets are present.
- Ashutosh Chauhan
On June 26, 2017, 8:41 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 26, 2017, 8:41 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out e78ceb3a4c
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out b3d735afb9
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 718bc13ed4
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out d10fd84a3a
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out ceccc88b2e
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 440e9d7b55
> ql/src/test/results/clientpositive/llap/subquery_views.q.out ad199e2336
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 8aabb62b9e
> ql/src/test/results/clientpositive/perf/query16.q.out 239f2b7236
> ql/src/test/results/clientpositive/perf/query94.q.out 9cc934fd1b
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out ae4e6fb4f4
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 433b9a2880
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out 2bb812cd73
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 0ff11df8a3
>
>
> Diff: https://reviews.apache.org/r/59984/diff/2/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/
-----------------------------------------------------------
(Updated June 26, 2017, 8:41 p.m.)
Review request for hive and Ashutosh Chauhan.
Bugs: HIVE-16838
https://issues.apache.org/jira/browse/HIVE-16838
Repository: hive-git
Description
-------
This patch improves plans for subqueries which have not equal corelated predicates.
Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
ql/src/test/results/clientpositive/llap/explainuser_1.q.out e78ceb3a4c
ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
ql/src/test/results/clientpositive/llap/subquery_in.q.out b3d735afb9
ql/src/test/results/clientpositive/llap/subquery_multi.q.out 718bc13ed4
ql/src/test/results/clientpositive/llap/subquery_notin.q.out d10fd84a3a
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out ceccc88b2e
ql/src/test/results/clientpositive/llap/subquery_select.q.out 440e9d7b55
ql/src/test/results/clientpositive/llap/subquery_views.q.out ad199e2336
ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out 8aabb62b9e
ql/src/test/results/clientpositive/perf/query16.q.out 239f2b7236
ql/src/test/results/clientpositive/perf/query94.q.out 9cc934fd1b
ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
ql/src/test/results/clientpositive/spark/subquery_in.q.out ae4e6fb4f4
ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 433b9a2880
ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
ql/src/test/results/clientpositive/subquery_in_having.q.out 2bb812cd73
ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 0ff11df8a3
Diff: https://reviews.apache.org/r/59984/diff/2/
Changes: https://reviews.apache.org/r/59984/diff/1-2/
Testing
-------
* Added new tests
* Pre-commit testing
Thanks,
Vineet Garg
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Ashutosh Chauhan <ha...@apache.org>.
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
> > Lines 137-139 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747770#file1747770line137>
> >
> > Seems like this if branch is unnecessary.
>
> Vineet Garg wrote:
> I don't think so, most of the calcite code/ hive rules implement RelShuttle interface. So accept should reroute accordingly
>
> Ashutosh Chauhan wrote:
> Its unnecessary in sense that even without this branch logic of this function doesn't change.
>
> Vineet Garg wrote:
> I am not sure if I understand this. If we remove 'if' branch and just have 'return shuttle.visit(this)' wouldn't this call RelShuttle's version 'visit(relnode)' instead of HiveRelShuttle's visit(HiveAggregate) ?
Nope. It will call most specific type which will be HiveRelShuttle.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review178248
-----------------------------------------------------------
On June 11, 2017, 7:17 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 11, 2017, 7:17 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975
> ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out d3586e0db2
> ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2
> ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 2f2609f03e
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 82bef24230
>
>
> Diff: https://reviews.apache.org/r/59984/diff/1/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Ashutosh Chauhan <ha...@apache.org>.
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
> > Lines 137-139 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747770#file1747770line137>
> >
> > Seems like this if branch is unnecessary.
>
> Vineet Garg wrote:
> I don't think so, most of the calcite code/ hive rules implement RelShuttle interface. So accept should reroute accordingly
Its unnecessary in sense that even without this branch logic of this function doesn't change.
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
> > Lines 3088 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747773#file1747773line3109>
> >
> > Might be better to initialize this variable to true. Since, value being false is known and limited set.
>
> Vineet Garg wrote:
> I think this works better for readability. e.g. Aggregate looks for grouping sets and if it finds one sets the flag to true and return, otherwise it traveses all its children. Changing the intial flag to true will entail negating this logic and will anyway keep all the code/cases.
Initialization with false is more robust since that implies by default valueGen is required. Thats a good thing since we know all cases when its not required, but not necessarily otherwise. e.g., if there are more visit methods added in this interface in future and this implementation is not updated, initialization with false will save us from bugs like those.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review178248
-----------------------------------------------------------
On June 11, 2017, 7:17 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 11, 2017, 7:17 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975
> ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out d3586e0db2
> ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2
> ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 2f2609f03e
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 82bef24230
>
>
> Diff: https://reviews.apache.org/r/59984/diff/1/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Vineet Garg <vg...@hortonworks.com>.
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
> > Lines 137-139 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747770#file1747770line137>
> >
> > Seems like this if branch is unnecessary.
>
> Vineet Garg wrote:
> I don't think so, most of the calcite code/ hive rules implement RelShuttle interface. So accept should reroute accordingly
>
> Ashutosh Chauhan wrote:
> Its unnecessary in sense that even without this branch logic of this function doesn't change.
I am not sure if I understand this. If we remove 'if' branch and just have 'return shuttle.visit(this)' wouldn't this call RelShuttle's version 'visit(relnode)' instead of HiveRelShuttle's visit(HiveAggregate) ?
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
> > Lines 3088 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747773#file1747773line3109>
> >
> > Might be better to initialize this variable to true. Since, value being false is known and limited set.
>
> Vineet Garg wrote:
> I think this works better for readability. e.g. Aggregate looks for grouping sets and if it finds one sets the flag to true and return, otherwise it traveses all its children. Changing the intial flag to true will entail negating this logic and will anyway keep all the code/cases.
>
> Ashutosh Chauhan wrote:
> Initialization with false is more robust since that implies by default valueGen is required. Thats a good thing since we know all cases when its not required, but not necessarily otherwise. e.g., if there are more visit methods added in this interface in future and this implementation is not updated, initialization with false will save us from bugs like those.
That's a good point. I'll update the code.
- Vineet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review178248
-----------------------------------------------------------
On June 11, 2017, 7:17 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 11, 2017, 7:17 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975
> ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out d3586e0db2
> ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2
> ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 2f2609f03e
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 82bef24230
>
>
> Diff: https://reviews.apache.org/r/59984/diff/1/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Vineet Garg <vg...@hortonworks.com>.
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
> > Lines 137-139 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747770#file1747770line137>
> >
> > Seems like this if branch is unnecessary.
I don't think so, most of the calcite code/ hive rules implement RelShuttle interface. So accept should reroute accordingly
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
> > Lines 47-49 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747771#file1747771line47>
> >
> > Seems like this if branch is unnecessary.
same as above
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java
> > Lines 46-48 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747772#file1747772line46>
> >
> > Seems like this if branch is unnecessary.
same as above
> On June 20, 2017, 12:17 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
> > Lines 3088 (patched)
> > <https://reviews.apache.org/r/59984/diff/1/?file=1747773#file1747773line3109>
> >
> > Might be better to initialize this variable to true. Since, value being false is known and limited set.
I think this works better for readability. e.g. Aggregate looks for grouping sets and if it finds one sets the flag to true and return, otherwise it traveses all its children. Changing the intial flag to true will entail negating this logic and will anyway keep all the code/cases.
- Vineet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review178248
-----------------------------------------------------------
On June 11, 2017, 7:17 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 11, 2017, 7:17 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975
> ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out d3586e0db2
> ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2
> ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 2f2609f03e
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 82bef24230
>
>
> Diff: https://reviews.apache.org/r/59984/diff/1/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 59984: Improve plans for subqueries with non-equi
co-related predicates
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59984/#review178248
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java
Lines 137-139 (patched)
<https://reviews.apache.org/r/59984/#comment252122>
Seems like this if branch is unnecessary.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java
Lines 47-49 (patched)
<https://reviews.apache.org/r/59984/#comment252123>
Seems like this if branch is unnecessary.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java
Lines 46-48 (patched)
<https://reviews.apache.org/r/59984/#comment252124>
Seems like this if branch is unnecessary.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 102 (patched)
<https://reviews.apache.org/r/59984/#comment252222>
We use org.slf4j.Logger;
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Line 1236 (original), 1257 (patched)
<https://reviews.apache.org/r/59984/#comment252223>
Lets use RexSimplify.simplify() here.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Line 1384 (original), 1419 (patched)
<https://reviews.apache.org/r/59984/#comment252224>
Add assert corDef.getPredicateKind() == SqlKind.EQUALS
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 1868 (patched)
<https://reviews.apache.org/r/59984/#comment252226>
Seems like value generator shouldn't have generated those predicates at all. Lets leave a TODO to remove this logic once value generator is fixed to not generate unnecessary predicates.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
Lines 3088 (patched)
<https://reviews.apache.org/r/59984/#comment252228>
Might be better to initialize this variable to true. Since, value being false is known and limited set.
- Ashutosh Chauhan
On June 11, 2017, 7:17 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59984/
> -----------------------------------------------------------
>
> (Updated June 11, 2017, 7:17 p.m.)
>
>
> Review request for hive and Ashutosh Chauhan.
>
>
> Bugs: HIVE-16838
> https://issues.apache.org/jira/browse/HIVE-16838
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This patch improves plans for subqueries which have not equal corelated predicates.
> Currently to retrieve all possible correlated predicates inner table is joined with outer query. This is un-necessary in most of the cases (exception is if subquery has an aggregate).
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveAggregate.java 63bbdaccfb
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveIntersect.java 19e1e026f4
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveUnion.java 7cfb007a9d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java 4c99932759
> ql/src/test/queries/clientpositive/subquery_in.q 4ba170a706
> ql/src/test/results/clientpositive/constprog_partitioner.q.out 8c7f9d3f29
> ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261
> ql/src/test/results/clientpositive/llap/subquery_exists.q.out 3004e36c9d
> ql/src/test/results/clientpositive/llap/subquery_in.q.out 1f9c9e4474
> ql/src/test/results/clientpositive/llap/subquery_multi.q.out 29516eff82
> ql/src/test/results/clientpositive/llap/subquery_notin.q.out b4af91579b
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out b78df8b9f5
> ql/src/test/results/clientpositive/llap/subquery_select.q.out 202980e975
> ql/src/test/results/clientpositive/llap/subquery_views.q.out 1a21a02a30
> ql/src/test/results/clientpositive/llap/vector_mapjoin_reduce.q.out d3586e0db2
> ql/src/test/results/clientpositive/perf/query16.q.out a7f93f9ec2
> ql/src/test/results/clientpositive/perf/query94.q.out c5fc9e7f00
> ql/src/test/results/clientpositive/spark/constprog_partitioner.q.out 3467215d63
> ql/src/test/results/clientpositive/spark/subquery_exists.q.out 8768b45166
> ql/src/test/results/clientpositive/spark/subquery_in.q.out 80a350656d
> ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 2f2609f03e
> ql/src/test/results/clientpositive/subquery_exists.q.out cfc76520ce
> ql/src/test/results/clientpositive/subquery_exists_having.q.out 2c41ff6c33
> ql/src/test/results/clientpositive/subquery_in_having.q.out c4569ba035
> ql/src/test/results/clientpositive/subquery_notexists.q.out 039df03819
> ql/src/test/results/clientpositive/subquery_notexists_having.q.out fda801d387
> ql/src/test/results/clientpositive/subquery_notin_having.q.out 462dda5e14
> ql/src/test/results/clientpositive/subquery_unqualcolumnrefs.q.out 03eb4b6ba4
> ql/src/test/results/clientpositive/vector_mapjoin_reduce.q.out 82bef24230
>
>
> Diff: https://reviews.apache.org/r/59984/diff/1/
>
>
> Testing
> -------
>
> * Added new tests
> * Pre-commit testing
>
>
> Thanks,
>
> Vineet Garg
>
>