You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Vineet Garg <vg...@hortonworks.com> on 2018/08/04 00:59:20 UTC
Review Request 68202: HIVE-20292 Bad join ordering in tpcds query93
with primary constraint defined
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/
-----------------------------------------------------------
Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
Bugs: HIVE-20292
https://issues.apache.org/jira/browse/HIVE-20292
Repository: hive-git
Description
-------
See JIRA description
Diffs
-----
itests/src/test/resources/testconfiguration.properties a13ad28313
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
Diff: https://reviews.apache.org/r/68202/diff/1/
Testing
-------
Thanks,
Vineet Garg
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Vineet Garg <vg...@hortonworks.com>.
> On Aug. 4, 2018, 1:26 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
> > Lines 6751-6753 (original), 6743-6747 (patched)
> > <https://reviews.apache.org/r/68202/diff/1/?file=2067461#file2067461line6751>
> >
> > Extra GBys and joins. Looks like a regression.
Since the pattern of plan is changed HiveRemoveSqCountCheck is not kicking in (it is off by default) which removes extra join if scalar subquery has aggregate with group by on constant. HiveRemoveSqCountCheck expects project b/w join (so that removal of HiveJoin won't change the row type). To remdy I have put logic to apply HiveJoinProjectTransposeRule only if HiveRemoveSqCountCheck config is off. This is a bit ugly but I couldn't come up with anything better.
> On Aug. 4, 2018, 1:26 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
> > Lines 6876-6878 (original), 6917-6921 (patched)
> > <https://reviews.apache.org/r/68202/diff/1/?file=2067461#file2067461line6925>
> >
> > Extra GBys and joins. Looks like a regression.
Same as above
- Vineet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/#review206858
-----------------------------------------------------------
On Aug. 4, 2018, 12:59 a.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68202/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2018, 12:59 a.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
>
>
> Bugs: HIVE-20292
> https://issues.apache.org/jira/browse/HIVE-20292
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA description
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties a13ad28313
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
> ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
> ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
> ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
> ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
> ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
> ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
> ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
> ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
> ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
> ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
> ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
> ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
> ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
> ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
> ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
> ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
> ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
> ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
> ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
> ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
> ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
> ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
>
>
> Diff: https://reviews.apache.org/r/68202/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/#review206858
-----------------------------------------------------------
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
Lines 6751-6753 (original), 6743-6747 (patched)
<https://reviews.apache.org/r/68202/#comment289955>
Extra GBys and joins. Looks like a regression.
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out
Lines 6876-6878 (original), 6917-6921 (patched)
<https://reviews.apache.org/r/68202/#comment289954>
Extra GBys and joins. Looks like a regression.
- Ashutosh Chauhan
On Aug. 4, 2018, 12:59 a.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68202/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2018, 12:59 a.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
>
>
> Bugs: HIVE-20292
> https://issues.apache.org/jira/browse/HIVE-20292
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA description
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties a13ad28313
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
> ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
> ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
> ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
> ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
> ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
> ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
> ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
> ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
> ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
> ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
> ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
> ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
> ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
> ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
> ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
> ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
> ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
> ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
> ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
> ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
> ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
> ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
>
>
> Diff: https://reviews.apache.org/r/68202/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Vineet Garg <vg...@hortonworks.com>.
> On Aug. 6, 2018, 1:21 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 1799 (patched)
> > <https://reviews.apache.org/r/68202/diff/3/?file=2067607#file2067607line1799>
> >
> > This is not ideal. We want joinProject Transpose rule to run regardless of sq_count_check config. Infact, since sq_count_check generates more optimal plan, this especially needs to work with its value true. Looks like we need to enhance sq_count_check optimization so that it doesnt assume presence of project and can optimize the join-join plan too.
> >
> > On a side note default value for sq_count_check should be true. Lets do that too.
>
> Vineet Garg wrote:
> sq_count_check can not be enhanced to work on JOIN-JOIN pattern since it removes JOIN and can change the rowtype of the plan (that is why subquery removal logic explicitly inserts Project b/w join so that removing join won't change the rowtype). The reason sq_count_check is disabled by default is because it adds extra project to all scalar subqueries preventing multiple joins merging as well as preventing join reordering. Since this only optimizes rare group by <constant> pattern but it impacts all scalar subqueries it was decided to keep it off by default.
In latest patch I moved sq_count_check rule before JoinProjectTranspose rule. Now both rules will work together.
- Vineet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/#review206867
-----------------------------------------------------------
On Aug. 6, 2018, 2:58 a.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68202/
> -----------------------------------------------------------
>
> (Updated Aug. 6, 2018, 2:58 a.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
>
>
> Bugs: HIVE-20292
> https://issues.apache.org/jira/browse/HIVE-20292
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA description
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties fa90750766
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java d2d5152eff
> ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
> ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
> ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
> ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
> ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
> ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
> ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
> ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
> ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
> ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
> ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
> ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
> ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
> ql/src/test/results/clientpositive/perf/spark/query2.q.out 105f2ee1ae
> ql/src/test/results/clientpositive/perf/spark/query59.q.out 73ea161fc4
> ql/src/test/results/clientpositive/perf/spark/query95.q.out 14636717af
> ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
> ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
> ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
> ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
> ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
> ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
> ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
> ql/src/test/results/clientpositive/spark/column_access_stats.q.out f372eefebb
> ql/src/test/results/clientpositive/spark/join28.q.out 40d328c2bf
> ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 9c1e4361d8
> ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out ab25791aa6
> ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out 112c1871e9
> ql/src/test/results/clientpositive/spark/runtime_skewjoin_mapjoin_spark.q.out 57ab0ecb63
> ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out 16fa81870c
> ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
> ql/src/test/results/clientpositive/spark/subquery_scalar.q.out e2d51e85cf
> ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 87983138aa
> ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
>
>
> Diff: https://reviews.apache.org/r/68202/diff/4/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Vineet Garg <vg...@hortonworks.com>.
> On Aug. 6, 2018, 1:21 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 1799 (patched)
> > <https://reviews.apache.org/r/68202/diff/3/?file=2067607#file2067607line1799>
> >
> > This is not ideal. We want joinProject Transpose rule to run regardless of sq_count_check config. Infact, since sq_count_check generates more optimal plan, this especially needs to work with its value true. Looks like we need to enhance sq_count_check optimization so that it doesnt assume presence of project and can optimize the join-join plan too.
> >
> > On a side note default value for sq_count_check should be true. Lets do that too.
sq_count_check can not be enhanced to work on JOIN-JOIN pattern since it removes JOIN and can change the rowtype of the plan (that is why subquery removal logic explicitly inserts Project b/w join so that removing join won't change the rowtype). The reason sq_count_check is disabled by default is because it adds extra project to all scalar subqueries preventing multiple joins merging as well as preventing join reordering. Since this only optimizes rare group by <constant> pattern but it impacts all scalar subqueries it was decided to keep it off by default.
- Vineet
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/#review206867
-----------------------------------------------------------
On Aug. 4, 2018, 6:28 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68202/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2018, 6:28 p.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
>
>
> Bugs: HIVE-20292
> https://issues.apache.org/jira/browse/HIVE-20292
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA description
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties a13ad28313
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
> ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
> ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
> ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
> ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
> ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
> ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
> ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
> ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
> ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
> ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
> ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
> ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
> ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
> ql/src/test/results/clientpositive/perf/spark/query2.q.out 105f2ee1ae
> ql/src/test/results/clientpositive/perf/spark/query59.q.out 73ea161fc4
> ql/src/test/results/clientpositive/perf/spark/query95.q.out 14636717af
> ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
> ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
> ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
> ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
> ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
> ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
> ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
> ql/src/test/results/clientpositive/spark/column_access_stats.q.out f372eefebb
> ql/src/test/results/clientpositive/spark/join28.q.out 40d328c2bf
> ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 9c1e4361d8
> ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out ab25791aa6
> ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out 112c1871e9
> ql/src/test/results/clientpositive/spark/runtime_skewjoin_mapjoin_spark.q.out 57ab0ecb63
> ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out 16fa81870c
> ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
> ql/src/test/results/clientpositive/spark/subquery_scalar.q.out e2d51e85cf
> ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 87983138aa
> ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
>
>
> Diff: https://reviews.apache.org/r/68202/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Ashutosh Chauhan <ha...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/#review206867
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 1799 (patched)
<https://reviews.apache.org/r/68202/#comment289966>
This is not ideal. We want joinProject Transpose rule to run regardless of sq_count_check config. Infact, since sq_count_check generates more optimal plan, this especially needs to work with its value true. Looks like we need to enhance sq_count_check optimization so that it doesnt assume presence of project and can optimize the join-join plan too.
On a side note default value for sq_count_check should be true. Lets do that too.
- Ashutosh Chauhan
On Aug. 4, 2018, 6:28 p.m., Vineet Garg wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/68202/
> -----------------------------------------------------------
>
> (Updated Aug. 4, 2018, 6:28 p.m.)
>
>
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
>
>
> Bugs: HIVE-20292
> https://issues.apache.org/jira/browse/HIVE-20292
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA description
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties a13ad28313
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
> ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
> ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
> ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
> ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
> ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
> ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
> ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
> ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
> ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
> ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
> ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
> ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
> ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
> ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
> ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
> ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
> ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
> ql/src/test/results/clientpositive/perf/spark/query2.q.out 105f2ee1ae
> ql/src/test/results/clientpositive/perf/spark/query59.q.out 73ea161fc4
> ql/src/test/results/clientpositive/perf/spark/query95.q.out 14636717af
> ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
> ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
> ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
> ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
> ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
> ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
> ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
> ql/src/test/results/clientpositive/spark/column_access_stats.q.out f372eefebb
> ql/src/test/results/clientpositive/spark/join28.q.out 40d328c2bf
> ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 9c1e4361d8
> ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out ab25791aa6
> ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out 112c1871e9
> ql/src/test/results/clientpositive/spark/runtime_skewjoin_mapjoin_spark.q.out 57ab0ecb63
> ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out 16fa81870c
> ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
> ql/src/test/results/clientpositive/spark/subquery_scalar.q.out e2d51e85cf
> ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 87983138aa
> ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
>
>
> Diff: https://reviews.apache.org/r/68202/diff/3/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Vineet Garg
>
>
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/
-----------------------------------------------------------
(Updated Aug. 6, 2018, 2:58 a.m.)
Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
Bugs: HIVE-20292
https://issues.apache.org/jira/browse/HIVE-20292
Repository: hive-git
Description
-------
See JIRA description
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties fa90750766
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java d2d5152eff
ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
ql/src/test/results/clientpositive/perf/spark/query2.q.out 105f2ee1ae
ql/src/test/results/clientpositive/perf/spark/query59.q.out 73ea161fc4
ql/src/test/results/clientpositive/perf/spark/query95.q.out 14636717af
ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
ql/src/test/results/clientpositive/spark/column_access_stats.q.out f372eefebb
ql/src/test/results/clientpositive/spark/join28.q.out 40d328c2bf
ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 9c1e4361d8
ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out ab25791aa6
ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out 112c1871e9
ql/src/test/results/clientpositive/spark/runtime_skewjoin_mapjoin_spark.q.out 57ab0ecb63
ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out 16fa81870c
ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
ql/src/test/results/clientpositive/spark/subquery_scalar.q.out e2d51e85cf
ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 87983138aa
ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
Diff: https://reviews.apache.org/r/68202/diff/4/
Changes: https://reviews.apache.org/r/68202/diff/3-4/
Testing
-------
Thanks,
Vineet Garg
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/
-----------------------------------------------------------
(Updated Aug. 4, 2018, 6:28 p.m.)
Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
Bugs: HIVE-20292
https://issues.apache.org/jira/browse/HIVE-20292
Repository: hive-git
Description
-------
See JIRA description
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties a13ad28313
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
ql/src/test/results/clientpositive/perf/spark/query2.q.out 105f2ee1ae
ql/src/test/results/clientpositive/perf/spark/query59.q.out 73ea161fc4
ql/src/test/results/clientpositive/perf/spark/query95.q.out 14636717af
ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
ql/src/test/results/clientpositive/spark/column_access_stats.q.out f372eefebb
ql/src/test/results/clientpositive/spark/join28.q.out 40d328c2bf
ql/src/test/results/clientpositive/spark/join32_lessSize.q.out 9c1e4361d8
ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out ab25791aa6
ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out 112c1871e9
ql/src/test/results/clientpositive/spark/runtime_skewjoin_mapjoin_spark.q.out 57ab0ecb63
ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out 16fa81870c
ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
ql/src/test/results/clientpositive/spark/subquery_scalar.q.out e2d51e85cf
ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 87983138aa
ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
Diff: https://reviews.apache.org/r/68202/diff/3/
Changes: https://reviews.apache.org/r/68202/diff/2-3/
Testing
-------
Thanks,
Vineet Garg
Re: Review Request 68202: HIVE-20292 Bad join ordering in tpcds
query93 with primary constraint defined
Posted by Vineet Garg <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/68202/
-----------------------------------------------------------
(Updated Aug. 4, 2018, 3:50 a.m.)
Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
Changes
-------
Review comments
Bugs: HIVE-20292
https://issues.apache.org/jira/browse/HIVE-20292
Repository: hive-git
Description
-------
See JIRA description
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties a13ad28313
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveJoinProjectTransposeRule.java 8214cc9a0d
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java e6e033066d
ql/src/test/queries/clientpositive/q93_with_constraints.q PRE-CREATION
ql/src/test/results/clientpositive/correlationoptimizer5.q.out aab38e5768
ql/src/test/results/clientpositive/join28.q.out 8a3d79da47
ql/src/test/results/clientpositive/llap/column_access_stats.q.out 1c873e2fe4
ql/src/test/results/clientpositive/llap/correlationoptimizer2.q.out cb5f357780
ql/src/test/results/clientpositive/llap/correlationoptimizer3.q.out c9bff2895c
ql/src/test/results/clientpositive/llap/dynamic_semijoin_reduction_sw.q.out 605686591f
ql/src/test/results/clientpositive/llap/join32_lessSize.q.out f1b4fb2ed3
ql/src/test/results/clientpositive/llap/llap_smb_ptf.q.out 33da686fe3
ql/src/test/results/clientpositive/llap/multiMapJoin1.q.out 261e153aba
ql/src/test/results/clientpositive/llap/q93_with_constraints.q.out PRE-CREATION
ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 48fb6794ea
ql/src/test/results/clientpositive/llap/tez_self_join.q.out a362411d41
ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 1bfe5af603
ql/src/test/results/clientpositive/mapjoin_subquery.q.out 6116d382e8
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 50e7ce0f3c
ql/src/test/results/clientpositive/perf/tez/query2.q.out 5f908948aa
ql/src/test/results/clientpositive/perf/tez/query59.q.out 29cf1366d8
ql/src/test/results/clientpositive/perf/tez/query95.q.out 3a8ed092fb
ql/src/test/results/clientpositive/ppd_repeated_alias.q.out b9434b107e
ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out c1f3401f42
ql/src/test/results/clientpositive/smb_mapjoin_25.q.out 7366408ef9
ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 00d5b6532a
ql/src/test/results/clientpositive/spark/spark_use_op_stats.q.out 26dd1f059f
ql/src/test/results/clientpositive/vector_outer_join6.q.out 3c24928ffa
Diff: https://reviews.apache.org/r/68202/diff/2/
Changes: https://reviews.apache.org/r/68202/diff/1-2/
Testing
-------
Thanks,
Vineet Garg