You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Harish Butani <rh...@gmail.com> on 2014/03/13 00:58:54 UTC
Review Request 19165: HIVE-6643: Add a check for cross products in plans and
output a warning
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19165/
-----------------------------------------------------------
Review request for hive and Gunther Hagleitner.
Bugs: HIVE-6643
https://issues.apache.org/jira/browse/HIVE-6643
Repository: hive-git
Description
-------
Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product.
For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z
So the JoinTree is:
A — B
__ D — C
Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C
Diffs
-----
itests/qtest/pom.xml f8b81a2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java d593d08
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a5e6cbf
ql/src/test/queries/clientpositive/cross_product_check_1.q PRE-CREATION
ql/src/test/queries/clientpositive/cross_product_check_2.q PRE-CREATION
ql/src/test/results/clientpositive/cross_product_check_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/cross_product_check_2.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/19165/diff/
Testing
-------
added new tests
Thanks,
Harish Butani
Re: Review Request 19165: HIVE-6643: Add a check for cross products in plans
and output a warning
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19165/#review37151
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java
<https://reviews.apache.org/r/19165/#comment68541>
this should say CrossProductCheck.class
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java
<https://reviews.apache.org/r/19165/#comment68543>
I think you'll always have at least one element - but that's not super obvious. a check for empty might be the defensive choice.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java
<https://reviews.apache.org/r/19165/#comment68544>
wondering if we should just simplify this and not go hunt for bigtable at all.
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java
<https://reviews.apache.org/r/19165/#comment68545>
not sure if the coding guidelines require uppercase/camelcase inner classes...
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java
<https://reviews.apache.org/r/19165/#comment68546>
i'm fine with this. but most folks put bug defense flag check in "optimizers". might help if this turns out to be expensive or if it fails for some reason. (definitely should be on by default tho).
- Gunther Hagleitner
On March 12, 2014, 11:58 p.m., Harish Butani wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/19165/
> -----------------------------------------------------------
>
> (Updated March 12, 2014, 11:58 p.m.)
>
>
> Review request for hive and Gunther Hagleitner.
>
>
> Bugs: HIVE-6643
> https://issues.apache.org/jira/browse/HIVE-6643
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product.
> For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z
> So the JoinTree is:
> A — B
> __ D — C
> Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C
>
>
> Diffs
> -----
>
> itests/qtest/pom.xml f8b81a2
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java d593d08
> ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a5e6cbf
> ql/src/test/queries/clientpositive/cross_product_check_1.q PRE-CREATION
> ql/src/test/queries/clientpositive/cross_product_check_2.q PRE-CREATION
> ql/src/test/results/clientpositive/cross_product_check_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/cross_product_check_2.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out PRE-CREATION
> ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out PRE-CREATION
>
> Diff: https://reviews.apache.org/r/19165/diff/
>
>
> Testing
> -------
>
> added new tests
>
>
> Thanks,
>
> Harish Butani
>
>
Re: Review Request 19165: HIVE-6643: Add a check for cross products in plans
and output a warning
Posted by Harish Butani <rh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/19165/
-----------------------------------------------------------
(Updated March 17, 2014, 12:42 a.m.)
Review request for hive and Gunther Hagleitner.
Changes
-------
changes suggested by gunther
Bugs: HIVE-6643
https://issues.apache.org/jira/browse/HIVE-6643
Repository: hive-git
Description
-------
Now that we support old style join syntax, it is easy to write queries that generate a plan with a cross product.
For e.g. say you have A join B join C join D on A.x = B.x and A.y = D.y and C.z = D.z
So the JoinTree is:
A — B
__ D — C
Since we don't reorder join graphs, we will end up with a cross product between (A join B) and C
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5
conf/hive-default.xml.template 906ea8c
itests/qtest/pom.xml f8b81a2
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CrossProductCheck.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java d593d08
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java a5e6cbf
ql/src/test/queries/clientpositive/cross_product_check_1.q PRE-CREATION
ql/src/test/queries/clientpositive/cross_product_check_2.q PRE-CREATION
ql/src/test/results/clientpositive/cross_product_check_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/cross_product_check_2.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out PRE-CREATION
ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/19165/diff/
Testing
-------
added new tests
Thanks,
Harish Butani