You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Ruslan Dautkhanov (JIRA)" <ji...@apache.org> on 2016/11/15 01:19:58 UTC
[jira] [Comment Edited] (SPARK-17076) Cardinality estimation of
join operator
[ https://issues.apache.org/jira/browse/SPARK-17076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665644#comment-15665644 ]
Ruslan Dautkhanov edited comment on SPARK-17076 at 11/15/16 1:19 AM:
---------------------------------------------------------------------
HIVE-13076 added FK constraint to Hive / HMS.
Joining two tables without declared relation can produce and number of rows.
On one extreme case, joining two tables produces zero rows (if there is no overlaps in those join set of ids).
On another extreme case, joining two tables will produce N x M rows where N and M are number of rows in each case.
Luckily, most often joining happens using columns that can have declared constraints on them.
For example,
customers.customer_id = purchases.customer_id AND purchases.product_id = products.product_id
If you'd build on HIVE-13076, Spark CBO will know exactly that customers.customer_id = purchases.customer_id AND purchases.product_id = products.product_id will always produce 1 row for each join (join cardinality = # of rows on the biggest table).
was (Author: tagar):
HIVE-13076 added FK constraint to Hive / HMS.
Joining two tables without declared relation can produce and number of rows.
On one extreme case, joining two tables produces zero rows (if there is no overlaps in those join set of ids).
On another extream case, joining two tables will produce N x M rows where N and M are number of rows in each case.
Luckliy, most often joining happens using columns that can have declared constraints on them.
For example,
customers.customer_id = purchases.customer_id AND purchases.product_id = products.product_id
If you'd build on HIVE-13076, Spark CBO will know exactly that customers.customer_id = purchases.customer_id AND purchases.product_id = products.product_id will always produce 1 row for each join (join cardinality = # of rows on the biggest table).
> Cardinality estimation of join operator
> ---------------------------------------
>
> Key: SPARK-17076
> URL: https://issues.apache.org/jira/browse/SPARK-17076
> Project: Spark
> Issue Type: Sub-task
> Components: Optimizer
> Affects Versions: 2.0.0
> Reporter: Ron Hu
>
> support cardinality estimates for equi-join, Cartesian product join, and outer join, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org